Difference between revisions of "Preparation for 2021 Run"

From GlueXWiki
Jump to: navigation, search
 
(33 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
'''Below  is a list of action items which we would like accomplish before the PrimEx Phase II run in Spring/Summer of 2021'''
 
'''Below  is a list of action items which we would like accomplish before the PrimEx Phase II run in Spring/Summer of 2021'''
 +
''' Installation in Hall D'''
  
 +
[https://halldweb.jlab.org/primexd/prep_run_2021/ccal/he_pipe.jpg He pipe installed]
 +
[https://halldweb.jlab.org/primexd/prep_run_2021/ccal/ccal_on_platform.jpg CCAL installed in the platform]
 +
[https://halldweb.jlab.org/primexd/prep_run_2021/ccal/shield_wall.jpg Beamline shield wall]
 +
 +
 +
[https://halldweb.jlab.org/primexd/prep_run_2021/primex_05_24_2021.pptx  Preparation for PrimEx run in 2021 (Talk at the collaboration meeting 05/24/2021]
 +
 +
[[CCAL preparation]]
 +
 +
[[FCAL preparation status]]
 +
 +
[https://logbooks.jlab.org/entry/3900608 TAGH preparation]
  
 
;Aligning FCAL in the Hall        <span style="color:brown">  -- Coordinated by Tim  Whitlatch  </span>
 
;Aligning FCAL in the Hall        <span style="color:brown">  -- Coordinated by Tim  Whitlatch  </span>
Line 6: Line 19:
  
  
;Perform preventative replacement of FCAL PMT bases in 4 inner rings, which is critical for reconstruction of Compton events <span style="color:brown"> -- Support needed from Chris Stanislav, Fernando Barbosa, Mark Dalton, Matt Shepherd </span>
+
;Preventative maintenance of FCAL PMT bases  <span style="color:brown"> -- Coordinated by Mark D, with support from Chris S., Nick S., Fernando B., Matt S., Ben R., Paul S.  </span>
: Several bases have already been replaced in this area. The status of FCAL bases in the 3 inner FCAL rings was provided by Colin and is presented in the link below
+
# '''Change out bases which draw excessive current (> 13 mA) when they are on or which show other signs of problems.'''    High current bases are associated with [https://logbooks.jlab.org/entry/3848122 increased temperature, increased deviation from setpoint] and future high voltage failure.  There are about 200 high current bases.  They are repaired using the standard procedure for cleaning and replacing resistors on the base.
:[https://logbooks.jlab.org/entry/3810770 Status of FCAL dividers in 3 inner rings]
+
#* Status:
 +
#** [https://logbooks.jlab.org/entry/3853228 The first 100 bases to remove have been identified]
 +
#** 2020-09-29  [https://logbooks.jlab.org/entry/3854247 21 bases have been removed from FCAL]
 +
#** 2020-10-05  [https://logbooks.jlab.org/entry/3854320 20 bases have been removed from FCAL]
 +
#** 2020-10-08  [https://logbooks.jlab.org/entry/3854448 21 bases have been removed from FCAL]
 +
#** 2020-10-13  [https://logbooks.jlab.org/entry/3854748 18 bases have been removed from FCAL]
 +
#** 2020-11-25  [https://logbooks.jlab.org/entry/3858649 91 Bases on test bench for testingover holiday weekend.] 9 Bases failed for whatever reason.
 +
#** 2020-11-30  [https://logbooks.jlab.org/entry/3858886 1 base has problem while testing.]  Will continue testing at 1800 V. 
 +
#** 2020-12-03 All 91 bases hold voltage at 1800 V. [https://logbooks.jlab.org/entry/3859079. Currents are okay and all bases acceptable.]
 +
# '''Diagnose bases which draw excessive current (> 13 mA) when they are on and off.'''  This curious symptom is not understood. [https://logbooks.jlab.org/entry/3846223]
 +
#* Status:
 +
#** [https://logbooks.jlab.org/entry/3853381 8 bases have been remove from the FCAL] for special diagnosis.  These plus 2 more have arrived with Paul at IU with details of the problem symptoms. 
 +
#** Report from Paul.  Base 23286 drew 14 mA when powered up.  The COMM board by itself draws 14 mA, so the problem is there.  It was not immediately obvious what is wrong with the COMM board.  The other 7 had current draws > 20 mA when powered up.  In all cases the COMM board by itself had most of the excess current draw.  In all 7 chip U9 on the COMM board was warm or hot to the touch.  This chip drives the test pulser on the SOCK board.  If there is a spark at the PMT socket the test pulser transistor is destroyed and U9 is damaged.
 +
# '''Improve robustness of firmware for handling communication failures.'''  When the beam is on we regularly have about 2-3 bases that lose communication per day.  The HV drops to zero and the only known way to recover is to power cycle all the bases at the end of the run.  This means that we permanently have some holes in the FCAL from perfectly working bases.  We will try to modify the firmware to fix this issues. <span style="color:brown">  Ben R., Mark D. </span>
 +
#* Plan
 +
## compile the existing code and load onto a test unit to make sure we can reproduce what you currently have
 +
## have it communicate over the CAN interface like normal
 +
## try to crash the micro-controller with a radiation source
 +
## make a small firmware change to enable the watchdog timer and try to crash the micro-controller again with source (we should plan to track/observe if a watchdog reset occurs which should match the rate of lockups from step 3 if conditions are the same, but in this case it should self recover)
 +
## look into  remote/boot loader firmware update code & power to try and optimize.
 +
#* Status
 +
#** 2020-10-08 Ordered a [https://www.digikey.com/en/products/detail/iotize/RLINK-STD/9923059 RLINK-STD] STM8 series Debugger, Programmer (In-Circuit/In-System) to program the base firmware.
 +
#*** 2020-10-15 The RLINK has arrived and has been delivered to Ben.
 +
#** 2020-10-16 Adam Hartburger toured EEL118 and began the paperwork to give us a test source.
 +
#** 2020-10-22 [https://logbooks.jlab.org/entry/3855349 Source testing has begun.]
 +
#*** 2020-11-06 More than a week of study have not been able to reproduce the problem.
 +
#** 2020-11-11 [https://logbooks.jlab.org/entry/3857469 No drops observed, source returned.]
 +
#** 2020-12-03 The firmware changes were made (v1.5) and Nick will be testing in the PMT base setup he has in EEL. We were able to load firmware and see the new version reported. Some bench top test shows the watchdog timer works and resets the unit when the CPU is forcefully halted (using a debugger to trash the CPU) state).
 +
 
 +
Now Nick will add new commands to the backend software (to get status and be able to test some various forced reset conditions). Once we see the various features working it will be good to port/test on whatever software  is used in the Hall D environment (and look into remote firmware updates if needed to lower power to make that work).
 +
 
 +
# '''Evaluation of bases in inner rings for Compton events'''
 +
#* 25 bases have already been replaced in this area.  [https://logbooks.jlab.org/entry/3810770 Status of FCAL bases in 3 inner rings] <span style="color:brown">  Colin </span>
 +
#* A possible 47 more could be replaced, we should evaluate which of these are showing any issues. <span style="color:brown">  Mark D. </span>
 +
 
  
  
;Perform preventative maintenance of FCAL PMT bases, which draw excessive current  <span style="color:brown">  -- Support needed from Chris Stanislav, Fernando Barbosa, Mark Dalton, Matt Shepherd </span>
+
; CCAL active bases  
:Diagnose and subsequently replace (reapair) bases which draw excessive current (typically larger than 15 mA), which is an indication of the possible failure of the base. Currently 200 - 300 FCAL bases exhibit such a big current.
+
* Refurbish CCAL active bases: -- <span style="color:brown">  Chris S.</span>
 +
:: - The amplifier is disabled by default. The amplifier circuit if modified (the amplifier can be enabled if needed). The gain is reduced to 3 by modifying resostors R21, R22, and R31. A diode is added to improve linearity
 +
* [https://halldweb.jlab.org/primexd/prep_run_2021/ccal_base_test.pdf Check modified dividers] -- <span style="color:brown">  Sasha.</span>
  
 +
* Install dividers to the CCAL <span style="color:brown">  Sasha, Vlad, Josh.</span>
  
;Refurbish CCAL active bases
 
: - manufacture 150 new active bases according to recent modifications by V. Popov and install them on the CCAL  -- <span style="color:brown">  Chris Stanislav and Fernando Barbosa </span>
 
: - Chris has already prepared a new PCB design for the active base
 
: - ( replace bases: A.Somov, J.Crafts, V. Berdnikov )
 
  
  
 
;Increase LED light pulse on the CCAL LMS system  
 
;Increase LED light pulse on the CCAL LMS system  
 
: - replace (use multiple LEDs) on the LMS system (A.Somov, J.Crafts, V. Berdnikov)
 
: - replace (use multiple LEDs) on the LMS system (A.Somov, J.Crafts, V. Berdnikov)
: - Driver(s) for multiple LEDs <span style="color:brown">  Chris Stanislav and Fernando Barbosa </span>
+
: - Driver(s) for multiple LEDs <span style="color:brown">  Chris S., Nick S., and Fernando B. </span>

Latest revision as of 12:56, 10 September 2021

Below is a list of action items which we would like accomplish before the PrimEx Phase II run in Spring/Summer of 2021 Installation in Hall D

He pipe installed CCAL installed in the platform Beamline shield wall


Preparation for PrimEx run in 2021 (Talk at the collaboration meeting 05/24/2021

CCAL preparation

FCAL preparation status

TAGH preparation

Aligning FCAL in the Hall -- Coordinated by Tim Whitlatch
Moving FCAL platform by about 5 mm in X-direction to make sure that the beam goes through the middle of the FCAL hole (3x3 modules). The exact position of the platform should be confirmed by the alignment group


Preventative maintenance of FCAL PMT bases -- Coordinated by Mark D, with support from Chris S., Nick S., Fernando B., Matt S., Ben R., Paul S.
  1. Change out bases which draw excessive current (> 13 mA) when they are on or which show other signs of problems. High current bases are associated with increased temperature, increased deviation from setpoint and future high voltage failure. There are about 200 high current bases. They are repaired using the standard procedure for cleaning and replacing resistors on the base.
  2. Diagnose bases which draw excessive current (> 13 mA) when they are on and off. This curious symptom is not understood. [1]
    • Status:
      • 8 bases have been remove from the FCAL for special diagnosis. These plus 2 more have arrived with Paul at IU with details of the problem symptoms.
      • Report from Paul. Base 23286 drew 14 mA when powered up. The COMM board by itself draws 14 mA, so the problem is there. It was not immediately obvious what is wrong with the COMM board. The other 7 had current draws > 20 mA when powered up. In all cases the COMM board by itself had most of the excess current draw. In all 7 chip U9 on the COMM board was warm or hot to the touch. This chip drives the test pulser on the SOCK board. If there is a spark at the PMT socket the test pulser transistor is destroyed and U9 is damaged.
  3. Improve robustness of firmware for handling communication failures. When the beam is on we regularly have about 2-3 bases that lose communication per day. The HV drops to zero and the only known way to recover is to power cycle all the bases at the end of the run. This means that we permanently have some holes in the FCAL from perfectly working bases. We will try to modify the firmware to fix this issues. Ben R., Mark D.
    • Plan
    1. compile the existing code and load onto a test unit to make sure we can reproduce what you currently have
    2. have it communicate over the CAN interface like normal
    3. try to crash the micro-controller with a radiation source
    4. make a small firmware change to enable the watchdog timer and try to crash the micro-controller again with source (we should plan to track/observe if a watchdog reset occurs which should match the rate of lockups from step 3 if conditions are the same, but in this case it should self recover)
    5. look into remote/boot loader firmware update code & power to try and optimize.
    • Status
      • 2020-10-08 Ordered a RLINK-STD STM8 series Debugger, Programmer (In-Circuit/In-System) to program the base firmware.
        • 2020-10-15 The RLINK has arrived and has been delivered to Ben.
      • 2020-10-16 Adam Hartburger toured EEL118 and began the paperwork to give us a test source.
      • 2020-10-22 Source testing has begun.
        • 2020-11-06 More than a week of study have not been able to reproduce the problem.
      • 2020-11-11 No drops observed, source returned.
      • 2020-12-03 The firmware changes were made (v1.5) and Nick will be testing in the PMT base setup he has in EEL. We were able to load firmware and see the new version reported. Some bench top test shows the watchdog timer works and resets the unit when the CPU is forcefully halted (using a debugger to trash the CPU) state).

Now Nick will add new commands to the backend software (to get status and be able to test some various forced reset conditions). Once we see the various features working it will be good to port/test on whatever software is used in the Hall D environment (and look into remote firmware updates if needed to lower power to make that work).

  1. Evaluation of bases in inner rings for Compton events
    • 25 bases have already been replaced in this area. Status of FCAL bases in 3 inner rings Colin
    • A possible 47 more could be replaced, we should evaluate which of these are showing any issues. Mark D.


CCAL active bases
  • Refurbish CCAL active bases: -- Chris S.
- The amplifier is disabled by default. The amplifier circuit if modified (the amplifier can be enabled if needed). The gain is reduced to 3 by modifying resostors R21, R22, and R31. A diode is added to improve linearity
  • Install dividers to the CCAL Sasha, Vlad, Josh.


Increase LED light pulse on the CCAL LMS system
- replace (use multiple LEDs) on the LMS system (A.Somov, J.Crafts, V. Berdnikov)
- Driver(s) for multiple LEDs Chris S., Nick S., and Fernando B.