Run Planning Meeting Notes, June 30 - July 06, 2022
Meeting are via ZoomGov: connect (Meeting ID: 160 199 0060 / Passcode 8394)
- 1 July 6
- 2 Tuesday July 5
- 3 Monday July 4
- 4 Sunday July 3
- 5 Saturday July 2
- 6 Friday July 1
- 7 Thursday June 30
Past 24 hours
- production data taking continued on Pb
- serious DAQ issues early afternoon. Sergey worked hard to fix. Started initially with rocFDC11 busy causing DAQ to hang. after necessary reboot of roctrig1 the trig_client process on gluon100 stopped communicating with roctrig1 so no trigger rate data went to EPICS anymore. see logbook
- life time variation are still an issue at times.
- Richard tested TAGM results from HV scan. New Thresholds for TAGM tested ok and loaded
- quite smoth data taking over night.
- new timing calibrations in the TAGGER look good but there are some "bumps/shoulders/backgrounds"
- after beam went away, start ramping the solenoid down for cyro compressor switch.
- Sean found that the TOF RF signal is mssing since run 100477: see logbook Our check procedures are clearly not vigilant enough anymore. RF signals are NOT in online monitoring anymore and Hydra is not doing its job as intended.
- switching back from AMO to Diamond the cross hair on the camera does not match the position with the Diamond. However beam seems to be in the right spot as there is a coeherent peak in the tagger microscope with the expected rates as well as the DAQ rates make sese. see lobook
- maintenance day: beam will go away 7:30am, 8 hours maintenance, followed by beam restoration and high current beam tuning for HallC
- please access hall after 9am: other halls have priority
- ramp solenoid to 800A for cyro comprssor swtch
- FCAL base replacement
- Tagger NMR inspection
- Fix TOF RF signal
Tuesday July 5
Past 24 hours
- established operating field of solenoid just shortly before 12 noon. First regular production run on Pb 100862
- blast from the past, TDC timing shifts by 32ns this time in a BCAL slot
- tagger micr. RF shift by 2ns in run 100864, in fact both tagh and tagm are off w.r.t RF by about 2ns: see here.
- Beam profiler removed from beam
- owl shift is very successful taking data.
- there are some indications that the location of the coherent peak is not that stable. during a phone call with HallD we observed the coherent edge move from a too high location to the nominal location over the course of about 30 seconds. This correspond, according to the shift leader, to the HallC beam coming on. Hmmmm. star date around 04 Jul 2022 06:21:46 PM EDT
- beam shifts/drifts can be significant
- occasional life time drops in the DAQ is still happening.
- some very noisy CDC channels: see logbook entry 4011141
- TDB: continue taking production data on Pb target.
- MCC discusses plans for Machine improvements. Current idea is to have a "maintenance" day tomorrow, while doing RF recovery the injector wien power supplies will be exchanged as the y-component seems to be a source of a 60Hz intensity modulation seen in the beam at higher currents at the level of 10%.
- Tomorrow Wednesday 7:30 beam is taken away for MAINTENANCE DAY estimate 8h + beam restoration.
Monday July 4
after switching back to Pb "full" target data taking till the thunderstorm passing though knocked out our power supply and and the solenoid ramped down. changing plans and switch to strait track running.
Past 24 hours
- beam was trippy but ok and we were taking data on Pb
- at 3:23pm a thunderstorm knocked out our solenoid: solenoid trip, cooling water glitch
- took action to prepare for strait track running over night with solenoid field off
- solenoid power supply and magnet all trips reset and is ready for ramp of if need be. seed solenoid status after dump
- prepare for strait track running, take some raw mode data and then move to 50M triggers strait tracks
- start ramping magnet back up. at 5:30am reach 800A and decided to take beam back on diamond radiator to continue data taking.
- Solenoid at nominal field at 11:52am
- Beam Size and Stability while the beam properties seem to have changed after beam came back on June 30th as seen by the AC, the Beam profiler was put into the beam for the past 24 hours to obtain a clear relation ship between AC asymmetry measurements and Beam Profiles beam positions in x/y [mm]. Beam Profiler Result
- solenoid ramping
- DAQ lifetime drops and fluctuations are still a reoccurring thing. It is usually accompanied with fluctuating input buffer size.
- once solenoid is at full field continue with production data taking on lead target.
Sunday July 3
Spent 3 shifts on empty target data taking. Rocky start with 3h per setting gradually improved to 2 hours per setting to 1.5h per setting. Over the past 24 hours no beam at all at 45.4%. Beam to the Halls in the past 36 hours does not look bad but the beam is very trippy.
Past 24 hours
- take data on PB till 10am when beam was off and decision was taken to move to empty target run
- taking empty target data: goal 1.5B events total PARA,PERP and Amo
- data taking on empty ended 3:22am, Ilya came in and changed target back to lead. Around 3:55am continue with nominal production running on lead.
- MCC reports that they encounter again some issues with cavity zone 1L10-5. This is the main source of trips in the past preventing high current to hallC.
- Monitoring plot Run 100825 last run before switch to empty
- Monitoring plot Run 100830 during empty target data taking
- Monitoring plot Run 100844 after empty target, Pb back in beam.
- Hydra does not seem to display plots: logbook entry. This seems to be related to root_spy not running correctly, it shows the processes up and running but no events being actually processed. Restarting the process seem to solve the issue but it seems this restart does not work on any arbitrary gluon machine. For example the first attempt of restart on gluon48 directly did not work but doing the same action on gluon05 did work.
- However, Hydra seems to not display on gluon machines in the counting house while on a "private" laptop accessing from outside Jlab it works.
- DAQ life time drops are still happening. rate fluctuations in grafana, owl shift
- taking production data on lead.
Saturday July 2
Yesterday: No beam during the day. Beam back after 4pm, taking data over night.
Past 24 hours
- beam taken away by MCC around 10am for beam studies and high current development to hallC
- Access to hall by Tim to move Pb target slightly off center to position frame in beam. Taking data in this configuration was abandoned since there will be no beam still swing shift. Ilya went back to the hall and put the Pb target back into nominal position
- postpone the target "off-position" test to a later time with more beam reliability.
- around 4:20pm beam is back, first do harp scans and the proceed with production data taking on Pb target.
- taking data was generally ok with the intermittent issue of beam trips and persistent DAQ buffer issues.
- shift crew uses "Buffer Depth" indicator of the DAQ to take action on stopping/restarting a run to battle lifetime issues.
- DAQ life time during owl shift is generally good
- early riser shift continues with data taking. The DAQ seems to behave somewhat but still hicks up form time to time.
- Hall Beam Currents over night
- The accidental scaling factor for run 100767 is around 1.08 which is somewhat higher than during regular GlueX running but comparable to PrimeX running.
- Naomi identified some CDC noisy channels requiring some threshold change.
- Request to any target movement/change: Report the change in the logbook with a picture of the "flag"-target-indicator and beam line
- the NMR probe for the TAGGER magnet is operational again providing reasonable data.
- start taking empty target data this afternoon: one full sequence (PARA,PARA,PERP,PERP,AMO+) + (PARA,PERP) to about 1.5B triggers total
- QUESTION: Do we really want 60nA e-beam current or do we want to be more conservative with 50nA? The issue: rates in the tagger.
Friday July 1
Overall a similar day as in the past: no beam during the day with some data taking over night. Generally the beam looked more stable but still lots of trips when HallC beam is on. (Beam in HallD and C).
Past 24 hours
- start out with no beam. Beam lost due to 1L15. whole day shift needed to replace "bracker" for zone 1L15
- after accelerator repair work, efforts are taken to establish beam in the machine and try to increase beam current for HallC
- Bobby and Ilya entered to hall to investigate the target moving mechanism.
- Justin rebooted crate U1-1-top to reestablish BCAL LED trigger
- the magnetic field measurement of the tagger magnet is still not working. See earlier report
- beam back after 10pm started doing harp scan first: Harp Scan Results
- shift crew was not able to write halld scan pictures into logbook: see owl shift summary
- take first data with Amo radiator on goniometer and then production with diamond. pretty much very time a runs was restarted the life time was bad and the DAQ required full restart.
- the beam position on the AC may be more stable in then in the past: requires closer look
- the DAQ liftime keeps posing issues: DAQ lifetime monitoring
- from Epics Overview all looks ok, the rates in the very low tagger energy counters are low, but the scale of the high rates counters dominates the plot. But maybe the rates in those counter may in fact be dominated by scraping? -> needs closer look see also report from Hovanes on harp scans
- Beam positions at AC seem to be more stable and variations smaller than in the past: beam positions in halld
- low invariant mass +/- pair tracks are indeed leptons: E/p vs Mass (and mostly electron/positrons!)
- Monitoring proposal for CPP signal: 
- target position mechanism and small variations:
- BCAL Gain studies over the current run period by M.D. For most sensors these look reasonable. Some outlier due to hardware changes.
- tagger NMR probe not operational,TAGGER NMR. The magnet itself is controlled by MCC
- DAQ issues with low life time seems to happen more frequently affecting almost every stop/start run action. The lifetime drop seems to happen mostly when starting a new run, not so much during a run (my interpretation from the logbook).
- Two pre-amp cards in the CDC may have developed more noise
- maximum 2 hour beam test with target in "wrong" position to assess the issue of different DAQ/count rates after the TAC run period
- TDB longer term schedule for the whole of the long week end
- take 12 hours (calendar time) "empty" target data nested between "full" target data taking over the course of the long week end
Thursday June 30
Past 24 hours
Overarching theme of the past 24 hours: "herding cats"
- no beam during the day, machine optimization to deliver high current beam to HallC (unsuccessful)
- beam back around 5:30pm
- did first two Harp scans with HallD laser on and off while hallC had around 17uA of beam
- general beam conditions are rather unstable. beam strip chart
- continue data taking over night
- DAQ issues again with life time dropping to 60%. Shift crew called Sergey.
- found problem with Slot 3 fiber 7 having trouble (I think this is seen in roctrig1?)
- after Sergey's intervention DAQ came slowly back to "normal"
- beam conditions are very unstable over night
- Loss of NMR comm in Tagger Manet
- again very bad life time 10% only! (stopped run 100778) but also beam lost beam at the same time.
- stopping and restarting the DAQ with new run 100779 cured the issue of low lifetime, but soon beam was lost again.
- beam on and off for some time, finally off with problems in several cavities one whole (1L15) zone no beam for several hours.
- HallD beam conditions(e,gamma): quality/stability
- Tuned beam and CW beam to hallD looks different ELOG entry
- HallC beam high current not reached, current reached 30uA
- MCC: Machine needs to replace "breaker" in 1L15
- last time this work took about 6 hours
- NL and SL to be taken down
- take this opportunity to assess relocation of some BLM in the machine
- expect beam no earlier then swing shift.
- CPP/NPP: TBD
- first do harp scans when beam comes back
- do first an Amo run using goniometer Al radiator if that does not work due to ion chamber try target ladder Al radiator.
- continue nominal production data taking on diamond.