Run Planning Meeting Notes, November 30 - December 7, 2022

Meetings are via ZoomGov: connect (Meeting ID: 160 199 0060 / Passcode 8394)

MCC whiteboard

Hall D counting room white board

Hall D Logbook

Back to Run Coordination Meetings:Fall2022

Wednesday Dec. 7th

Last 24h

• Tuesday was spent running on empty target. Beam was down between 9:45am to 15:00 due to several accelerator issues. In all, we still got 15.5h of ABU (7am - 7am).

Issues

• Tagger magnetic field:
  • Starting at, 3:45am Tagger NMR lost its lock. Then latter, it lost communication
  • At the same time we lost communication with NMR, we also lost communication with accurate magnet current readout (MTAG5Cdctt), but the coarser one (MTAG5CM) still seems to be alive on EPICS main screen, so we are using this one to monitor the tagger magnet status. Comparison between energy monitorings after Hall D ramp and after the tagger magnet also shows that the tagger field is fine.
• BCAL bad channels

Short term run plan

• Work during Today's RF recovery (from 8am to 4pm):
  • Hall (Rapid Access should be sufficient):
    • DIRC checkout (Justin+1, 30min).
    • Mark Dalton, to change one of the BCal fadc (30min).
    • A. Somov, Vlad, Lubomir, Sergei and Arshak: work in the PS test area (~1h).
    • Mark Steven: walkthrough for OPS.
    • Tim and Eugene will check the manlift (30min)
    • Tim and Sacha will check by how much FCal can be pushed back (for some tentative PrimEx systematic measurements).
  • Tagger:
    • Modules tested by Beni Saturday needs to be screwed back in.
    • Two broken power supplies need to be surveyed and (hopefully) cleared for release by RadCon.
  • Facility wants to get into the tagger to change a light bulb.
• After beam comes back:
  • Harp Scans
  • Run on full target

Tentative longer term run plan

• TAC run?
• Luminosity study?
• Fall 2022 run period ends on Dec. 19th, 7am.

Tuesday Dec. 6th

Last 24h

• Monday was very efficient, with 20h of ABU (7am - 7am), and beamtime spent on production on full target until 16:30 and then run on empty target.

Reports

• Update on electronic support? (Eugene)

Short term run plan

• Work Wednesday during RF recovery (from 8am to 4pm):
  • Hall (Rapid Access should be sufficient):
    • DIRC checkout (Justin+1, 30min).
    • Mark Dalton, to change one of the BCal fadc (30min).
    • A. Somov, Vlad, Lubomir, Sergei and Arshak: work in the PS test area (~1h).
    • Mark Steven: walkthrough for OPS.
    • Tim and Eugene will check the manlift (30min)
    • Tim and Sacha will check bu how much FCal can be pushed back (for some tentative PrimEx systematic measurements).
  • Tagger:
    • Modules tested by Beni Saturday needs to be screwed back in.
    • Two broken power supplies need to be surveyed and (hopefully) cleared for release by RadCon.

Tentative longer term run plan

• TAC run?
• Luminosity study?
• Fall 2022 run period ends on Dec. 19th, 7am.

Monday Dec. 5th

Last 72h

• Friday, Saturday and Sunday were spent mostly with production on full target. 49 of ABU (Sat.7am - Monday 7am).
• Biggest ABU loss occurred Saturday due to a dead hodoscope HV crate. Fixing it took a long time because we have no off-work-hours electronic support (see report below).
• Hall C re-started running Friday around 8pm. Now Hall A, C and D are running concurrently.

Reports

• Friday:~1h Hall Rapid Access between 11am and 12:10pm to change a dying FCal basis (Malte/Mark Dalton) and check the possibly faulty FDC fADC that had been installed Wednesday. It was eventually not replaced (Lubomir). Also, RadCon surveyed some modules that needed to be taken out of the Hall.
• Friday 18:20: Harp scans (first ones since 11 days). Beam profiles looks good. (Hovanes)

Issues

• Friday:timing shift, also Sunday. Sean and Alex Somov looked at it. Who is the Analysis Coordinator when Alex A. is away?
• Friday/Saturday: Beam locks sometime do not work and the beam transmission drops. (Plot for Saturday,~6am. It also happened Friday around 11:50pm) MCC explains that when Hall C is off or ramping up. (I also saw this instance Friday at 1pm when Hall A was ramping up, although at the time, I wondered if it was just a coincidence). The crew chief was made aware of it and found a solution.
• Saturday: At 10:14am, a crate for the hodoscope HV died, compromising 120 chanels on the photon high energy side.
  • The first diagnostic from the Hodoscope main expert was that the lost information was not critical for the experiment, so we continued running for about 30min until Igal informed us that the data was not usable in these conditions.
  • Counting room and RC called Boris, but without success at first. Then we called Hovanes and Sasha for help. Sasha was able to come in around noon and accessed the tagger area. However, Sasha could not reset the crate and diagnosed that it was dead and needed to be swapped. However, the knowledge of good crate spares and expertise to replace crates is with the electronics group, which has no on-call contact. RC called many colleagues (Beni, Eugene, Tim, Hovanes, Lubomir...) to see if they had a useful phone number (Chris Stanislav, Fernando or Chris Cuevas) to call, but none had one. RC emailed Chris Stanislav in case he would check his emails and be able to help/instruct. (Chris replied Sunday morning.)
  • After discussion between Eugene and RC, it was decided that without the problem fixed, data taking would be worthless and we should stop running over the week end (standby shifts) unless we could fix the problem. At 3pm, RC asked for the beam to be stopped and for the Tagger Hall to be surveyed in case a spare is found.
  • Meanwhile, Beni had come to JLab and looked for a spare in the EEL building. He diagnosed that the problem was only with the crate power supplies and had found some spare in the EEL. At 15:45, Beni went in the tagger hall to swap the power supplies. Boris was by then appraised of the problem and went in to help Beni do the replacement.
  • After Beni saved the day (the week-end), we could restart data taking around 17:00.
  • [https://logbooks.jlab.org/entry/4095474 The power for Tagger halo monitors was on the crate that died. The halo monitors thus went off around 10:15am and were restarted only Sunday at 8:30am (this issue was noted only Sunday by the Early Riser shift, and acted upon after RC was informed around 8:30).
• Sunday: gluon03 logged out for some reason and usual hdops pwd did not allowed to log back.
• Sunday: Remote Connection from newer Macs OS
• Sunday: Twice, (around midnight and 7:50am) the trigger rate froze/stopped being sent to EPCIS database. The second time, it was frozen to a typical value so it was hard to spot without a monitoring chart.
• Monday: Beam current instable. MCC was notified by shift leader.

Short term run plan

• Empty target run?
• Wednesday during RF recovery (Control Access should be sufficient):
  • DIRC work.
  • Beam off for more than 1h: rapid access for A. Somov, Vlad, Lubomir, Sergei and Arshak for oportunistic work in the PS test area.

Tentative longer term run plan

• 8h of RF recovery Wednesday Dec. 7th, from 8am to 4pm.
• TAC run?
• Luminosity study?
• Fall 2022 run period ends on Dec. 19th, 7am.

Friday Dec. 2nd

Last 24h

• Thursday was spent mostly with production on full target. It was an efficient day, with 17.8h of ABU (7am-7am).
• A. Somov did a low beam current test of the Active Colim. high gain in anticipation of a future TAC run. It looks good.
• Hall C was supposed to get back to running in mid-swing shift, which would be followed by us doing Harp scans (last ones done: Oct. 21st), but it did not happen (Hall C vacuum issue).

Reports

• FCAL Base lockup reset

Issues

• The FDC fadc installed Wednesday by Naomi and Lubomir needed baseline calibration. Done during swing shift (Sergei/Lubomir).
• cdc p_diff alarms are becoming rarer: 1 (day shift) + 1 (swing shift) + 0 reported in owl shift + 1 early riser.

Short term run plan

• Hall C is expected to be back to running ??at some point??.
• Harp scan after Hall C beam is tuned (last harps done on Nov. 21st, but Transmission is fine.)
• FDC fadc: 20 min raw mode run with beam. Beni will analyse it and determine if the fadc needs to be replaced
• If beam is off, we have opportunistic work:
  • Beam off for more than 1h: rapid access for A. Somov and Vlad for ??
  • Beam off for more than 2h: rapid access for Lubomir or Beni IF FDC fadc needs to be changed
  • Beam off for more than 2h and RadCon is available, need to survey fadc to removed them from the Hall (Lubomir/Codi)

Tentative longer term run plan

• Possible RF recovery Wednesday Dec. 7th.
• TAC run
• Luminosity study

Thursday Dec. 1st

Last 24h

• Wednesday started with production on empty target. It was an efficient day, with 17.1h of ABU (7am-7am).
• 9:10am-11:30 Rapid access to change fadc for FCal and FDC, and fill the target.
• Filled the target, from 10am-1pm. It took 3h to reach nominal density, and the first production runs are done while it was still filling. Target status indicated that it was full&ready by 11:30am.
• Hall A re-started running at 17:48. A consequence is that the Hall D beam energy was quite instable from 6pm (Wed.) to 1am (Thursd.), with total drift of about 14 MeV.

Reports

• PrimEx Analysis progress (Drew)

Issues

• The new FDC fadc is worst than the one replaced (Naomi/Lubomir). The problem is only for raw mode run.
• Many cdc p_diff alarms (e.g. 0.8 alarm/hour during swing and owl shifts) leading to runs often shorter than 2h (this has already been discussed several time. This is just a report).

Run plan

• Hall C is expected to be back to running during swing shift today.
• 20 min test at low beam current/high Active Colim. gain to see if adjustment are needed (A. Somov)?
• Last harp scan was done on Nov. 21st. Should we do a new one? (Transmission is fine.)
• Luminosity study?