GlueX Software Meeting, March 5, 2019

GlueX Software Meeting
Tuesday, March 5, 2019
3:00 pm EST
JLab: CEBAF Center A110
BlueJeans: 968 592 007

Agenda

1. Announcements
   1. Moving simple_email_list
   2. Another way to provide feedback on wiki pages
2. Review of minutes from the February 5 meeting (all)
3. Report from the February 26 HDGeant4 Meeting (all)
4. Report from the SciComp Meeting on February 28
   1. HOW2019
   2. Switch-over to 100% Slurm should be announced soon.
   3. SciComp is interested in feedback on the Slurm experience.
5. Discussion/proposal for making transition to G4 (Matt)
6. Review of recent issues and pull requests:
   1. halld_recon
      • Issues
      • pull requests
   2. halld_sim
      • Issues
      • pull requests
   3. CCDB
      • Memory Leak issue
7. Review of recent discussion on the GlueX Software Help List (all)
8. Action Item Review (all)

Slides

Talks can be deposited in the directory /group/halld/www/halldweb/html/talks/2019 on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2019/ .

Minutes

Present:

• CMU: Naomi Jarvis
• JLab: Shankar Adhikari, Alexander Austregesilo, Thomas Britton, Eugene Chudakov, Mark Dalton, Sean Dobbs, Stuart Fegan, Mark Ito (chair), David Lawrence, Justin Stevens, Simon Taylor, Beni Zihlmann

There is a recording of this meeting on the BlueJeans site. Use your JLab credentials to access it.

Announcements

1. Moving simple_email_list. The scripts are moving from Subversion to Git. Cron jobs run by the gluex account should use /home/gluex/simple_email_list rather than /group/halld/Software/scripts/simple_email_list to access the lists and the script. (Reported by Mark)
2. Another way to provide feedback on wiki pages. You can use the "discussion" tab present on all wiki pages. The email will be sent to the author(s). (Reported by Mark)
3. Did you submit a Project to MCwrapper-bot that you no longer need? Or does your project have enough data that you don't care about the outstanding data? You can now go to the dashboard and right click on your project row to Recall outstanding jobs from a project. Or Declare the project completed. Or cancel the project. All these come with stipulation of ownership. (Reported by Thomas)

Review of minutes from the February 5 meeting

We went over the minutes.

• Thomas will be giving two talk on MCwrapper one in a general session and one at the OSG user training session.
• David reported that batch 1 of the the Spring 2018 reconstruction has gone through NERSC and batch 2 is in progress. We are limited by the observed 2.5 Gb/s rate of transfer of the raw data out of the Lab. Bryan Hess is looking at why this is less than the 10 Gb/s bandwidth that is advertised. With the additional 9 M units we received from Hall B, we have just enough allocation to finish the run. Recall that we only received about 1/3 of our requested allocation.
• David and Chris Larrieu are looking at using an allocation at the Pittsburgh Supercomputing Center. We will probably try to do a batch on the JLab farm as well.
• Thomas has submitted a test, using XROOTD to stream random trigger data to OSG nodes from a UConn server. We await the results.
• Edgar Fajardo of the OSG has advocated using StashCache to distribute the random trigger data.
• David Lawrence will be speaking at the Machine Learning Lunch tomorrow.
  • From the Slack channel #ml: Reminder that the ML Lunch Series will meet tomorrow (Wednesday, 3/6) in CC F324-25 (assuming it's available) from 12-1PM. David Lawrence will present recent work on "Machine Learning for Particle Tracking". Look forward to seeing everyone!

Report from the February 26 HDGeant4 Meeting

We went over the minutes.

On Alex's ρ analysis, there were some comments:

• Thomas confirmed that the geometry used for HDG3 and HDG4 were the same.
• The start counter problem noted last time was fixed by Richard Jones. Alex re-ran the comparisons and several of the plots look to be in better agreement, however...
• The difference in efficiency as a function of energy persists. Differences remain even if the comparison is done with no trigger simulation cut.
• Sean believes the differences are due to shower energy and timing problems when using HDG4.

Report from the SciComp Meeting on February 28

1. HOW2019 is coming up.
2. Switch-over of the farm to 100% Slurm should be announced soon.
3. SciComp is interested in feedback on the Slurm experience.

Discussion/proposal for making transition to G4

We discussed the proposal Matt Shepherd sent to the Offline Email List. In essence it proposes that the collaboration support HDG3 for data up to including Spring 2017, and support HDG4 for Spring 2018 data and beyond. This will save having to tune both versions to match all of our data sets.

• Mark I. thought that the proposal was reasonable.
• Justin also thought the proposal was reasonable.
• Eugene objected to the mechanism where mcsmear bombs when using smearing constants inconsistent with the version of Geant used.
  • Mark I. thought that bombing has the advantage of preventing the naive user from making mistakes.
  • Thomas pointed out that MCwrapper would have to pass errors of this sort back to the user.
• There was a question about whether the HDDM output from HDG3 and HDG4 records information about which program produced it. No one knew for sure.
• Eugene gave an alternate proposal: he feels strongly that all results, including those that we have completed, and in particular the J/ψ analysis, should be analyzed using both HDG3 and HDG4. We should not go forward and adopt HDG4 as the default until these comparisons with full physics analyses are done and differences understood.
• Justin proposed a compromise via modification of Matt's proposal:
  1. Drop the wording about which versions of HDG3/4 are supported for specific data sets.
  2. Add that going forward HDG4 is the default. Comparative studies between HDG3 and HDG4 are not discouraged and the ability to do such studies should be maintained, but HDG4 is the default choice.
• Mark D. thought that tuning both versions is not really twice the work. Once a system is worked out for one, it can be applied to the other.

In the end Mark I. did not think that we had reached a consensus on any of the proposals. He agreed to come up with compromise wording and send it to the group. Thomas pointed out that we can discuss this further at the HDGeant4 meeting.

[Added in press: after reviewing the discussion carefully, Mark was not able to come up with a proposal about which, in his judgment, a consensus could be formed. There is not enough middle ground.]

---

Review of recent issues and pull requests

• David found and fixed a problem with the use of mutex in the CCAL monitoring code. It will come in as a future pull request.
• Sean suspects there are some deep pathologies in the analysis library, causing memory leaks and code crashes. This came up in the discussion of halld_recon issue #111.
• We noted that Dmitry has acknowledged Sean's report of a memory leak in CCDB. See ccdb issue #61.
• Eugene made a feature request for a variable length list of floats for user-defined uses in the HDDM generated event format. See halld_sim issue #44
• Eugene make a request for development of an C++ interface to the HDDM event format to hide details of handling the HDDM format from the event-generator developer. See halld_sim issue #43.