Difference between revisions of "GlueX Software Meeting, May 28, 2019"

From GlueXWiki
Jump to: navigation, search
(add minutes)
 
(4 intermediate revisions by one other user not shown)
Line 14: Line 14:
 
# Review of [[GlueX Software Meeting, April 30, 2019#Minutes|minutes from the last Software Meeting]] (all)
 
# Review of [[GlueX Software Meeting, April 30, 2019#Minutes|minutes from the last Software Meeting]] (all)
 
# [[HDGeant4 Meeting, May 21, 2019#Minutes|Report from the last HDGeant4 Meeting]] (all)
 
# [[HDGeant4 Meeting, May 21, 2019#Minutes|Report from the last HDGeant4 Meeting]] (all)
# Discussion of splitting ANALYSIS library from halld_recon (all)
+
# [https://mailman.jlab.org/pipermail/halld-offline/2019-May/003628.html Discussion of splitting analysis libraries from halld_recon (all)]
 +
#* Goal: maintain ability to analyze REST data produced with fixed halld_recon version using an updated version of "analysis" libraries currently in halld_recon.
 +
#* Proposal: extract ANALYSIS, KINFITTER, and PID libraries (and required plugins and programs) to a separate repo halld_ana. 
 
# Review of recent issues and pull requests:
 
# Review of recent issues and pull requests:
 
## halld_recon
 
## halld_recon
Line 34: Line 36:
  
 
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2019</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2019/ .
 
Talks can be deposited in the directory <code>/group/halld/www/halldweb/html/talks/2019</code> on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2019/ .
 +
 +
== Minutes ==
 +
 +
Present:
 +
 +
* ''' CMU: ''' Naomi Jarvis
 +
* ''' FIU: ''' Mahmoud Kamel, Joerg Reinhold
 +
* ''' JLab: ''' Shankar Adhikari, Alexander Austregesilo, Thomas Britton, Mark Dalton, Sean Dobbs, Mark Ito (chair), David Lawrence, Justin Stevens, Simon Taylor, Beni Zihlmann
 +
 +
 +
There is a [https://bluejeans.com/s/QUAf6/ recording of this meeting] on the BlueJeans site. Use your JLab credentials to access it.
 +
 +
=== Announcements ===
 +
 +
# [https://mailman.jlab.org/pipermail/halld-offline/2019-May/003635.html New version set consistent with recon-2018_08, version 00], as announced May 8.
 +
# [https://mailman.jlab.org/pipermail/halld-offline/2019-May/003642.html MCwrapper v2.2.0 released]. MCwrapper can now sense viable version sets. The new release uses XROOTD to stream random triggers. Those files are now kept on the work disk. Version 2.2.1 is due out soon. The robot will now submit to the JLab farm when it senses that the OSG is full.
 +
# [https://mailman.jlab.org/pipermail/jlab-scicomp-briefs/2019q2/000203.html PBS to Slurm on June 3rd]. This morning SciComp had a meeting on this. Beni reminded us that either SWIF or Auger can be used with the new system. Changes under the new system should be transparent to users, except for the fact when choosing a custom directory for standard output and error files, that directory must be made in advance under slurm.
 +
#* Added in press: during the meeting, SciComp sent out [httpd://halldweb.jlab.org/talks/2019/Slurm-rev2.pdf slides from this morning's presentation].
 +
# [https://mailman.jlab.org/pipermail/halld-offline/2019-May/003652.html A script for backups of disk files]. Mark described the new script. He has been using it to clean-up recent congestion on the work disk.
 +
 +
=== Discussion of splitting analysis libraries from halld_recon ===
 +
 +
Justin referred us to [https://mailman.jlab.org/pipermail/halld-offline/2019-May/003628.html a recent discussion on the offline email list] and introduced the topic:
 +
* '''Goal''': maintain ability to analyze REST data produced with fixed halld_recon version using an updated version of "analysis" libraries currently in halld_recon.
 +
* '''Proposal''': extract ANALYSIS, KINFITTER, and PID libraries (and required plugins and programs) to a separate repo halld_ana. 
 +
 +
Some notes on the discussion:
 +
 +
* Fundamental problem: analysis launch data is created from REST data which was produced with a known version of halld_recon. The analysis launch itself may be run with a different version of halld_recon to take advantage of changes made since the reconstruction launch. The problem is best illustrated in the case of Monte Carlo. There, to maintain fidelity with real data, one should do the reconstruction of the simulated data with the reconstruction-launch version of halld_recon, and do analysis with the analysis-launch version of halld_recon. We have no mechanism to insure this practice and most users will not be aware that this is even an issue. The current version set system has no mechanism to provide two different versions of a package (halld_recon in this case), and even if it did the user must know which hd_root is the right one for reconstruction and which is the right one for analysis. On top of this, it is possible to do reconstruction and analysis both starting from raw (or simulated raw) data. In that the case "wrong" version of reconstruction or analysis will be done by construction.
 +
* The solution we have been discussing before is the one Justin proposed: break out some of the libraries in a separate package that can be version controlled separately. Then there is a natural distinction between reconstruction versions and and analysis versions based on the tagged version of analysis libraries used. David quickly recognized that there is fundamental difficulty in this scheme. Currently the analysis libraries depend on code in the other parts of halld_recon, and other other parts of halld_recon depend on the analysis libraries. Since there is no dependency hierarchy in the code, they cannot be separated into independent packages, i.e., neither one can be built first (chicken and egg problem).
 +
* Beni pointed out that if it were the case that analysis could be done only on REST data, then the necessary dependency hierarchy would obtain and the analysis packages could be separated. That case could be enforced by policy. That would break the current feature where reconstruction and analysis can be done in the same invocation of hd_root. People were reluctant to give up this feature.
 +
* David thought that the chicken and egg problem could be solved, but it would require careful separation of routines into reconstruction libraries, analysis libraries, and common libraries. Such a clean separation with the roster of routines in the current libraries may not be possible.
 +
* Justin suggested having two version of halld_recon in the version sets (version.xml files), one for reconstruction and one for analysis.
 +
* David commented that a system where the analysis libraries were plug-ins (selected at run-time) that supersede statically linked versions of the analysis libraries is possible.
 +
* Thomas pointed out that it is possible to modify MCwrapper to switch from one version set, used for reconstruction, to another, used for analysis, in a single job. This does not solve the problem of which reconstruction version goes with which analysis launch, and it does no good at all for folks not using MCwrapper.
 +
* Another possibility for MCwrapper jobs is to copy the analysis-compatible version of hd_root along with the job on the OSG or elsewhere.
 +
* In order to save on disk space with a possibly exploding number of builds needed (depending on the solution we settle on) Sean suggested making stripped down version of halld_recon that exclude the plug-ins and other appropriately chosen files.
 +
* It was noted that having to deal with two complete version sets is confusing; an independent package solution is easier to understand.
 +
* A method for maintaining the correct correlation of reconstruction versions and analysis versions should be a part of any solution.
 +
* Alex pointed out the the name of the analysis launch from which ROOT trees were generated is encoded in each tree file.
 +
* We did not arrive at a solution. Mark encouraged us to think about one.
 +
 +
=== Review of the previous Software and HDGeant4 meetings ===
 +
 +
We went over the [[GlueX Software Meeting, April 30, 2019#Minutes|minutes from the April 30 Software meeting]] and [[HDGeant4 Meeting, May 21, 2019#Minutes|those from the May 21 HDGeant4 meeting]] without significant comment.
 +
 +
=== Review of recent issues and pull requests ===
 +
 +
Two issues were reviewed:
 +
 +
[https://github.com/JeffersonLab/halld_recon/issues/71 halld_recon: crash in ST matching #71]. This problem is still causing a huge fraction of monitoring launch jobs to crash. Work continues on this.
 +
 +
[https://github.com/JeffersonLab/halld_recon/issues/166 halld_recon: Vertex z dependent reconstruction efficiency #166]. Alex recently reported that the vertex position for &rho; events shows structure as a function of z. The effect is seen in HDGeant, HDGeant4, and real data. This has not been understood yet.
 +
 +
=== Review of recent discussion on the GlueX Software Help List ===
 +
 +
We went over [https://groups.google.com/forum/#!forum/gluex-software message on the help list].
 +
 +
[https://groups.google.com/forum/#!topic/gluex-software/WuKlWUsEydo Matt and Mark's convo on gluex_install] went private at some point, but in the end the problem was resolved.

Latest revision as of 12:26, 31 May 2019

GlueX Software Meeting
Tuesday, May 28, 2019
3:00 pm EDT
JLab: CEBAF Center A110
BlueJeans: 968 592 007

Agenda

  1. Announcements
    1. New version set consistent with recon-2018_08, version 00
    2. MCwrapper v2.2.0 released
    3. PBS to Slurm on June 3rd
    4. A script for backups of disk files
  2. Review of minutes from the last Software Meeting (all)
  3. Report from the last HDGeant4 Meeting (all)
  4. Discussion of splitting analysis libraries from halld_recon (all)
    • Goal: maintain ability to analyze REST data produced with fixed halld_recon version using an updated version of "analysis" libraries currently in halld_recon.
    • Proposal: extract ANALYSIS, KINFITTER, and PID libraries (and required plugins and programs) to a separate repo halld_ana.
  5. Review of recent issues and pull requests:
    1. halld_recon
    2. halld_sim
    3. CCDB
    4. RCDB
  6. Review of recent discussion on the GlueX Software Help List (all)
  7. Action Item Review (all)

Slides

Talks can be deposited in the directory /group/halld/www/halldweb/html/talks/2019 on the JLab CUE. This directory is accessible from the web at https://halldweb.jlab.org/talks/2019/ .

Minutes

Present:

  • CMU: Naomi Jarvis
  • FIU: Mahmoud Kamel, Joerg Reinhold
  • JLab: Shankar Adhikari, Alexander Austregesilo, Thomas Britton, Mark Dalton, Sean Dobbs, Mark Ito (chair), David Lawrence, Justin Stevens, Simon Taylor, Beni Zihlmann


There is a recording of this meeting on the BlueJeans site. Use your JLab credentials to access it.

Announcements

  1. New version set consistent with recon-2018_08, version 00, as announced May 8.
  2. MCwrapper v2.2.0 released. MCwrapper can now sense viable version sets. The new release uses XROOTD to stream random triggers. Those files are now kept on the work disk. Version 2.2.1 is due out soon. The robot will now submit to the JLab farm when it senses that the OSG is full.
  3. PBS to Slurm on June 3rd. This morning SciComp had a meeting on this. Beni reminded us that either SWIF or Auger can be used with the new system. Changes under the new system should be transparent to users, except for the fact when choosing a custom directory for standard output and error files, that directory must be made in advance under slurm.
    • Added in press: during the meeting, SciComp sent out [httpd://halldweb.jlab.org/talks/2019/Slurm-rev2.pdf slides from this morning's presentation].
  4. A script for backups of disk files. Mark described the new script. He has been using it to clean-up recent congestion on the work disk.

Discussion of splitting analysis libraries from halld_recon

Justin referred us to a recent discussion on the offline email list and introduced the topic:

  • Goal: maintain ability to analyze REST data produced with fixed halld_recon version using an updated version of "analysis" libraries currently in halld_recon.
  • Proposal: extract ANALYSIS, KINFITTER, and PID libraries (and required plugins and programs) to a separate repo halld_ana.

Some notes on the discussion:

  • Fundamental problem: analysis launch data is created from REST data which was produced with a known version of halld_recon. The analysis launch itself may be run with a different version of halld_recon to take advantage of changes made since the reconstruction launch. The problem is best illustrated in the case of Monte Carlo. There, to maintain fidelity with real data, one should do the reconstruction of the simulated data with the reconstruction-launch version of halld_recon, and do analysis with the analysis-launch version of halld_recon. We have no mechanism to insure this practice and most users will not be aware that this is even an issue. The current version set system has no mechanism to provide two different versions of a package (halld_recon in this case), and even if it did the user must know which hd_root is the right one for reconstruction and which is the right one for analysis. On top of this, it is possible to do reconstruction and analysis both starting from raw (or simulated raw) data. In that the case "wrong" version of reconstruction or analysis will be done by construction.
  • The solution we have been discussing before is the one Justin proposed: break out some of the libraries in a separate package that can be version controlled separately. Then there is a natural distinction between reconstruction versions and and analysis versions based on the tagged version of analysis libraries used. David quickly recognized that there is fundamental difficulty in this scheme. Currently the analysis libraries depend on code in the other parts of halld_recon, and other other parts of halld_recon depend on the analysis libraries. Since there is no dependency hierarchy in the code, they cannot be separated into independent packages, i.e., neither one can be built first (chicken and egg problem).
  • Beni pointed out that if it were the case that analysis could be done only on REST data, then the necessary dependency hierarchy would obtain and the analysis packages could be separated. That case could be enforced by policy. That would break the current feature where reconstruction and analysis can be done in the same invocation of hd_root. People were reluctant to give up this feature.
  • David thought that the chicken and egg problem could be solved, but it would require careful separation of routines into reconstruction libraries, analysis libraries, and common libraries. Such a clean separation with the roster of routines in the current libraries may not be possible.
  • Justin suggested having two version of halld_recon in the version sets (version.xml files), one for reconstruction and one for analysis.
  • David commented that a system where the analysis libraries were plug-ins (selected at run-time) that supersede statically linked versions of the analysis libraries is possible.
  • Thomas pointed out that it is possible to modify MCwrapper to switch from one version set, used for reconstruction, to another, used for analysis, in a single job. This does not solve the problem of which reconstruction version goes with which analysis launch, and it does no good at all for folks not using MCwrapper.
  • Another possibility for MCwrapper jobs is to copy the analysis-compatible version of hd_root along with the job on the OSG or elsewhere.
  • In order to save on disk space with a possibly exploding number of builds needed (depending on the solution we settle on) Sean suggested making stripped down version of halld_recon that exclude the plug-ins and other appropriately chosen files.
  • It was noted that having to deal with two complete version sets is confusing; an independent package solution is easier to understand.
  • A method for maintaining the correct correlation of reconstruction versions and analysis versions should be a part of any solution.
  • Alex pointed out the the name of the analysis launch from which ROOT trees were generated is encoded in each tree file.
  • We did not arrive at a solution. Mark encouraged us to think about one.

Review of the previous Software and HDGeant4 meetings

We went over the minutes from the April 30 Software meeting and those from the May 21 HDGeant4 meeting without significant comment.

Review of recent issues and pull requests

Two issues were reviewed:

halld_recon: crash in ST matching #71. This problem is still causing a huge fraction of monitoring launch jobs to crash. Work continues on this.

halld_recon: Vertex z dependent reconstruction efficiency #166. Alex recently reported that the vertex position for ρ events shows structure as a function of z. The effect is seen in HDGeant, HDGeant4, and real data. This has not been understood yet.

Review of recent discussion on the GlueX Software Help List

We went over message on the help list.

Matt and Mark's convo on gluex_install went private at some point, but in the end the problem was resolved.