Difference between revisions of "GlueX Offline Software"

From GlueXWiki
Jump to: navigation, search
(→‎GlueX and Containers)
 
(394 intermediate revisions by 11 users not shown)
Line 1: Line 1:
 +
{| border
 +
|-
 +
|
 +
'''News:'''
 +
* July 19, 2022: [[GlueX_Software_Meeting,_July_18,_2022#Minutes|Minutes of the July 18th Software Meeting]] are available
 +
* July 12, 2022: [https://halldweb.jlab.org/halld_versions/version_5.8.0.xml New version set: 5.8.0].
 +
* June 7, 2022:  [[GlueX_Software_Meeting,_June_6,_2022#Minutes|Minutes of the June 6th Software Meeting]] are available
 +
* May 23, 2022" [https://halldweb.jlab.org/wiki/index.php/GlueX_Tutorial_2022 GlueX Software and Analysis Tutorial 2022]
 +
* May 19, 2022: [https://mailman.jlab.org/pipermail/halld-offline/2022-May/008816.html New version set: 5.7.1].
 +
* [[Offline Software News Archive|Previous news items...]]
 +
|
 +
'''Quick Links:'''
 +
* [https://scicomptest.jlab.org/scicomp/tapeFile Listing of files missing from the tape library]
 +
* [[GlueX Offline FAQ|Frequently Asked Questions]]
 +
* [[Offline HOWTO List|HOWTO List]]
 +
* [[GlueX_Offline_FAQ#Where_do_I_find_version_set_files.3F|Version Set Files]]
 +
* [https://github.com/orgs/JeffersonLab/teams/gluex/repositories Repositories on GitHub]
 +
* [[GlueX Offline Software Meetings]]
 +
|}
 +
 
The legacy version of this page is [[Offline Software|here]].
 
The legacy version of this page is [[Offline Software|here]].
 +
 +
==General Information==
 +
 +
[https://halldweb.jlab.org/docs/build_scripts_web/ Build Scripts: A Version Management System for GlueX] describes a standard directory structure for GlueX software, how to create complete or partial builds, and how to specify versions of individual packages, both for building and for use. It is also available as [https://halldweb.jlab.org/doc-public/DocDB/ShowDocument?docid=2793 GlueX Note 2793].
 +
 +
See below for other helpful links.
 +
 +
===Shell Environment Set-Up===
 +
 +
* [https://halldweb.jlab.org/docs/build_scripts_web/node6.html#SECTION00062400000000000000 Simple Environment Set Up]
 +
 +
===Building GlueX Software===
 +
 +
* [[Building_Private_Versions_of_GlueX_Software:_my_sim-recon.sh|Building private versions of GlueX software at JLab]]
 +
* [https://halldweb.jlab.org/docs/build_scripts_web/node10.html Scripts for Installing GlueX Software]: Do a complete build of all standard components of GlueX software or add new versions to an existing build tree.
 +
* [[Hall D Package Manager|hdpm - Hall D Package Manager]]: A package manager for Hall-D software, which provides an alternative method for installing GlueX software.
 +
* [[Legacy Build Instructions]]: Guides to various aspects of the build process. Up-to-date-ness may vary.
 +
* [[GlueX_Offline_FAQ#Where_do_I_find_version_set_files.3F|Version Sets Files]]
 +
 +
=== GlueX and Containers ===
 +
 +
* [[GlueX Software on Oasis]]
 +
* [https://github.com/JeffersonLab/hd_singularity Package for Building Singularity Containers]
 +
* [[GlueX Containers Meetings]]
 +
* [[HOWTO use the GlueX Singularity Container]]
  
 
==Software Documentation==
 
==Software Documentation==
 +
 +
Packages used in the GlueX software stack with links to package-specific documentation.
  
 
===GlueX Software===
 
===GlueX Software===
−
Documentation on software developed by GlueX collaborators.
 
  
−
====Building GlueX Software====
+
Documentation for software packages specific to GlueX or to JLab.
−
* [[Scripts for Installing GlueX Software]]: Do a complete build of all standard components of GlueX software or add new versions to an existing build tree.
+
* A brief overview of the GlueX-specific software can be found [[ Software | here ]]
−
* [[Legacy Build Instructions]]: Guides to various aspects of the build process. Up-to-date-ness may vary.
+
* Best practices in writing code for the HallD/GlueX software environment check out this link of [[Coding Conventions| '''''HallD/GlueX Coding Conventions''''']]
 +
* A convenient way to enhance the impact of comments inside the code is to use [https://halldweb.jlab.org/talks/2021/usedoxy.pdf '''''Doxygen''''']. The hooks and infrastructures are already implemented for the packages ''halld_recon'' and ''halld_sim''.
 +
* [[Documentation Initiative]]
 +
 
 +
====Software Packages====
 +
 
 +
* [https://github.com/mashephe/AmpTools AmpTools]: Amplitude analysis package (aka partial wave analysis)
 +
* [[Calibration Database|CCDB]]: Calibration Database
 +
** [[Policy on CCDB Variations for Reconstructing Simulated Data]]
 +
* [https://github.com/sdobbs/HDEventStore EventStore]: a package for managing and deploying data files and event lists
 +
* [https://coda.jlab.org/drupal/content/event-io-evio EVIO]: The [https://coda.jlab.org/drupal/ CODA] event format.
 +
* [https://github.com/jeffersonlab/halld_recon halld_recon]
 +
* [https://github.com/jeffersonlab/halld_sim halld_sim]
 +
* [[HDDM Programmer's Interface|HDDM]]: The Hall D Data Model, a compact xml-based format for event-based data.
 +
* HDDS: Detector geometry specification
 +
** [https://halldsvn.jlab.org/repos/trunk/hdds/HDDS-1_1.xsd HDDS Schema]: Includes description of the HDDS mark-up language.
 +
** [https://halldweb.jlab.org/doc-public/DocDB/ShowDocument?docid=64 Geometry Specification for Hall D]: General Description and Philosophy, September 2003
 +
** [https://halldweb.jlab.org/doc-public/DocDB/ShowDocument?docid=654 Detector Models for GlueX Monte Carlo Simulation: A Status Report], June 2006
 +
** [https://halldweb.jlab.org/doc-public/DocDB/ShowDocument?docid=732 Detector Models for GlueX Monte Carlo Simulation: the CD2 Baseline], January 2007
 +
** [[HDDS Tagged Releases]]
 +
* [[HOWTO install and run HDGeant4|HDGeant4]]: Geant4-based simulation of the GlueX Detector and Hall D beamline
 +
* [https://www.jlab.org/JANA/ JANA]
 +
* MCwrapper
 +
**[https://halldweb.jlab.org/gluex_sim/Dashboard.html '''Active Sample Dashboard''']
 +
**[https://halldweb.jlab.org/gluex_sim/SubmitSim.html '''MC Submission Form''']
 +
* [https://pypwa.jlab.org/index.html PyPWA:] [https://pypwa.readthedocs.io/ Python-based Partial Wave Analysis Toolkit]
 +
* [https://github.com/JeffersonLab/rcdb/wiki RCDB]: Run Conditions Database
 +
** [https://halldweb.jlab.org/rcdb/ Web Interface]
 +
** [[GlueX Implementation of the RCDB]]
 +
* sim-recon (deprecated, July 2018)
 +
** [http://www.jlab.org/Hall-D/software/HDSoftware_Documentation/ Doxygen Documentation]: Descriptions of sim-recon C++ classes and their relationships.
 +
** [[Sim-Recon Tagged Releases]]
 +
** [[GlueX_Analysis_Software | Analysis Software in sim-recon]]
 +
** [[SCons Build System]] (SBMS)
 +
 
 +
====Special Topics====
 +
 
 +
* [[Offline HOWTO List]]: Guides to performing various specific tasks.
 +
* [[Beam Simulations]]: Geant simulations of the Hall D Tagger Hall
 +
* [[Magnetic Field Maps for Solenoid]]
 +
* [[Experimental_Sensitivity_to_Solenoidal_Field|Sensitivity to Magnetic Field Strength]]
 +
* [[How HDGeant defines time-zero for physics events]]
 +
* [[Guide to Monte Carlo event timing and detached vertices in HDGeant/4]]
 +
* [[Guide to roll-your-own python hddm transforms]]
 +
* [[Geometry]]
 +
* [[Reconstruction Software]]
 +
* [[Kinematic Fitting]]
 +
# [[Splitoff_Information | Material on Splitoffs]]
 +
 
 +
===Documentation for External Software Packages===
 +
 
 +
Documentation on additional software packages used by GlueX. Development and maintenance of these are not directly related to GlueX.
 +
 
 +
{|
 +
|-
 +
|
 +
* [http://cernlib.web.cern.ch/cernlib/ CERNLIB]
 +
* [http://proj-clhep.web.cern.ch/proj-clhep/ CLHEP]
 +
* [https://en.wikipedia.org/wiki/C%2B%2B C++]
 +
|
 +
* [http://www.stack.nl/~dimitri/doxygen/ Doxygen]
 +
* [https://halldweb.jlab.org/manuals/geant.pdf GEANT 3 (manual)]
 +
* [http://geant4.cern.ch/ GEANT4]
 +
* [https://git-scm.com/ Git]
 +
|
 +
* [http://www.gnu.org/software/make/ GNU Make]
 +
* [http://www.mysql.com/ MySQL]
 +
* [https://www.python.org/ Python]
 +
|
 +
* [http://root.cern.ch/ ROOT]
 +
* [http://www.scons.org/ SCons]
 +
* [http://www.sqlite.org/ SQLite]
 +
|
 +
* [https://subversion.apache.org/ Subversion]
 +
* [http://xerces.apache.org/ Xerces]
 +
* [http://www.w3.org/TR/xmlschema11-1/ XSD]
 +
|}
 +
 
 +
==Data Sets==
 +
 
 +
* [[Simulations]]: a guide to simulated data sets.
 +
 
 +
==Offline Data Monitoring==
 +
 
 +
* [[Data_Monitoring_Procedures]]: Information on offline monitoring of recently-taken data
 +
 
 +
==Computing Facilities==
 +
 
 +
===JLab===
 +
* [https://halldweb.jlab.org/disk_management/halld_status.html Hall D Offline System Status Plots] [[File:User jobs 0 thumb.png|link=https://halldweb.jlab.org/disk_management/halld_status.html]]
 +
* [[Computing Services and Servers|Hall D and JLab Servers]]: Web servers, database servers, etc.
 +
* [https://cc.jlab.org/ Computer Center]
 +
* [https://scicomptest.jlab.org/scicomp/ Scientific Computing]
 +
** [https://scicomp.jlab.org/scicomp/#/auger/jobs Farm Job Status] Auger job status from SciComp
 +
*** [https://halldweb.jlab.org/cgi-bin/jproj_status.pl jproj Job Status] Status of jobs using the jproj system
 +
** [https://scicomp.jlab.org/scicomp/#/jasmine/jobs Tape Request Status]
 +
* Disk space usage
 +
** Group
 +
*** [https://halldweb.jlab.org/disk_management/group_report.html /group/halld leader board]
 +
** [https://scicomp.jlab.org/scicomp/#/disk/cache-family Cache]
 +
*** [https://halldweb.jlab.org/disk_management/cache_oldest.html oldest files under /cache/halld]
 +
** [https://scicomp.jlab.org/scicomp/#/disk/volatile Volatile]
 +
*** [https://halldweb.jlab.org/disk_management/volatile_oldest.html oldest files under /volatile/halld]
 +
** [https://scicomp.jlab.org/scicomp/#/disk/work Work]
 +
*** [https://halldweb.jlab.org/disk_management/work_report.html /work/halld leader board]
 +
* [[GlueX-related shared accounts on the JLab CUE]]
 +
* [[Hall D MySQL/MariaDB Servers and Replication Relationship]]: a list of database servers
 +
 
 +
===Grid===
 +
 
 +
* [http://www.opensciencegrid.org/ Open Science Grid]
 +
** [https://mailman.jlab.org/pipermail/halld-offline/2017-June/002827.html Using the Grid with Containers]
 +
** [[Using the Grid]]
 +
** [[Updating Oasis for the GlueX VO]]
 +
** [https://halldweb.jlab.org/talks/2011-2Q/gridmake-6-2011.ppt Gridmake]
 +
 
 +
===Off-Site Computing Resources===
 +
 
 +
* [[Off-site Computing Resources]]: Status of applications, current and past allocations
  
−
===Other Software Documentation===
 
−
Documentation on software used by GlueX, but developed outside of the collaboration.
 
 
==Software Management==
 
==Software Management==
−
Topics on governance, conventions, standard practices, etc.
+
 
 +
===Source Code Management===
 +
 
 +
* [https://git-scm.com/ Git]
 +
** [https://github.com/orgs/JeffersonLab/teams/gluex/repositories GlueX Git Repositories]
 +
** [[Instructions for Working with GlueX Git Repositories]]. Workflow description.
 +
** [https://mailman.jlab.org/pipermail/halld-offline/2015-July/002086.html Instructions for joining the GlueX team on GitHub]
 +
** [[Git Help Resources]], July 14
 +
** [[Guide to Using Git]]. General usage tips.
 +
** [[GlueX_Offline_FAQ#Git|Git questions and answers in the Offline FAQ]]
 +
** [[Splitting sim-recon]]
 +
** [[Conversion from Subversion to Git]] (Summer 2015)
 +
* [https://subversion.apache.org/ Subversion]
 +
** [http://halldweb.jlab.org/websvn/prod/ Browse the Hall D Subversion Repository]
 +
*** [http://clasweb.jlab.org/websvn/prod/ Legacy version]
 +
 
 +
===Testing and Debugging===
 +
 
 +
* [[Automatic Builds of GlueX Software]]: nightly build, pull-request-initiated test build, scan-build
 +
* [[Automatic Tests of GlueX Software]]
 +
* [[Diagnosing segmentation faults in reconstruction software]]
 +
 
 +
===Governance, Conventions, Standard Practices===
 +
 
 +
* [https://github.com/JeffersonLab/sim-recon/issues Offline Issue Tracking] using GitHub Issue Tracking
 +
* [https://halldweb.jlab.org/mantisbt Offline Issue Tracking] using the Mantis Bug Tracker
 +
* [[Coding Conventions|GlueX Coding Standards]]
 +
* [[Version Management]]
 +
* [[GlueX Cron Jobs]]
 +
* [[Data Challenges]]: Large-scale tests of computing infrastructure
 +
* [https://halldweb.jlab.org/doc-public/DocDB/ShowDocument?docid=2808 Data Management Plan for Hall D]: Long-term preservation of data and data-analysis tools
 +
* [https://data.jlab.org/drupal/?q=system/files/Offline%20Plan%20FY10.pdf JLab Offline Computing Plan, FY2010]
 +
* [[Role of the Software Coordinator]]
 +
 
 +
=== Containers ===
 +
 
 +
* [[GlueX and Containers]]: an overview of the use of containers in GlueX
 +
 
 
==Meetings and Reviews==
 
==Meetings and Reviews==
  
−
# [[Quick Start Guide to building GlueX Software| Using a tagged release to quickly start with Hall-D Software]]
+
* [[GlueX Offline Software Meetings]]: Agendas and minutes
−
# [[Getting started with GlueX Software|Detailed information to get started with all the Hall-D Software]]
+
* [[HDGeant4 Meetings]]
−
# [[Building and Using the GlueX Software|Build and use all Software used in GlueX]]
+
* [[GlueX Containers Meetings]]
−
## [[Quick Start Guide to building GlueX Software]]
+
* [[Software and Computing Reviews]]
−
## [[Build Scripts and Makefiles]]
+
* [[Geometries for 2008 DC Review]]
−
## [[HOWTO set up the GlueX environment]]
+
 
−
## [[HOWTO use a pre-built release]]
+
==Communication and Help==
−
# [[Getting started with Hall-D Software on the JLab CUE|If you are at JLAB get started with Hall-D Software on the CUE cluster]]
+
 
−
# [[GlueX Offline FAQ]]
+
* Offline Software Email List: announcements and discussion
−
# [http://halldweb1.jlab.org/websvn/prod/ Browse the subversion code repository]
+
** [mailto:halld-offline@jlab.org Send email to the list]
−
# [http://clasweb.jlab.org/websvn/prod/ legacy version]
+
** [https://mailman.jlab.org/pipermail/halld-offline/ Message archive]
−
# [[Using the Grid]]
+
** [https://www.google.com/cse/publicurl?cx=001547825138043056762:dydinmymrvu Google search of archive]
−
# [https://halldweb1.jlab.org/talks/2011-2Q/gridmake-6-2011.ppt Gridmake]
+
** [https://mailman.jlab.org/mailman/listinfo/halld-offline Information page] (subscribe, unsubscribe, list members, etc.)
−
# [[Weekly Tests of GlueX Software]]
+
* [https://groups.google.com/forum/#!forum/gluex-software Google Group for GlueX Software]
−
# [https://halldweb1.jlab.org/b1pi/ b1π results]
+
**[[GlueX-Related Google Groups|Help on GlueX-related Google Groups]]  
−
# [https://halldweb1.jlab.org/single_track/ Single track results]
+
** [https://mailman.jlab.org/pipermail/halld-offline/2016-April/002297.html Original announcement]
−
# [[Data Challenges]]
+
* [https://slack.com/ Slack]
−
## [[Data Challenge 2: Ideas]]
+
** Chat application using the workspace '''jlab12gev'''
−
## [[Online Data Challenge 2013]]
+
** Click [https://jlab12gev.slack.com/signup here] to join. You must use your JLab email address.
−
## [[GlueX Data Challenge Meetings]]
+
* [[Email Lists#Simple Email Lists|Simple Email Lists]]: get notifications from automated tests
−
# [[Offline HOWTO List| Offline HOWTO List]]
+
* [[GlueX_Communications|Communication Instructions]]: Guides to communication systems used by GlueX
−
# [[Trigger/DAQ/Monitoring/Controls#Software_HOWTO.27s|This is a link to the ONLINE HOWTO's]]
+
 
−
# [http://www.jlab.org/Hall-D/software/GlueXCodingStandard.html GlueX Coding Conventions]
+
===FAQ===
−
# [[Data_Monitoring_Procedures]]
+
 
−
# [[Media:Data Management Plan Hall D v2.pdf|Data Management Plan]]
+
* [[GlueX Offline FAQ|The GlueX Offline Frequently Asked Questions List]]
−
# [[Beam Simulations]]
+
 
−
# [[Offline Software Topics]]
+
==Legacy Links==
−
## [[Hall-D Software Meetings]]
+
 
−
## [http://www.jlab.org/Hall-D/software/wiki/index.php/GlueX_Communications Communication Instructions]
+
* [[Offline Computing Project Management]]
−
## [[EVO]]
+
* [[Versioning of Calibration Constants]]
−
## [[Role of the Software Coordinator]]  
+
* [[Releases of GlueX Software]]
−
## [[Offline Computing Project Management]]  
+
* [[Reconstruction Tasks and Topics for Further Development]]
−
## [[Versioning of Calibration Constants]]
+
* [[Running jobs on the grid]]
−
## [[Releases of GlueX Software]]
+
* [[Calibration Constants, Tagged Versions]]
−
## [https://data.jlab.org/drupal/?q=system/files/Offline%20Plan%20FY10.pdf JLab Offline Computing Plan, FY2010]
+
* [[Action Items Archive (May 4, 2010 and before)]]
−
## [[Reconstruction Tasks and Topics for Further Development]]
+
* [http://www.jlab.org/Hall-D/offline/Software_tasks.php Software Task List]
−
## [http://www.jlab.org/Hall-D/software/HDSoftware_Documentation Doxygen generated documentation]
+
* [[Tracking resolution estimator (REZEST)]]
−
## [http://clasweb.jlab.org/websvn/prod/ Browse the subversion code repository]
+
* [[ded]]
−
## [[Needed Studies]]
+
−
## [[How HDGeant defines time-zero for physics events]]
+
−
## [[Calibration Constants, Tagged Versions]]
+
−
## [[Experimental_Sensitivity_to_Solenoidal_Field|Sensitivity to Magnetic Field Strength]]
+
−
## [[Sim-Recon Tagged Releases]]
+
−
## [[HDDS Tagged Releases]]
+
−
## [[Nightly Builds of GlueX Software]]
+
−
## [[Weekly Tests of GlueX Software]]
+
−
## [[GlueX shared account on the JLab CUE]]
+
−
## [[GlueX Offline FAQ]]
+
−
## [[Running jobs on the grid]]
+
−
## [[Diagnosing segmentation faults in reconstruction software]]
+
−
## [https://halldweb1.jlab.org/manuals/geant.pdf GEANT 3 manual]
+
−
## [[osrelease.pl script]]
+
−
## [https://halldweb1.jlab.org/mantisbt Offline Issue Tracking]
+
−
##* Legacy page: [[Action Items From Hall-D Software Meetings]]
+
−
## [http://www.jlab.org/Hall-D/offline/Software_tasks.php Software Task List]
+
−
## [[Coding Conventions]]
+
−
## [[JANA/DANA]]
+
−
## [[Simulation Software]]
+
−
## [[Geometry]]
+
−
## [[Magnetic Field Maps for Solenoid]]
+
−
## [http://argus.phys.uregina.ca/cgi-bin/public/DocDB/ShowDocument?docid=473 The Build Management System (BMS)]
+
−
## [[Reconstruction Software]]
+
−
## [[GlueX_Analysis_Software | Analysis Software]]
+
−
## [[Deprecated Software Links]]
+
−
# [http://halldweb1.jlab.org/mantisbt/my_view_page.php Mantis Bug Tracker] <--
+
−
# [[Calibration Database]] (CCDB)
+
−
# [[SCons Build System]] (SBMS)
+
−
# [[Offline Projects]]
+
−
# [[Version Management]]
+
−
# [[Splitoff_Information | Material on Splitoffs]]
+
−
# [[EVO]]
+
−
# [[GlueX Communications]] More on ESNET,EVO ...
+
−
# [[Simple Email Lists]]: get notifications from automated tests
+
−
# [[Computer Farms]]
+
−
# [[Computing Services and Servers]]
+
−
# [[Private:Using VNC]]
+
−
# [[2013 Software Review]]
+
−
# [[2012 Software Review]]
+
−
# [http://argus.phys.uregina.ca/cgi-bin/private/DocDB/ShowDocument?docid=1782 IT in the 12-GeV Era: close-out slides]
+
−
# [[Geometries for 2008 DC Review]]
+
−
# [[Tracking resolution estimator (REZEST)]]
+
−
# [http://clasweb.jlab.org/wiki/index.php/HDDS2GEANT4_Installation_Notes Installing HDDS-to-GEANT4 converter]
+
−
# [http://dustbunny.physics.indiana.edu/mailman/private/gluex/2009-April/002627.html Duties of the GlueX Software Coordinator]
+
−
# [[ded]]
+
−
# [[Private:Members of the halld group at JLab]]
+
−
# [[GlueX Cron Jobs]]
+

Latest revision as of 18:52, 31 January 2024

News:

Quick Links:

The legacy version of this page is here.

General Information

Build Scripts: A Version Management System for GlueX describes a standard directory structure for GlueX software, how to create complete or partial builds, and how to specify versions of individual packages, both for building and for use. It is also available as GlueX Note 2793.

See below for other helpful links.

Shell Environment Set-Up

Building GlueX Software

GlueX and Containers

Software Documentation

Packages used in the GlueX software stack with links to package-specific documentation.

GlueX Software

Documentation for software packages specific to GlueX or to JLab.

  • A brief overview of the GlueX-specific software can be found here
  • Best practices in writing code for the HallD/GlueX software environment check out this link of HallD/GlueX Coding Conventions
  • A convenient way to enhance the impact of comments inside the code is to use Doxygen. The hooks and infrastructures are already implemented for the packages halld_recon and halld_sim.
  • Documentation Initiative

Software Packages

Special Topics

  1. Material on Splitoffs

Documentation for External Software Packages

Documentation on additional software packages used by GlueX. Development and maintenance of these are not directly related to GlueX.

Data Sets

Offline Data Monitoring

Computing Facilities

JLab

Grid

Off-Site Computing Resources

Software Management

Source Code Management

Testing and Debugging

Governance, Conventions, Standard Practices

Containers

Meetings and Reviews

Communication and Help

FAQ

Legacy Links

Retrieved from "https://halldweb1.jlab.org/wiki/index.php?title=GlueX_Offline_Software&oldid=124508"