Difference between revisions of "Offline Data Processing"

From GlueXWiki
Jump to: navigation, search
(→‎Run Information)
(→‎Monitoring)
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
== Triggering ==
+
== To Include ==
  
−
* [[Level-1_Trigger | Level-1 Trigger]]
+
- Eventstore, grid, university sites, calibration validation, reconstruction validation, skim validation
−
* [[Level-3_Trigger | Level-3 Trigger]]
+
  
−
== Low-Level Monitoring ==
+
- Code versioning, config scripts, software version restoration, database access (web, batch scripts, JANA)
  
−
This is performed by running the monitoring plugins for the different detector systems on data as it arrives. The results are viewable in the counting house with RootSpy.
+
- Documentation, backwards compatibility
−
https://halldweb1.jlab.org/wiki/index.php/Online_Monitoring_plugins
+
 
 +
== Monitoring ==
 +
 
 +
* Monitoring jobs will be run on new data as it is written to tape. These jobs will do several things:
 +
 
 +
* If calibration / reconstruction-improvement work is ongoing, monitoring jobs will also be rerun on tape data at regular intervals. This may be weekly or bi-weekly, and may run over all of the data for the given run period, or only a subset of it, as the situation demands.
  
 
== Calibrations ==
 
== Calibrations ==
  
−
Calibration plugins and scripts are automatically run on the data as it arrives. These scripts are responsible for fitting peaks, checking constants into the calibration database, and iterating as necessary.
+
== Run Information ==
  
−
Talk to David: Make sure calibrations are integrated into scripts that are automatically run
+
== Reconstruction ==
  
−
== Reconstruction Monitoring ==
+
REST-format reconstruction data will be saved to tape at JLab. If the disk footprint is small enough, all of the REST reconstruction results will be stored on the /halld/volatile/ disk. If not, then a 2-track skim of the data will be available.
  
−
This is performed by running the monitoring_hists plugin on data after it is calibrated. The results are viewable in the counting house with RootSpy.
+
== Run Classification (Good / Junk) ==
  
−
== Run Information ==
+
== Physical Skimming ==
  
−
* The run-control database (RCDB) stores information about each run. The RCDB is located at:
+
* Physical, EVIO-format skims will be made available to the calibration on the /halld/volatile/ disk as needed to assist with calibration studies. No other physical skims will be created.  
  
−
WHERE AM I?
+
== Logical Skimming ==
  
−
* During run start, information about the run setup are recorded in the RCDB. This includes run type, start time, trigger type, etc. This is performed by the script:
+
* Logical skims will be stored in EventStore. These will be created by running the
  
−
https://halldsvn.jlab.org/repos/trunk/online/daq/scripts/run_go
+
== Analysis Trains ==
  
−
* Since the DAQ crashes sometimes, it cannot be relied on to report run summary information (e.g. run end-time, total # events, scalars, etc.).  Instead, cron jobs on the online system periodically runs a script that crawls through the files and sets this information.  The cron job scripts:
+
=== Daily, Yesterday's-Runs ===
  
−
https://halldsvn.jlab.org/repos/trunk/online/etc/crontabs/crontab.hdsys.gluonraid1
+
=== Weekly, Subset-of-all-past-runs ===
−
https://halldsvn.jlab.org/repos/trunk/online/etc/crontabs/crontab.hdsys.gluonraid2
+
  
−
Launch the RCDB update script at:
+
=== Monthly, All-Past-Data ===
  
−
https://halldsvn.jlab.org/repos/trunk/online/packages/monitoring/src/scripts/update_runinfodb.sh
+
== Simulation ==
  
−
== Run Classification ==
+
=== Channel-by-channel Phase-space ===
  
−
Talk to David: Shift-takers recording if production data or not
+
=== bggen ===
  
−
talk to sergey: list of info needed at beginning and end of run
+
== Acceptance studies ==

Latest revision as of 17:55, 20 January 2015

To Include

- Eventstore, grid, university sites, calibration validation, reconstruction validation, skim validation

- Code versioning, config scripts, software version restoration, database access (web, batch scripts, JANA)

- Documentation, backwards compatibility

Monitoring

  • Monitoring jobs will be run on new data as it is written to tape. These jobs will do several things:
  • If calibration / reconstruction-improvement work is ongoing, monitoring jobs will also be rerun on tape data at regular intervals. This may be weekly or bi-weekly, and may run over all of the data for the given run period, or only a subset of it, as the situation demands.

Calibrations

Run Information

Reconstruction

REST-format reconstruction data will be saved to tape at JLab. If the disk footprint is small enough, all of the REST reconstruction results will be stored on the /halld/volatile/ disk. If not, then a 2-track skim of the data will be available.

Run Classification (Good / Junk)

Physical Skimming

  • Physical, EVIO-format skims will be made available to the calibration on the /halld/volatile/ disk as needed to assist with calibration studies. No other physical skims will be created.

Logical Skimming

  • Logical skims will be stored in EventStore. These will be created by running the

Analysis Trains

Daily, Yesterday's-Runs

Weekly, Subset-of-all-past-runs

Monthly, All-Past-Data

Simulation

Channel-by-channel Phase-space

bggen

Acceptance studies

Retrieved from "https://halldweb1.jlab.org/wiki/index.php?title=Offline_Data_Processing&oldid=62823"