Difference between revisions of "Offline Data Processing"

From GlueXWiki
Jump to: navigation, search
(→‎Run Classification)
Line 1: Line 1:
−
== Triggering ==
+
== Monitoring ==
  
−
* [[Level-1_Trigger | Level-1 Trigger]]
+
* Monitoring jobs will be run on new data as it is written to tape. These jobs will do several things:
−
* [[Level-3_Trigger | Level-3 Trigger]]
+
  
−
== Low-Level Monitoring ==
+
* If calibration / reconstruction-improvement work is ongoing, monitoring jobs will also be rerun on tape data at regular intervals. This may be weekly or bi-weekly, and may run over all of the data for the given run period, or only a subset of it, as the situation demands.  
−
 
+
−
This is performed by running the monitoring plugins for the different detector systems on data as it arrives. The results are viewable in the counting house with RootSpy.  
+
−
https://halldweb1.jlab.org/wiki/index.php/Online_Monitoring_plugins
+
  
 
== Calibrations ==
 
== Calibrations ==
−
 
−
Calibration plugins and scripts are automatically run on the data as it arrives. These scripts are responsible for fitting peaks, checking constants into the calibration database, and iterating as necessary.
 
−
 
−
Talk to David: Make sure calibrations are integrated into scripts that are automatically run
 
−
 
−
== Reconstruction Monitoring ==
 
−
 
−
This is performed by running the monitoring_hists plugin on data after it is calibrated. The results are viewable in the counting house with RootSpy.
 
  
 
== Run Information ==
 
== Run Information ==
  
−
* The run-control database (RCDB) stores information about each run. The RCDB is located at:
+
== Reconstruction ==
  
−
WHERE AM I?
+
REST-format reconstruction data will be saved to tape at JLab. If the disk footprint is small enough, all of the REST reconstruction results will be stored on the /halld/volatile/ disk. If not, then a 2-track skim of the data will be available.
  
−
* During run start, information about the run setup are recorded in the RCDB. This includes run type, start time, trigger type, etc. This is performed by the script:
+
== Run Classification (Good / Junk) ==
  
−
https://halldsvn.jlab.org/repos/trunk/online/daq/scripts/run_go
+
== Physical Skimming ==
  
−
* Since the DAQ crashes sometimes, it cannot be relied on to report run summary information (e.g. run end-time, total # events, scalars, etc.).  Instead, cron jobs on the online system periodically runs a script that crawls through the files and sets this information. The cron job scripts:
+
* Physical, EVIO-format skims will be made available to the calibration on the /halld/volatile/ disk as needed to assist with calibration studies. No other physical skims will be created.  
  
−
https://halldsvn.jlab.org/repos/trunk/online/etc/crontabs/crontab.hdsys.gluonraid1
+
== Logical Skimming ==
−
https://halldsvn.jlab.org/repos/trunk/online/etc/crontabs/crontab.hdsys.gluonraid2
+
  
−
Launch the RCDB update script at:
+
* Logical skims will be stored in EventStore. These will be created by running the
  
−
https://halldsvn.jlab.org/repos/trunk/online/packages/monitoring/src/scripts/update_runinfodb.sh
+
== Analysis Trains ==
  
−
== Run Classification ==
+
=== Daily, Yesterday's-Runs ===
  
−
* During run pre-start, shift takers must indicate what the run type is for the run that is about to start. This run type is stored in the RCDB so that it can be used to help analyze the data. Possible run types include:
+
=== Weekly, Subset-of-all-past-runs ===
  
−
Type -1: Unknown
+
=== Monthly, All-Past-Data ===
−
Type 0: Production Run
+
−
Type 1: DAQ Test
+
−
Type 2: Cosmic Run
+
−
Type 3: Pulser Run
+
−
Type 4: Zero-Field Run
+
−
Type 5: Empty Target Run
+
−
Type 6: Misc. Test
+
  
−
* Further classification is performed by analyzing the rates in the various detector systems. If the rates in any system are lower than some threshold, then
+
== Simulation ==
  
−
* Status Bits by File (Multiple bits may be set at once):
+
=== Channel-by-channel Phase-space ===
  
−
Status -1: Unknown
+
=== bggen ===
−
Status 0: Production Data, Everything OK
+
  
−
Bit 1: Non-Production Data
+
== Acceptance studies ==
−
Bit 2: Low detector rates
+
−
Bit 3: Detector Malfunction
+
−
Bit 4: Flux Unknown
+
−
Bit 5: Polarization Unknown
+

Revision as of 14:41, 20 January 2015

Monitoring

  • Monitoring jobs will be run on new data as it is written to tape. These jobs will do several things:
  • If calibration / reconstruction-improvement work is ongoing, monitoring jobs will also be rerun on tape data at regular intervals. This may be weekly or bi-weekly, and may run over all of the data for the given run period, or only a subset of it, as the situation demands.

Calibrations

Run Information

Reconstruction

REST-format reconstruction data will be saved to tape at JLab. If the disk footprint is small enough, all of the REST reconstruction results will be stored on the /halld/volatile/ disk. If not, then a 2-track skim of the data will be available.

Run Classification (Good / Junk)

Physical Skimming

  • Physical, EVIO-format skims will be made available to the calibration on the /halld/volatile/ disk as needed to assist with calibration studies. No other physical skims will be created.

Logical Skimming

  • Logical skims will be stored in EventStore. These will be created by running the

Analysis Trains

Daily, Yesterday's-Runs

Weekly, Subset-of-all-past-runs

Monthly, All-Past-Data

Simulation

Channel-by-channel Phase-space

bggen

Acceptance studies

Retrieved from "https://halldweb1.jlab.org/wiki/index.php?title=Offline_Data_Processing&oldid=62816"