Data Monitoring Procedures

Master List of File / Database / Webpage Locations

Run Conditions

• Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/
• Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/
• Run Info vers. 1
• Run Info vers. 2
• RCDB

Monitoring Output Files

• Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15
• Online monitoring histograms: /work/halld/online_monitoring/root/
• Offline monitoring histogram ROOT files (merged): /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles
• individual files for each job (ROOT, REST, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/

Monitoring Database

• Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring

Monitoring Webpages

• Help
• Plot Browser
• Run Browser
• Version Browser
• Time Series
• Launch Analysis
• Recon Tests

SciComp Job Links

Main

• Scientific Computing Home Page
• Auger Job Status Page
• JasMine Tape Job Status Page

Documentation

• Batch System
• Mass Storage System
• Write-Through Cache
• SWIF
• SWIF Command Line

Job Tracking

• Completed Job History
• Job Stats By Project
• Job Stats By Track
• Cluster Report
• Walltime Distribution

Procedures: Overview

Online Monitoring: During Experimental Running

After every run is finished, a ROOT file containing histograms from the online monitoring system and a file containing some run conditions are copied to directories under /work/halld/online_monitoring . A cronjob running in the counting house performs this function.

This ROOT file is processed similarly to the offline monitoring results, and are made available under the same webpages as "ver00" of the relevant run period.

For more details on the online monitoring system, see this page.

Offline Monitoring and Reconstruction: During Experimental Running

During experimental running, the following offline monitoring procedures should be performed, each with a different gxprojN account, so that they don't interfere with each other:

1. Incoming: Monitor the first 5 files of each newly-recorded run as soon as it hits the tape.
2. Monitoring Launches: Every two weeks, do a monitoring launch over the first 5 files of all runs currently available on the tape.
3. Initial Reconstruction Launch: As soon as a new group (e.g. ~100 runs) of data is initially semi-well calibrated, do a preliminary full reconstruction launch over all files in that group.
   • We can add user analysis plugins to this launch, including those with ROOT TTree output, provided that they work and don't take much memory.

Note that the monitoring is limited to the first 5 files of each run, because data is being recorded to tape at a faster rate than the monitoring can keep up with. Also, during the experimental run, each run will only be fully-reconstructed once, because it will be difficult enough to keep up with the incoming data.

Offline Monitoring and Reconstruction: After Experimental Running

After experimental running, the following offline monitoring procedures should be performed, each with a different gxprojN account, so that they don't interfere with each other:

1. Monitoring Launches: Every two weeks, do a monitoring launch over the first 5 files of all runs currently available on the tape.
2. Initial Reconstruction Launch: As soon as a new group (e.g. ~100 runs) of data is initially semi-well calibrated, do a preliminary full-reconstruction launch over all files in that group.
   • We can add user analysis plugins to this launch, including those with ROOT TTree output, provided that they work and don't take much memory.
3. Further Reconstruction Launches: Every ~3 months, if there have been significant improvements to the reconstruction / calibrations, do a new full-reconstruction launch over all of the data.
   • We can add user analysis plugins to this launch, including those with ROOT TTree output, provided that they work and don't take much memory.

Note that the monitoring is limited to the first 5 files of each run, since there will be a significant amount of data.

Saving to Tape (Write-through Cache): Monitoring Launches

All job output will be directly written to the write-thru cache. However, only the following will be saved to tape:

• REST files: All files.
• ROOT files: One merged file per run.
  • After merge, the individual files are deleted (so they won't be saved).
• Job stdout/stderr: One tarball per run
  • After launch analysis, the log files are deleted (so they won't be saved).
• Browser png's: One tarball per launch

Saving to Tape (Write-through Cache): Full Reconstruction Launches

• REST files: All files.
• ROOT files: All files, AND one merged file per run.
• Job stdout/stderr: One tarball per run
  • After launch analysis, a tarball is created and the individual log files are deleted (so they won't be saved).
• Browser png's: One tarball per launch

Procedures: Details

• Offline Monitoring: Running Over Incoming Data
• Offline Monitoring: Running Over Archived Data
• Offline Monitoring: Post-Processing
• DEPRECATED (Except plots): Offline Monitoring: Running Over Archived Data
• DSelector SWIF Jobs
• Analysis Launch: Merging Trees

On- and Offline Monitoring Data Validation

• Offline Monitoring: Data Validation
• Offline Monitoring: Data Validation of PrimEx data
• Offline Monitoring: Data Validation of CPP data
• Online Monitoring: Data Validation

Software Tests

• Software Test: Experimental Data Reconstruction
  • Test Results