Data Monitoring Procedures
Contents
Master List of File / Database / Webpage Locations
Run Conditions
- Online Run-by-run condition files (B-field, current, etc.): /work/halld/online_monitoring/conditions/
- Offline monitoring run conditions (software versions, jana config): /group/halld/data_monitoring/run_conditions/
- Run Info vers. 1
- Run Info vers. 2
- RCDB
Monitoring Output Files
- Run Periods 201Y-MM is for example 2015-03, launch ver verVV is for example ver15
- Online monitoring histograms: /work/halld/online_monitoring/root/
- Offline monitoring histogram ROOT files (merged): /work/halld/data_monitoring/RunPeriod-201Y-MM/verVV/rootfiles
- individual files for each job (ROOT, REST, log, etc.): /volatile/halld/offline_monitoring/RunPeriod-201Y-MM/verVV/
Monitoring Database
- Accessing monitoring database (on ifarm): mysql -u datmon -h hallddb.jlab.org data_monitoring
Monitoring Webpages
SciComp Job Links
Main
Documentation
Job Tracking
Procedures
Saving Online Monitoring Data
The procedure for writing the data out is given in, e.g., Raid-to-Silo Transfer Strategy.
Once the DAQ writes out the data to the raid disk, cron jobs will copy the file to tape, and within ~20 min., we will have access to the file on tape at /mss/halld/$RUN_PERIOD/rawdata/RunXXXXXX.
All online monitoring plugins will be run as data is taken. They will be accessible within the counting house via RootSpy, and for each run and file, a ROOT file containing the histograms will be saved within a subdirectory for each run.
For immediate access to these files, the raid disk files may be accessed directly from the counting house, or the tape files will be available within ~20 min. of the file being written out.
More Procedure Links
Running Over Data As It Comes In
A special user gxproj1 will have a cron job set up to run the plugins as new data appears on /mss. During the week, gxproj1 will submit offline plugin jobs with the same setup as the weekly jobs run the previous Friday. The procedure for this is shown below.
Running the cron job
IMPORTANT: The cron job should not be running while you are manually submitting jobs using the jproj.pl script for the same project, or else you will probably multiply-submit a job.
- Go to the cron job directory:
cd /u/home/gxproj1/halld/monitoring/newruns
- The cron_plugins file is the cronjob that will be executed. During execution, it runs the exec.sh command in the same folder. This command takes two arguments: the project name, and the maximum file number for each run. These fields should be updated in the cron_plugins file before running.
- The exec.sh command updates the job management database table with any data that has arrived on tape since it was last updated, ignoring file numbers greater than the maximum file number. It then submits jobs for these files.
- To start the cron job, run:
crontab cron_plugins
- To check whether the cron job is running, do
crontab -l
- To remove the cron job do
crontab -r
Data Versions
To document the conditions of the monitoring data that is created, for the sake of reproducability and further analysis we save several pieces of information. The format is intended to be comprehensive enough to document not just monitoring data, but versions of raw and reconstructed data, so that this database table can be used for the event database as well.
We store one record per pass through one run period, with the following structure:
Field | Description |
---|---|
data_type | The level of data we are processing. For the purposes of monitoring, "rawdata" is the online monitoring, "recon" is the offline monitoring |
run_period | The run period of the data |
revision | An integer specifying which pass through the run period this data corresponds to |
software_version | The name of the XML file that specifies the different software versions used |
jana_config | The name of the text file that specifies which JANA options were passed to the reconstruction program |
ccdb_context | The value of JANA_CALIB_CONTEXT, which specifies the version of calibration constants that were used |
production_time | The data at which monitoring/reconstruction began |
dataVersionString | A convenient string for identifying this version of the data |
An example file used as as input to ./register_new_version.py is:
data_type = recon run_period = RunPeriod-2014-10 revision = 1 software_version = soft_comm_2014_11_06.xml jana_config = jana_rawdata_comm_2014_11_06.conf ccdb_context = calibtime=2014-11-10 production_time = 2014-11-10 dataVersionString = recon_RunPeriod-2014-10_20141110_ver01