Old Calibration Train

Spring 2016 Processing

Run Groups

SWIF workflow	Run Range	Total # Jobs	Notes	Total Skim Size (TB)	FCAL Skim (TB)	BCAL Skim (TB)	PS Skim (TB)
GlueX-CalibRun-2016-02-25	10457 - 10529		first pass, abandoned
GlueX-CalibRun-2016-02-29	10531 - 10647		first pass, abandoned
GlueX-CalibRun-2016-03-04	10649 - 10724		first pass, abandoned
GlueX-CalibRun-2016-03-18	10331 - 10913	6570	second pass	23	8.1	0.94	14
GlueX-CalibRun-2016-04-04	11048 - 11145	2993	first pass	9.4	4.7	1.0	3.5
GlueX-CalibRun-2016-04-11	11150 - 11312	3573	first pass	9.7	6.2	1.4	2.1
GlueX-CalibRun-2016-04-27	11366 - 11668	10599	first pass	25	16	2.8	5.5
TOTAL				67	35	6	25

Job Structure

Currently 3 "passes" through the data are performed. The first pass tries to do as many calibrations as possible with one file of data. The second pass tries to do calibrations that need a larger data set. The final pass runs through a full run to use the full statistics in a run and to generate outputs for other calibration procedures that can't be done automatically yet (e.g. pi0 calibrations).

The plugins that are run on each pass are:

• Pass 1
  • Step 1 - RF_online
  • Step 2 - HLDetectorTiming (coarse timing+ADC/TDC alignment),TOF_TDC_shift
  • Step 3 - HLDetectorTiming (track-based timing),BCAL_TDC_Timing (timewalks)
• Pass 2
  • Step 1 - TAGH_timewalk,BCAL_attenlength_gainratio,BCAL_TDC_Timing (full)
  • Step 2 - st_tw_corr_auto
• Pass 3
  • HLDetectorTiming,PSC_TW,BCAL_attenlength_gainratio,BCAL_gainmatrix,FCALgains,FCALpedestals,ST_Tresolution,ST_Propagation_Time,p2gamma_hists,imaging,pedestal_online,BCAL_LEDonline,PS_timing,TOF_calib,pi0fcalskim,pi0fcalskim,ps_skim

Output

The output of the calibration jobs can be found at this webpage or at /volatile/halld/home/gxproj3/calib_jobs . There is a subdirectory for each calibration launch. Each launch has a directory for each run that is processed. For each run, several types of outputs are kept

• The results for the processing of each file are kept in a subdirectory with the same number as the file, in the format NNN
• The summed results for each pass through the data are kept in ROOT files with names of the form" hd_calib_passN_RunRRRRRR.root"
• The processed results for each pass (e.g. constants files and figures) are kept in subdirectories with names of the form "passN/"

The relevant directories are:

• ROOT files & calibration constants: /volatile/halld/home/gxproj3/calib_jobs/[workflow]/output
• EVIO skims: /volatile/halld/home/gxproj3/calib_jobs/[workflow]/skims

General Information

Instructions

The software from the calibration train can be obtained from

git clone https://github.com/sdobbs/calibration_train

Job submission is controlled by the following configuration file:

# data.config - example configuration file
jobname = GlueX-CalibRun-2016-04-27    # name of the SWIF workflow
# data/memory sizes in GB
mem_requested = 9                      # max vmem for jobs
disk_space = 30                        # max disk space for jobs
nthreads = 8                           # number of threads requested
# time in hours
time_limit = 36                        # max run time for jobs

# this file is used for debugging
ccdb_table_file = configs/calib_tables

Building a SWIF workflow is controlled by the job_manager.py command

ifarm1102> ./job_manager.py 
usage: job_manager.py init [config_file]
       job_manager.py build [-z] [config_file] [run file]
       job_manager.py run [-L] [config_file]

Some descriptions of the sub-commands:

• init - This takes the configuration file as an input, does the basic setup for the job: creates directories, saves configurations, builds CCDB SQLite file
• build - This takes the configuration file and a file with one run number per line as an input, and creates the SWIF workflow
• run - This just starts the SWIF workflow. Standard SWIF commands can be used to deal with the workflow from here.

Currently, there are two major commands to run

Calibration Challenge

• Calibration Challenge 1

Old Planning

Organization

• The jobs will be submitted every Tuesday at noon, JLab time.
• The jobs will be run from the gxproj3 account [parallel use with EventStore jobs]

• The output of the jobs will be stored in ...

Run Ranges

The following runs will be processed:

RunPeriod-2015-03

• 2931, 3079, 3179, 3180, 3183, 3185

Calibrations

Job Requirements

Each calibration process should include the following:

• Plugin stored in standard location
  • https://halldsvn.jlab.org/repos/trunk/sim-recon/src/plugins/Calibration
• ROOT/other scripts stored in standard location
  • https://halldsvn.jlab.org/repos/trunk/sim-recon/src/scripts/calibrations ?
  • Individual git repos?
• Output to files
• QA routines

What is Being Run

The following plugins are currently being run:

• RF_online (RF signal)
• BCAL_TDC_Timing
• HLDetectorTiming
• PSC_TW

Working on adding:

• BCAL_attenlength_gainratio
• BCAL gains - /work/halld/home/wmcginle/Gain_Calib
• TOF calibrations - https://halldsvn.jlab.org/repos/trunk/home/zihlmann/TOF_calib/