Offline Monitoring Archived Data

Setup the Software & Environment

1) Update the version xml file to indicate which versions of the software will be used. For hdds & sim-recon, set the names to those of the tags that you will be about to create (in a later step below):

~/builds/version_monitoring_launch.xml

2) Source the environment. It may be tempting, but do NOT put this in the auto-exec-on-login scripts. It is dangerous, in case you need to switch between environments. Just call:

bash
source ~/env_monitoring_launch.sh

3) Updating & building hdds:

cd $HDDS_HOME
git pull                # Get latest software
scons -c install        # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers
scons install -j4       # Rebuild and re-install with 4 threads

4) Updating & building halld_recon:

cd $HALLD_RECON_HOME/src
git pull                # Get latest software
scons -c install        # Clean out the old install: EXTREMELY IMPORTANT for cleaning out stale headers
scons install -j4       # Rebuild and re-install with 4 threads

5) Create a new sqlite file containing the very latest calibration constants. Original documentation on creating sqlite files are here.

cd `dirname $SQLITE_PATH`
$CCDB_HOME/scripts/mysql2sqlite/mysql2sqlite.sh -hhallddb.jlab.org -uccdb_user ccdb | sqlite3 ccdb.sqlite
mv ccdb.sqlite ccdb_monitoring_launch.sqlite

6) Tag the software, where "<type>" below is either "offmon" for offline monitoring launches, or "recon" for full reconstruction launches:

cd $HALLD_HOME/src/
git tag -a <type>-20YY_MM-verVV -m "Used for offline monitoring 20YY-MM verVV"
git push origin <type>-20YY_MM-verVV

cd $HDDS_HOME/
git tag -a <type>-20YY_MM-verVV -m "Used for offline monitoring 20YY-MM verVV"
git push origin <type>-20YY_MM-verVV

7) Checkout (or svn update) the launch scripts if needed:

cd ~/
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/

Starting a new run period

When a new run period is started, it is best to make sure all top-level directories are created with the right permissions. This can save headaches later on when a different gxprojN account is used for offline monitoring.

1) Create top-level directories

/cache/halld/RunPeriod-20YY-MM/recon/
/cache/halld/offline_monitoring/RunPeriod-20YY-MM/
/work/halld2/recon/RunPeriod-20YY-MM/
/work/halld/data_monitoring/RunPeriod-20YY-MM/
/work/halld2/data_monitoring/RunPeriod-20YY-MM/
/group/halld/data_monitoring/run_conditions/RunPeriod-20YY-MM/

2) Make sure other gxprojN users can write in with chmod g+w [dir name] (NB: use jcache chmod for /cache).
Check that permissions match those from previous run periods.

3) ???Deprecated??? Since the publish-to-web scripts are webpage-update scripts, you need to have pre-existing, template versions of a few files for the new run period. It also expects to find comment hooks that include the new run period name, so make sure those are edited in the new template files.

Prepare and Do the Launch

1) Update the appropriate job config file, depending on the type of launch (e.g. jobs_offmon.config for monitoring launches, jobs_recon.config for full reconstruction). Definitely be sure to update RUNPERIOD, VERSION, and BATCH where appropriate (if jobs submitted in batches).

~/monitoring/launch/jobs_offmon.config

2) Update the appropriate jana config file, depending on the type of launch (e.g. jana_offmon.config for monitoring launches, jana_recon.config for full reconstruction). This contains the command line arguments given to JANA. Definitely be sure to update REST:DATAVERSIONSTRING and JANA_CALIB_CONTEXT.

~/monitoring/launch/jana_offmon.config

3) Create the SWIF workflow. The workflow should have a name like "recon_2016-02_ver05" for monitoring launches and "recon_2016-02_ver01_batch01" for full reconstruction launches. It should also match the workflow name in the job config file (e.g. jobs_offmon.config).

swif create -workflow <my_workflow>

4) Backup the software versions & appropriate jana config script, using appropriate names for the launch, e.g.:

cp ~/builds/version_monitoring_launch.xml /group/halld/data_monitoring/run_conditions/RunPeriod-2016-02/version_recon_2016_02_ver01.xml
cp ~/monitoring/launch/jana_recon.config /group/halld/data_monitoring/run_conditions/RunPeriod-2016-02/jana_recon_2016_02_ver01_batch01.config
cp ~/monitoring/launch/jobs_recon.config /group/halld/data_monitoring/run_conditions/RunPeriod-2016-02/jobs_recon_2016_02_ver01_batch01.config

5) Register jobs for the workflow, where <job_config_file> is (e.g.) "~/monitoring/launch/jobs_offmon.config":

~/monitoring/launch/launch.py <job_config_file> <run_min> <run_max>

You can optionally specify specific file numbers to use. For example, to submit jobs for the first 5 files of each run:

~/monitoring/launch/launch.py <job_config_file> <run_min> <run_max> -f '00[0-4]'

6) Run the workflow:

swif run -workflow <my_workflow>