Merging Analysis Trees

Analysis Launch: Merging Analysis Trees

The large number of root trees produced during the analysis launch (number of plugins x number of input files, e.g. 60 x 40k = 2.4M) with very small individual sizes (<1MB) is not compatible with the requirements of the tape library. Therefore, we write the trees to a temporary disk, merge all files for each run and put only the merged trees on tape. We do this with a cron job during the launch to minimize the occupation of the temporary disk space.

Preparing the software

• Do the exact same steps as detailed for the offline monitoring and reconstruction setup BUT make sure:

1) that OUTDIR_LARGE is not the write-through cache

2) that there is enough space on the temporary disk area

Launching the merging for a new analysis launch

1) Download the "monitoring" scripts directory from svn. For the gxprojN accounts, use the directory ~/monitoring/:

cd ~/
svn co https://halldsvn.jlab.org/repos/trunk/scripts/monitoring/
cd monitoring/merge_trees

2) Update the jobs_merge.config job config file. RUNPERIOD, VERSION, BATCH and WORKFLOW have to match the running analysis launch.

3) Create the SWIF workflow. The workflow must have the same name as the analysis lauch, appended by "_merge".

swif create -workflow <my_workflow>

4) In ~/monitoring/incoming/cron_exec.csh, modify the MAILTO email address used in case of problems.

~/monitoring/incoming/cron_exec.csh

5) Before launching the cron job, manually run the script first. This is just in case there are already a lot of files on disk, and it takes longer than 15 minutes to run the first execution. In this case, jobs may be double-submitted! So, first execute the python script manually.

python ~/monitoring/merge_trees/merge_trees.py ~/monitoring/merge_trees/jobs_merge.config

6) Check if the cron demon is running on that node:

ps aux | grep crond

7) Now that the initial batch of jobs have been submitted, launch the cron job by running:

crontab cron_merge

8) To check whether the cron job is running (on the same machine you launched the cron job, i.e. for CentOS7: ifarm1401 or ifarm1402), do

crontab -l

9) The stdout & stderr from the cronjob are piped to a log file located at:

~/merge_trees.log

10) To remove the cron job (e.g. at the end of the run) do

crontab -r