Micro Data Challenge, July 30, 2012

Status

• Two "projects"
  1. bggen: generate, simulate, and smear HDGeant events, write events to tape
     • surrogate for data taking with GlueX
  2. hd_root: read events from bggen project from tape, do reconstruction, write root trees to tape
     • surrogate for reconstruction, producing DSTs
• 1000 events per file
• all work on JLab batch farm
• all actions initiated and recorded by jproj.pl (perl script)

ifarm1102:marki:792> mysql -hhalldweb1 -ufarmer farming
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 225647
Server version: 5.1.36-log MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> select count(file), sum(submitted), sum(output), sum(jput_submitted), sum(silo) from bggen;
+-------------+----------------+-------------+---------------------+-----------+
| count(file) | sum(submitted) | sum(output) | sum(jput_submitted) | sum(silo) |
+-------------+----------------+-------------+---------------------+-----------+
|        1000 |           1000 |         436 |                 436 |       211 | 
+-------------+----------------+-------------+---------------------+-----------+
1 row in set (0.01 sec)

mysql> select count(file), sum(submitted), sum(output), sum(jput_submitted), sum(silo) from hd_root;
+-------------+----------------+-------------+---------------------+-----------+
| count(file) | sum(submitted) | sum(output) | sum(jput_submitted) | sum(silo) |
+-------------+----------------+-------------+---------------------+-----------+
|         196 |            196 |           8 |                   8 |         0 | 
+-------------+----------------+-------------+---------------------+-----------+
1 row in set (0.00 sec)

mysql> quit
Bye

Notes

• output written to new tape volume set: /mss/halld/halld-scratch
• random number seeds varied for event generation (bggen)

Early lessons learned

• JLab farm batch priority scheme
  • opaque
  • wrong-headed
  • meeting scheduled for tomorrow with SciComp
• database schema improvements:
  • multiple output files
  • capture of AUGER job ID
• script improvements:
  • configuration file for input/output/tape locations

Scope Expansion

• create output in REST format
• longer runs
• analysis system for results