Difference between revisions of "Goals for Spring 2014 Data Challenge"

From GlueXWiki
Jump to: navigation, search
m
m
Line 6: Line 6:
  
 
The primary goal of the Dec 2013 Online Data Challenge is to test the entire Trigger/DAQ/monitoring chain from the front-end
 
The primary goal of the Dec 2013 Online Data Challenge is to test the entire Trigger/DAQ/monitoring chain from the front-end
−
ROC's to the tape silo using the production computing and networking systems being installed in Oct.
+
ROC's to the tape silo using production computing and networking systems at relatively low rates using simulated event data.
  
−
We will use the full TS/TD/SD/TI system unless the trigger fibers are not ready, in which case polling on the TI's will be used instead.  Triggers will come from a random pulser in the TS (note that SSP's and GTP's will not be used).
+
We will use the full TS/TD/SD/TI system unless the trigger fibers are not ready, in which case polling on the TI's will be used instead.  Triggers will come from a random pulser in the TS, i.e. the CTP/SSP/GTP system will not be used.  
−
CODA3 will be used to collect data from the ROCs (as many as are available, minimum 12), build them into full events in two stages, and pass them to the event recorder.  
+
CODA3 will be used to collect data from the ROCs (as many as are available, minimum 12), build them into full events in two stages, and pass them to the event recorder via the L3 farm.  
−
A preliminary version of the farm manager CODA component will manage L3 and monitoring farm processes.  The production run control facility will be used to manage the DAQ system.
+
A preliminary version of the farm manager CODA component will manage L3 and monitoring farm processes.  The CODA3 run control facility will be used to manage the DAQ system.
−
We will further use production RAID-to-silo transfer mechanism, initiated manually since the CC has not written the automated scripting system yet.
+
We will further use the production RAID-to-silo transfer mechanism, initiated manually since the CC has not developed an automated system yet.
−
Various L3 rejection algorithms, including no rejection, will be tested.
+
Various L3 rejection algorithms, including no rejection, will be employed.  RootSPY will be used to aggregate monitoring histograms, and all detectors will be monitored.
 +
 
 +
Input to the ROCs will come from simulated data converted to EVIO format via the MC2CODA package.  The front-end ROC's will get the simulated data from EVIO files and forward the relevant
 +
data to CODA as if the ROC's had read the data out of the front-end digitizer boards.  Note that only a subset of all ROCS will participate so the remaining simulated event data will be added at a
 +
separate stage downstream from the ROC's.
  
−
Input to the ROCs will come from simulated data converted to EVIO format via the MC2CODA package.  The front-end ROC's will get the simulated readout data from EVIO files and forward the
 
−
data to CODA as if the ROC's had read the data out of the front-end digitizer boards.  Note that only a subset of all ROCS will participate so the remaining simulated event data will be added at a separate stage downstream from the ROC's.
 
  
 
(to be worked on...)
 
(to be worked on...)
  
 
The goals of the Dec 2013 ODC are:
 
The goals of the Dec 2013 ODC are:
 +
 +
# Test DAQ stream from ROCS to silo:
 +
#* generate simulated data with L1 rejection applied, create simulated EVIO data files via MC2CODA package
 +
#* read MC2CODA data into each ROC, forward relevant information to event builders as if the data had been read out of the front end modules
 +
#* build events in two stages
 +
#* add remaining ROC data after final event event builder stage
 +
#* send data to L3 farm, evaluate various L3 rejection algorithms including no rejection
 +
#* forward accepted data to ER to write to RAID storage disks
 +
#* transfer data files to silo using production mechanism involving dedicated tape unit.
 +
 +
 +
# Test complete RootSpy detector monitoring system:
 +
#* implement plugins for all detectors and for L3 monitoring
 +
#* deploy L3 and monitoring farm processes, ET systems and ET transfer facilities
 +
#* use RootSpy to collect, aggregate and display histograms on multiple monitors in the counting house
 +
#* compare histograms to archived reference set
 +
#* archive histograms at the end of each run
 +
#* automatically produce web pages showing L3 and monitoring histograms
 +
 +
 +
# Test farm manager:
 +
#* implement codaObject communications in JANA-based L3 farm process
 +
#* develop farm manager CODA component
 +
#* use farm manager to start/stop/monitor L3 farm processes at run start/stop
 +
#* cripple farm so it fails to meet minimum requirements, ensure farm manager takes appropriate actions
  
  
−
#Test the RootSpy system in the counting house environment:
 
−
#* run monitoring processes on multiple nodes that read events from a common ET system and create root histograms
 
−
#* view summed monitoring histograms on at least 4 separate monitors in the counting room simultaneously
 
−
#* create archive (ROOT file) of summed monitoring histograms
 
−
#* read in and compare archived histograms via overlay with "live" histograms
 
−
#:
 
 
# Test data rates of EVIO formatted raw data files:
 
# Test data rates of EVIO formatted raw data files:
 
#* use bggen generated data files that have been passed through a L1 event filter and converted into EVIO format that represent what is expected for the real data.
 
#* use bggen generated data files that have been passed through a L1 event filter and converted into EVIO format that represent what is expected for the real data.

Revision as of 17:04, 8 October 2013

Goals for the Dec GlueX online Data Challenge 2013

8-Oct-2013 E. Wolin


The primary goal of the Dec 2013 Online Data Challenge is to test the entire Trigger/DAQ/monitoring chain from the front-end ROC's to the tape silo using production computing and networking systems at relatively low rates using simulated event data.

We will use the full TS/TD/SD/TI system unless the trigger fibers are not ready, in which case polling on the TI's will be used instead. Triggers will come from a random pulser in the TS, i.e. the CTP/SSP/GTP system will not be used. CODA3 will be used to collect data from the ROCs (as many as are available, minimum 12), build them into full events in two stages, and pass them to the event recorder via the L3 farm. A preliminary version of the farm manager CODA component will manage L3 and monitoring farm processes. The CODA3 run control facility will be used to manage the DAQ system. We will further use the production RAID-to-silo transfer mechanism, initiated manually since the CC has not developed an automated system yet. Various L3 rejection algorithms, including no rejection, will be employed. RootSPY will be used to aggregate monitoring histograms, and all detectors will be monitored.

Input to the ROCs will come from simulated data converted to EVIO format via the MC2CODA package. The front-end ROC's will get the simulated data from EVIO files and forward the relevant data to CODA as if the ROC's had read the data out of the front-end digitizer boards. Note that only a subset of all ROCS will participate so the remaining simulated event data will be added at a separate stage downstream from the ROC's.


(to be worked on...)

The goals of the Dec 2013 ODC are:

  1. Test DAQ stream from ROCS to silo:
    • generate simulated data with L1 rejection applied, create simulated EVIO data files via MC2CODA package
    • read MC2CODA data into each ROC, forward relevant information to event builders as if the data had been read out of the front end modules
    • build events in two stages
    • add remaining ROC data after final event event builder stage
    • send data to L3 farm, evaluate various L3 rejection algorithms including no rejection
    • forward accepted data to ER to write to RAID storage disks
    • transfer data files to silo using production mechanism involving dedicated tape unit.


  1. Test complete RootSpy detector monitoring system:
    • implement plugins for all detectors and for L3 monitoring
    • deploy L3 and monitoring farm processes, ET systems and ET transfer facilities
    • use RootSpy to collect, aggregate and display histograms on multiple monitors in the counting house
    • compare histograms to archived reference set
    • archive histograms at the end of each run
    • automatically produce web pages showing L3 and monitoring histograms


  1. Test farm manager:
    • implement codaObject communications in JANA-based L3 farm process
    • develop farm manager CODA component
    • use farm manager to start/stop/monitor L3 farm processes at run start/stop
    • cripple farm so it fails to meet minimum requirements, ensure farm manager takes appropriate actions


  1. Test data rates of EVIO formatted raw data files:
    • use bggen generated data files that have been passed through a L1 event filter and converted into EVIO format that represent what is expected for the real data.
    • read events from a file and place them into ET system on a node designated as the Event Builder (EB) node.
    • read events from EB node's ET system into L3 client processes running on farm nodes, write them to Event Recorder node (ER) running on the RAID disk.
    • transfer events from ET note's ET system to ET system on remote monitoring server, serve to remote monitoring nodes.
  2. Test prototype L3 rejection algorithm:
    • run rejection algorithm in farm nodes, mark accepted events based on algorithm results, reject some events.
    • create and view histograms that display L3 rejection results.
  3. Test data rate to tape silo:
    • transport files written to RAID disk to tape silo while simultaneously recording data on the RAID disk.
  4. Monitor health of farm and DAQ system nodes (counting house only) via farm monitoring software 9such as Ganglia).




Additional Goals

These goals will depend on the success of the goals listed above and the state of the software development at the time of the ODC.

  1. Test deferred processing system for L3:
    • Investigate different monitoring/L3 architectures
    • Multiple nodes pulling events from EB node and writing to local files
    • Separate process on L3 nodes detect files, read them in and write ~10% of them to ER node where they are aggregated into single a file
Retrieved from "https://halldweb1.jlab.org/wiki/index.php?title=Goals_for_Spring_2014_Data_Challenge&oldid=51721"