Online Monitoring plugins

From GlueXWiki
Revision as of 23:34, 23 September 2014 by Davidl (Talk | contribs)

Jump to: navigation, search


The Online Monitoring Plugins are used to create histograms that monitor the health of the data in an online environment. The goal is to ensure the quality of the data as it is being read in and to identify any issues quickly so that they can be corrected. The system is deigned using the JANA framework and the plugins will have access to the entire set of sim-recon reconstruction libraries.

The system is made up of a set of plugins allowing individual detector groups to maintain control over the monitoring of their systems. Each detector will have one or more monitoring plugins to generate histograms and, in some cases, provide macros for complex displays of those histograms. The plugins for all detectors will be attached to a single multi-threaded process to optimize efficiency. A small farm of monitoring computer nodes will be used, each processing a portion of the incoming data stream. The events will be distributed to the nodes using the same ET software system that the DAQ system in Hall-D uses.

The system is designed to allow the plugins to be developed completely in an offline environment and then used for online monitoring without any special coding requirements. This is due to RootSpy's ability to read histogram definitions from global memory and publish them to the network. A central program will grab all histograms from all nodes and automatically sum them. A set of initial plugins exists that detector groups should modify. If a new plugin is needed, one can make it using instructions found here.

Getting Started

To develop a plugin for use in the online, there a few steps you will want to follow:

1.) Check out and build the sim-recon code (find instructions here)

2.) Simulate some bggen events to produce an hdgeant_smeared.hddm file (find instructions here)

3.) Convert the HDDM file into EVIO format (find instructions here)

4.) Checkout the online monitoring software package and SBMS system:

> mkdir ONLINE
> svn co
> svn co

5.) Build the plugin of interest. e.g.:

> cd monitoring/plugins/src/BCAL_online
> scons -u install

6.) Set your environment to find the plugin and run it along with the DAQ and TTab plugins. e.g.:

> cd ONLINE/monitoring/$BMS_OSNAME
> hd_root -PPLUGINS=DAQ,TTab,BCAL_online rawevent_000002.evio

Histograms will be written to the file hd_root.root

Coupling to RootSpy

The RootSpy system will access and publish ROOT histograms by simply adding the rootspy plugin to the list of plugins. For this system to work though, one needs a cMsg server running.

Starting the cMsg server

First note that if you are running on the gluon cluster in the Hall-D counting house then a cMsg server should already be running on the gluondb1 node. Just set your ROOTSPY_UDL environment variable to cMsg://gluondb1/cMsg/rootspy

If you are trying to develop code on your own system, you will need to install cMsg. After you have done this, set your CMSGROOT environment variable and then you can run the java server this way:

java -cp $CMSGROOT/jar/cMsg-*.jar org/jlab/coda/cMsg/cMsgDomain/server/cMsgNameServer

The location of the server is specified in the ROOTSPY_UDL environment variable. Thus, you can run it on one node and have the RootSpy servers and clients run on different nodes. Here is how to set it if running the RootSpy processes on the same node as the cMsg server.

> setenv ROOTSPY_UDL cMsg://localhost/cMsg/rootspy


Several options exist for testing the system with the RootSpy GUI. These are:

  1. Use rootspy_filepub to publish the histograms and trees from an existing ROOT file
  2. Use the rootspy plugin to publish histograms as they are being filled from a DANA program reading events from a file
  3. Use ET to distribute events to DANA processes using the rootspy plugin

Most likely you will want to use option 2.

Publish histograms using rootspy_filepub

The rootspy_filepub program comes with RootSpy and can be used to publish histograms and trees from an existing ROOT file to the RootSpy system. This does not support macros so only single histograms can be displayed. Use like this:

> rootspy_filepub file.root

then fire up RootSpy and hit the "Select" button to see the histograms.

View histograms being filled from a data file

Perhaps the most useful way of testing plugins and macros with RootSpy is to use an EVIO file and tell the DAQ plugin to continually re-open the input file and read events from it. This will allow the filling of the histograms to continue forever while you operate the RootSpy GUI program. Here is an example of how to do this:

> hd_ana rawevent_000002.evio -PEVIO:LOOP_FOREVER=1 -PPLUGINS=DAQ,TTab,rootspy,CDC_online,BCAL_online

You can also use HDDM files here, but there is no "LOOP_FOREVER" option so once the events run out, the server program will quit and RootSpy won't be able to show anything. You will also not have access to the lower level objects such as DBCALDigiHit if using HDDM.

Fill histograms from ET system

This is probably overkill for most people and not needed to setup a complete development system for online monitoring plugins. However, for the sake of completeness ...

You need to install ET and have your ETROOT environment variable pointing to it. The DAQ plugin mush be compiled with ETROOT set, otherwise it is built automatically without ET support.

1.) Start up ET server

> $ETROOT/bin/et_start -f /tmp/et_sys_test -s 21000 -n 10

2.) In another window feed events from an EVIO file into the ET system using the file2et program that is part of the Hall-D "etuils" package. You probably already have the packages directory checked out if you've gotten to this point so just go to the etutils directory and build the program with scons

> cd /path/to/online/packages/etutils
> scons -u install ginstall

Then run the program like this:

> file2et -loop -f /tmp/et_sys_test rawevent_000002.evio

The "-loop" option tells the program to repeatedly re-open the file and read events from it so that events flow forever allowing one to more easily focus on testing the plugins.

To read events from the ET system into a DANA program:

> hd_ana -PPLUGINS=DAQ,TTab,BCAL_online ET:/tmp/et_sys_test:MON

Specifying Macros for RootSpy to plot

RootSpy can plot single histograms easily, but to make more sophisticated plots with things like divided canvases or overlays, one needs a macro. The RootSpy system supports this in allowing server programs (i.e. online monitoring plugins) to provide macros using the same communication channel that it uses for providing histograms and trees. What's more, the build system will automatically compile any macro files it finds in the monitoring plugin's source directory into the plugin itself. For example, in the CDC_online plugin source directory, there is a macro file named "CDC_occupancy.C". This will be made into a string and compiled into the plugin automatically by the build system. Any file with a ".C" suffix will automatically be processed in this way.

The one thing that cannot be easily automated is for RootSpy to know what histograms a macro actually needs. To address this, special comments can be added to the macro to indicate which histograms RootSpy should request so that the macro will have them available. Here is an example from the BCAL_online plugin:


// hnamepath: /bcal/bcal_shower_x
// hnamepath: /bcal/bcal_shower_y
// hnamepath: /bcal/bcal_shower_z   
// hnamepath: /bcal/bcal_shower_t		

	TDirectory *dir = (TDirectory*)gDirectory->FindObjectAny("bcal");
	if(dir) dir->cd();

	TH1I *h_x = (TH1I*)gDirectory->FindObjectAny("bcal_shower_x");
	TH1I *h_y = (TH1I*)gDirectory->FindObjectAny("bcal_shower_y");
	TH1I *h_z = (TH1I*)gDirectory->FindObjectAny("bcal_shower_z");
	TH1I *h_t = (TH1I*)gDirectory->FindObjectAny("bcal_shower_t");
	// Just for testing
	if(gPad == NULL){
		TCanvas *c1 = new TCanvas("c1");
	if(!gPad) return;
	TCanvas *c1 = gPad->GetCanvas();
	c1->Divide(2, 2);
	if(h_x) h_x->Draw();

	if(h_y) h_y->Draw();

	if(h_z) h_z->Draw();

	if(h_t) h_t->Draw();

In this example, the histograms of interest are in the "bcal" directory inside of ROOT's internal directory structure. The lines starting with // hnamepath: are the special comments that tell RootSpy which histograms to request. For example /bcal/bcal_shower_x means to request the histogram named bcal_shower_x from the directory named bcal. The directories can be any depth. When the macro is run, it will be in a directory when the full directory structure is reproduced below it. Histograms will be automatically summed from ones obtained from all servers.

Note that because the histograms may not all be obtained by the time a macro first runs, it should be careful to check pointers before using as is done in the above example.

The section labelled with the comment "// Just for testing" in the above example is there to ensure a canvas exists if you are running the macro offline. In fact, the system is designed to allow you to generate a ROOT file using you plugin and then develop the macro based on the file. Only at the end does one need to verify it works correctly with RootSpy.