Locking in JANA

General Comments

• Be careful when creating / filling histograms & trees.
  • Be VERY careful when doing so inside of a factory or action (follow the example code!).
• Be careful when modifying / reading JEventProcessor class member data.
• Be careful when returning early from functions: Make sure you release the lock first.
• Avoid nested locks.

Basics: Factories & Plugin Processors

• Plugin Processors: For each plugin, there is a total of one instance of the processor object, regardless of how many threads you run with.
  • Thus, since the same object is shared amongst all threads, you must be careful about modifying it.
• Factories: Each thread has it's own instance of each factory.
  • Thus, you can easily modify them without worrying about interfering with other threads.
  • However, sharing/modifying objects for a factory between the threads is difficult, and you must be careful about doing so.

Plugin Processor Variables

Basics

• Do not place use any locks in JEventProcessor::init() or ::fini(): they are unnecessary, and a waste of time.
  • JANA is guaranteed to be single-threaded in these functions.

• Do not use or modify any global variables, unless you are in a lock.
  • Or even better, just avoid them at all possible. It's terrible programming practice.

Class members

• Do NOT use or modify any class member-variables that you define for your JEventProcessor (i.e. in header file), unless you are in a lock (or in init(), fini()). These variables can be modified by other threads. This locking can be done via:

LockState(); //ACQUIRE PROCESSOR LOCK
UnlockState(); //RELEASE PROCESSOR LOCK

• For data updated in brun() and read in evnt(), it is sometimes unfeasible to work around this, such as when working with calibration constants. As long as the run number doesn't change, this should still be OK (they should just be overwriting the same value).
  • However, running your program over data with two different runs may result in unpredictable behavior.

• Just to be safe, have as few class member variables as possible, and reduce the amount of time spent in locks. In other words, use function-scope (local) variables instead.
  • Where possible of course. E.g. you can't avoid it with histograms, but you need to lock to fill those anyway.

Creating Histograms, TTrees, TDirectories, or TFiles in a plugin processor

• If you are creating them inside of JEventProcessor::init() or ::fini(), you do not need a lock. Just create them.
• If you are creating them outside of these function (e.g. brun()): Use the global ROOT JANA lock functions below.
  • This is because they modify gDirectory, or depend on gDirectory being constant.

japp->RootWriteLock(); //ACQUIRE ROOT LOCK
japp->RootUnLock(); //RELEASE ROOT LOCK

Creating Histograms, TTrees, TDirectories, or TFiles somewhere else (e.g. a factory, or an analysis action)

• Remember, each thread has it's own instance of each factory.
  • This is also true of analysis actions, unless they are created directly in the plugin processor.
• Thus, you must be careful to only create these objects once, and not once for each thread.
  • Init() and fini() are not inherently safe: They are called once on each object, so once by each thread.

Example: Creating a histogram

//Factory header file:
TH1I* dMyHist;
 
//Creation code:
japp->RootWriteLock(); //ACQUIRE ROOT LOCK
{
   dMyHist = (TH1I*)gDirectory->Get("my_hist");
   if(dMyHist == NULL) //else already created by another thread, and already retrieved
      dMyHist = new TH1I("my_hist", "title", 100, 0.0, 1.0);
}
japp->RootUnLock(); //RELEASE ROOT LOCK

Example: Creating a tree

• You must be extra careful here. Not only do you need to do something similar to the histogram-creation code, but you must be careful about the branch memory.
• Each factory/action must have access to the same branch memory objects for the TTree, so they can set them prior to filling. Either that, or you have to change all of the branch addresses prior to filling every time.
  • If you are simply filling branches with fundamental data types (floats, ints, etc.) or arrays, then you can just let the TTree keep track of the memory for you.
  • For other objects, you must keep track of the memory objects yourself. To safely share them amongst threads, you must use static variables that are defined locally within functions, and return references to them. These functions should only be called within a lock. See src/libraries/ANALYSIS/DEventWriterROOT.* for an example.
• Example code:

//Factory header file:
TTree* dTree;
 
//Creation code:
japp->RootWriteLock(); //ACQUIRE ROOT LOCK
{
   dTree = (TH1I*)gDirectory->Get("my_tree");
   if(dTree == NULL) //else already created by another thread, and already retrieved
   {
      dTree = new TTree("my_tree", "title");
      dTree->Branch("my_branch", new Float_t, "my_branch/F");
   }
}
japp->RootUnLock(); //RELEASE ROOT LOCK

Filling Histograms

• If you are filling histograms, you do NOT need to grab the global ROOT JANA lock functions. Try to instead grab a lock with a smaller scope, so that other threads can write to other histograms that don't interfere with yours.
  • In you are directly in the plugin processor, there's a special function to do this, that locks for the current plugin only. It is:

japp->RootFillLock(this); //ACQUIRE ROOT FILL LOCK //this: JEventProcessor pointer
japp->RootFillUnLock(this); //RELEASE ROOT FILL LOCK //this: JEventProcessor pointer

Filling TTrees

hd_root.root

• If you are filling TTrees to the global ROOT file (i.e. hd_root.root), write to the TTrees with the global ROOT JANA lock functions below:
  • This is because TTree::Fill() periodically writes to the file, modifying it.

japp->RootWriteLock(); //ACQUIRE ROOT LOCK
japp->RootUnLock(); //RELEASE ROOT LOCK

Other Output Files

• If you are filling TTrees to a local ROOT file (i.e. NOT hd_root.root), you do NOT need to grab the global ROOT JANA lock functions. Try to instead grab a lock with a smaller scope, so that other threads can write to other trees that don't interfere with yours.
  • Grab the same for every tree that is written to the same output file. This is because TTree::Fill() periodically writes to the file, modifying it.
  • This can be done with:

japp->WriteLock("MyFileName"); //ACQUIRE FILE LOCK
japp->Unlock("MyFileName"); //RELEASE FILE LOCK

Analysis Actions

• Just use the pre-generated code and it will work.
  • Unless you're creating TTrees and not saving them into a unique file. In which case, follow the instructions in the appropriate sections.

• The pre-generated code acquires a action-local lock, such that you can fill the histograms in the current action without interfering with threads filling in other actions.

Nested Locks

Overview

• Don't EVER acquire a lock from within another lock. This is extremely dangerous, and can easily result in deadlock: program hangs, because two threads are waiting for each other to unlock.
  • If you think you need to do a nested lock, you are probably doing it wrong.
  • If you absolutely must have a nested lock, be sure to acquire & release the locks in the exact same order everywhere. This is of course, very difficult to guarantee, because once someone starts doing it one plugin, someone else may have a different convention in another plugin ...

Avoiding Nested Locks

• To avoid nested locking, when you're in a lock:
  • Never query JANA for calibration constants
  • Never query JANA for global parameters
  • Never query JANA for objects (JEventLoop::Get()) //This is also potentially incredibly slow
  • Never call the event writers to save your EVIO skims, REST files, ROOT trees, etc.
  • Never call any function that you don't understand exactly what it's doing. It might be acquiring a lock underneath.