Online farm testing 5-dec-2012
From GlueXWiki
Current status of online farm tests in the counting house
- Infrastructure
- Main GBit Ethernet switch working (has 10 GBit capability).
- Big UPS working.
- Borrowed nodes gluon02-gluon05 working (64-bit RHEL6, 16 cores each, lots of memory).
- Dave A will install borrowed 40 GBit/sec InfiniBand switch (will eventually purchase 60 GBit).
- gluon02-05 have both GBit and IB interfaces.
- Software
- DAQ software installed and working (EVIO, cMsg, ET).
- rawevent plugin converts hddm file to EVIO format using mc2coda package from Dave A.
- Event size is 24 kB/event out of mc2coda.
- mc2coda_compress compresses raw EVIO file, 8 GB/evt.
- montest reads EVIO files and inserts events into ET system
- Results for local transfers
- Injection rate about 110 kHz per producer for uncompressed events, 350 kHz for compressed events.
- Injection rate scales with the number of producers.
- Each producer takes 100% of one core.
- ET system overhead about 5% of a core.
- Local consumers can keep up with producer rate.
- Results for one-way network transfers
- Remote consumer saturates GBit Ethernet at 95% level
- 5 kHz for uncompressed events, 15 kHz for compressed events.
- Consumers use about 25% of a core.
- ET system about 20% of a core.
- Results mostly independent of the number of consumer nodes.
- InfiniBand testing
- Expect factor 10 increase, not factor 40.
- DAQ tests show you get 25% of 40 GBit using TCP over IB.
- We expect to purchase 60 GBit IB.
- JANA analysis processes
- Waiting for development of evioETchannel class (me, soon).
- Dave L has updated JANA to read uncompressed EVIO files.
- Dave L is working on reading compressed EVIO data.
- Full test requires implementing translation table in JANA in a standard way.
- Farm manager
- Vardan and I will develop this.
- Additional testing
- RootSpy when JANA online processes working.
- Multiple ET systems on different nodes, different network architectures.
- Two-way network transfers.
- Simulated event recorder?