Infiniband testing 19-dec-2012

Testing Infiniband in Rack Room

Setup

Have 4 nodes, 16 cores each.
Each node has one GBit and one Infiniband port.
Infiniband switch rated at 20 Gbit/sec.
Running IP over Infiniband, which decreases effective throughput.
Run ET system and event producer on one node, run remote consumers on the others.
Event producer continually reads from file and inserts events into ET system.
Some consumers just get events and dump them (blaster mode), others process the events.
Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec).
Note: did not tune any IP parameters for IP over IB.

Measurements in Blaster Mode

Single (local) producer can insert events at 380 kHz, avg event size about 7.4 kBytes.
- producer uses 98% of one core, et system uses 8%.
Single remote consumer receives events at 74 kHz (4.4 Gbit/s), 75% cpu usage.
- producer 25%, et 23%.
Two consumers (different nodes) 148 kHz aggregate (8.8 Gbit/s), 75% cpu.
- producer 48%, et 42%.
Three consumers 188 kHz (11.1 Gbit/s), 63% cpu.
- producer 60%, et 58%.
Adding more consumers on the nodes doesn't change aggregate bandwidth.

Measurements in Processing Mode

Single remote consumer receives events at 18 kHz, 93% cpu usage.
- producer 9%, et 8%.
Two consumers 36 kHz, 93% cpu.
- producer 15%, et 11%.
Three consumers 53 kHz, 93% cpu.
- producer 17%, et 17%.
Add another consumer on one of the nodes, 58 kHz, 93% cpu.
- producer 19%, et 18%.
Add another consumer on one of the nodes, 64 kHz, 93% cpu.
- producer 30%, et 22%.
Add another, 68 kHz, 93% cpu.
- producer 20%, et 20%.

Observations

Seem to saturate IB producer link at 11 Gbit/sec in blaster mode, requires three consumers.
Cannot saturate IB producer link in processing mode despite many consumers.
Adding consumers on a node doesn't scale rate up proportionally.
Not sure what's going on with IB. N.B. Gbit Ethernet scaled as expected.