Infiniband testing 19-dec-2012

Testing Infiniband in Rack Room

Setup

• Have 4 nodes, 16 cores each.
• Each node has one GBit and one Infiniband port.
• Infiniband switch rated at 20 Gbit/sec.
• Running IP over Infiniband, which decreases effective throughput.
• Run ET system and event producer on one node, run remote consumers on the others.
• Event producer continually reads from file and inserts events into ET system.
• Some consumers just get events and dump them (blaster mode), others process the events.
• Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec).

Observations

• Single (local) producer can insert events at 380 kHz, avg event size about 7.4 kBytes.
  • producer uses 98% of one core, et system uses 8%.
• Single remote consumer running in blaster mode gets events at 74 kHz (4.4 Gbit/s), 75% cpu.
  • producer 25%, et 23%.
• Two consumers 148 kHz aggregate (8.8 Gbit/s), 75% cpu.
  • producer 48%, et 42%.
• Three consumers 188 kHz (11.1 Gbit/s), 63% cpu.
  • producer 60%, et 58%.