Difference between revisions of "Infiniband testing 19-dec-2012"
From GlueXWiki
(Created page with "== Testing Infiniband in Rack Room == '''Setup''' * Have 4 nodes, 16 cores each. * Each node has one GBit and one Infiniband port. * Infiniband switch rated at 20 Gbit/sec. * R...") |
m |
||
| Line 9: | Line 9: | ||
* Run ET system and event producer on one node, run remote consumers on the others. | * Run ET system and event producer on one node, run remote consumers on the others. | ||
* Event producer continually reads from file and inserts events into ET system. | * Event producer continually reads from file and inserts events into ET system. | ||
| − | * Some consumers | + | * Some consumers just get events and dump them (blaster mode), others process the events. |
* Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec). | * Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec). | ||
| − | ''' | + | '''Observations''' |
| + | * Single (local) producer can insert events at 380 kHz, avg event size about 8 kBytes. | ||
| + | ** producer uses 98% of one core, et system uses 8%. | ||
| + | * Single remote consumer running in blaster mode gets events at 74 kHz (4.7 Gbit/s), 75% cpu. | ||
| + | ** producer 25%, et 23%. | ||
| + | * Two consumers 148 kHz aggregate (9.5 Gbit/s), 75% cpu. | ||
| + | ** producer 48%, et 42%. | ||
| + | * Three consumers 188 kHz (12 Gbit/s), 63% cpu. | ||
| + | ** producer 60%, et 58%. | ||
Revision as of 10:13, 18 December 2012
Testing Infiniband in Rack Room
Setup
- Have 4 nodes, 16 cores each.
- Each node has one GBit and one Infiniband port.
- Infiniband switch rated at 20 Gbit/sec.
- Running IP over Infiniband, which decreases effective throughput.
- Run ET system and event producer on one node, run remote consumers on the others.
- Event producer continually reads from file and inserts events into ET system.
- Some consumers just get events and dump them (blaster mode), others process the events.
- Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec).
Observations
- Single (local) producer can insert events at 380 kHz, avg event size about 8 kBytes.
- producer uses 98% of one core, et system uses 8%.
- Single remote consumer running in blaster mode gets events at 74 kHz (4.7 Gbit/s), 75% cpu.
- producer 25%, et 23%.
- Two consumers 148 kHz aggregate (9.5 Gbit/s), 75% cpu.
- producer 48%, et 42%.
- Three consumers 188 kHz (12 Gbit/s), 63% cpu.
- producer 60%, et 58%.
