Difference between revisions of "Infiniband testing 19-dec-2012"
From GlueXWiki
m |
m |
||
(2 intermediate revisions by the same user not shown) | |||
Line 11: | Line 11: | ||
* Some consumers just get events and dump them (blaster mode), others process the events. | * Some consumers just get events and dump them (blaster mode), others process the events. | ||
* Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec). | * Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec). | ||
+ | * Note: did not tune any IP parameters for IP over IB. | ||
− | ''' | + | |
+ | '''Measurements in Blaster Mode''' | ||
* Single (local) producer can insert events at 380 kHz, avg event size about 7.4 kBytes. | * Single (local) producer can insert events at 380 kHz, avg event size about 7.4 kBytes. | ||
** producer uses 98% of one core, et system uses 8%. | ** producer uses 98% of one core, et system uses 8%. | ||
− | * Single remote consumer | + | * Single remote consumer receives events at 74 kHz (4.4 Gbit/s), 75% cpu usage. |
** producer 25%, et 23%. | ** producer 25%, et 23%. | ||
− | * Two consumers 148 kHz aggregate (8.8 Gbit/s), 75% cpu. | + | * Two consumers (different nodes) 148 kHz aggregate (8.8 Gbit/s), 75% cpu. |
** producer 48%, et 42%. | ** producer 48%, et 42%. | ||
* Three consumers 188 kHz (11.1 Gbit/s), 63% cpu. | * Three consumers 188 kHz (11.1 Gbit/s), 63% cpu. | ||
** producer 60%, et 58%. | ** producer 60%, et 58%. | ||
+ | * Adding more consumers on the nodes doesn't change aggregate bandwidth. | ||
+ | |||
+ | |||
+ | |||
+ | '''Measurements in Processing Mode''' | ||
+ | * Single remote consumer receives events at 18 kHz, 93% cpu usage. | ||
+ | ** producer 9%, et 8%. | ||
+ | * Two consumers 36 kHz, 93% cpu. | ||
+ | ** producer 15%, et 11%. | ||
+ | * Three consumers 53 kHz, 93% cpu. | ||
+ | ** producer 17%, et 17%. | ||
+ | * Add another consumer on one of the nodes, 58 kHz, 93% cpu. | ||
+ | ** producer 19%, et 18%. | ||
+ | * Add another consumer on one of the nodes, 64 kHz, 93% cpu. | ||
+ | ** producer 30%, et 22%. | ||
+ | * Add another, 68 kHz, 93% cpu. | ||
+ | ** producer 20%, et 20%. | ||
+ | |||
+ | |||
+ | |||
+ | '''Observations''' | ||
+ | * Seem to saturate IB producer link at 11 Gbit/sec in blaster mode, requires three consumers. | ||
+ | * Cannot saturate IB producer link in processing mode despite many consumers. | ||
+ | * Adding consumers on a node doesn't scale rate up proportionally. | ||
+ | * Not sure what's going on with IB. N.B. Gbit Ethernet scaled as expected. |
Latest revision as of 14:55, 18 December 2012
Testing Infiniband in Rack Room
Setup
- Have 4 nodes, 16 cores each.
- Each node has one GBit and one Infiniband port.
- Infiniband switch rated at 20 Gbit/sec.
- Running IP over Infiniband, which decreases effective throughput.
- Run ET system and event producer on one node, run remote consumers on the others.
- Event producer continually reads from file and inserts events into ET system.
- Some consumers just get events and dump them (blaster mode), others process the events.
- Note: when using Ethernet instead of IB got about 95% bandwidth of a Gbit link (950 Mbits/sec).
- Note: did not tune any IP parameters for IP over IB.
Measurements in Blaster Mode
- Single (local) producer can insert events at 380 kHz, avg event size about 7.4 kBytes.
- producer uses 98% of one core, et system uses 8%.
- Single remote consumer receives events at 74 kHz (4.4 Gbit/s), 75% cpu usage.
- producer 25%, et 23%.
- Two consumers (different nodes) 148 kHz aggregate (8.8 Gbit/s), 75% cpu.
- producer 48%, et 42%.
- Three consumers 188 kHz (11.1 Gbit/s), 63% cpu.
- producer 60%, et 58%.
- Adding more consumers on the nodes doesn't change aggregate bandwidth.
Measurements in Processing Mode
- Single remote consumer receives events at 18 kHz, 93% cpu usage.
- producer 9%, et 8%.
- Two consumers 36 kHz, 93% cpu.
- producer 15%, et 11%.
- Three consumers 53 kHz, 93% cpu.
- producer 17%, et 17%.
- Add another consumer on one of the nodes, 58 kHz, 93% cpu.
- producer 19%, et 18%.
- Add another consumer on one of the nodes, 64 kHz, 93% cpu.
- producer 30%, et 22%.
- Add another, 68 kHz, 93% cpu.
- producer 20%, et 20%.
Observations
- Seem to saturate IB producer link at 11 Gbit/sec in blaster mode, requires three consumers.
- Cannot saturate IB producer link in processing mode despite many consumers.
- Adding consumers on a node doesn't scale rate up proportionally.
- Not sure what's going on with IB. N.B. Gbit Ethernet scaled as expected.