Nvidia DGX H100 User Manual

NVIDIA DGX H100 User Guide
1.2.3. Network Modules
▶ New form factor for aggregate PCIe network devices
▶ Consolidates four ConnectX-7 networking cards into a single device
▶ Two networking modules are installed on interposer board
▶ Interposer board connects to CPUs on one end and to GPU tray on the other
▶ DensiLink cables are used to go directly from ConnectX-7 networking cards to OSFP connectors
at the back of the system
Each DensiLink cable has two ports, one from each ConnectX-7 card
Table 6: Table 6. Network Modules
Port ConnectX Device Network Module/CPU GPU Default RDMA
OSFP1P1 CX0 1 7 ibp220s0 mlx5_11
OSFP1P2 CX1 1 4 ibp154s0 mlx5_6
OSFP2P1 CX2 1 6 ibp206s0 mlx5_10
OSFP2P2 CX3 1 5 ibp192s0 mlx5_9
OSFP3P1 CX2 0 2 ibp79s0 mlx5_4
OSFP3P2 CX3 0 1 ibp64s0 mlx5_3
OSFP4P1 CX0 0 3 ibp94s0 mlx5_5
OSFP4P2 CX1 0 0 ibp24s0 mlx5_0
1.2. Network Connections, Cables, and Adaptors 13

Nvidia DGX H100 Specifications

General

GPU	8x NVIDIA H100 Tensor Core GPUs
GPU Memory	640 GB HBM3 (80GB per GPU)
System Memory	2 TB DDR5
Form Factor	6U rackmount
Storage	30TB NVMe SSD
Power Supply	10kW
Interconnect	NVLink 4.0