Nvidia DGX A100 User Manual

Network Configuration
NVIDIA DGX A100 DU-09821-001 _v01|45
$ sudo vi /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
<port-designation>:
dhcp4: no
dhcp6: no
addresses: [10.10.10.2/24]
gateway4: 10.10.10.1
nameservers:
search: [<mydomain>, <other-domain>]
addresses: [10.10.10.1, 1.1.1.1]
Consult your network administrator for the appropriate information for the items in bold,
such as network, gateway, and nameserver addresses, and use the port designations that
you determined in step 1.
3. After you complete your edits, press ESC to switch to command mode, then save the file to
the disk and exit the editor.
4. Apply the changes.
$ sudo netplan apply
Note: If you are not returned to the command line prompt after a minute, reboot the system.
For additional information, see https://help.ubuntu.com/lts/serverguide/network-
configuration.html.en.
7.7. Switching Between InfiniBand and
Ethernet
The NVIDIA DGX A100 System is equipped with up to eight NVIDIA ConnectX-6 or ConnectX-7
single-port network cards on the I/O board, typically used for cluster communications. By
default, these are configured as InfiniBand ports, but you have the option to convert these to
Ethernet ports.
For these changes to work properly, the configured port must connect to a networking
switch that matches the port configuration. In other words, if the port configuration is set to
InfiniBand, then the external switch should be an InfiniBand switch with the corresponding
InfiniBand cables. If the port configuration is set to Ethernet, the switch should also be
Ethernet.
The DGX A100 is also equipped with one (and optionally two) dual-port connections typically
used for network storage and configured by default for Ethernet. These can also be configured
for InfiniBand.
Note: On the dual-port cards, if one of the ports is configured for Ethernet and the other port is
configured for InfiniBand, the following limitations apply.
‣
FDR is not supported on the InfiniBand port (port 1 or 2).
‣
If port 1 is InfiniBand, then port 2 (Ethernet) does not support 40 GbE/10GbE.

Nvidia DGX A100 Specifications

General

GPU	8 x NVIDIA A100 Tensor Core GPUs
System Memory	1 TB DDR4
Storage	15 TB NVMe SSD
GPU Memory	320 GB total (40 GB per GPU)
CPU	2 x 64-Core AMD EPYC 7742
Networking	8 x 200 Gb/s InfiniBand or Ethernet
Interconnect	NVIDIA NVLink