What Does GPU Infrastructure for AI Workloads Look Like in 2027?
GPU infrastructure for AI workloads is built in layers: accelerators grouped into servers, servers joined into racks and clusters by very fast networking, and all of it fed with power and cooled. The two building blocks NVIDIA publishes are an 8-GPU server and a 72-GPU rack. The DGX B200 holds 8 Blackwell GPUs with 1,440 GB of total GPU memory, and the GB200 NVL72 "connects 36 Grace CPUs and 72 Blackwell GPUs in a rack-scale, liquid-cooled design."
Most teams do not buy this hardware outright. Amazon EC2 P5 instances offer up to 8 H100 GPUs per instance and scale in UltraClusters up to 20,000 H100 or H200 GPUs. The limit on all of it is increasingly power: the IEA says data centres used around 415 TWh in 2024 and projects around 945 TWh by 2030.
This page does not give cloud hourly prices, because the AWS page read for it publishes none, and it does not give rack power for the GB200 NVL72, because the NVIDIA page read did not state it.
The building block: an 8-GPU server
The basic unit is a server that carries several GPUs on a shared board. NVIDIA's DGX B200 is a good published example.
- GPUs8 NVIDIA Blackwell GPUs
- GPU memory1,440 GB total, with 64 TB/s HBM3e bandwidth
- GPU-to-GPU link14.4 TB/s aggregate NVLink bandwidth
- CPUs and system memory2 Intel Xeon Platinum 8570 processors, 112 cores total; 2 TB system memory
- Size and power10 rack units; system power approximately 14.3 kW max
Notice how much of the box is not GPU. The CPUs, system memory, NVMe storage and network cards are part of the same design, and the networking is listed as 8 single-port ConnectX-7 adapters at up to 400 Gb/s plus 2 BlueField-3 DPUs.
The older H200 shows how memory drives the generations. NVIDIA calls it "the first GPU to offer 141 gigabytes (GB) of HBM3e memory" at 4.8 TB/s, with a maximum thermal design power of up to 700W for the SXM version, and describes those specifications as preliminary.
The rack: 72 GPUs acting as one
The next step is to make a whole rack behave like one large accelerator. NVIDIA's GB200 NVL72 is the published example. It uses 36 Grace CPUs and 72 Blackwell GPUs, and each GB200 Superchip pairs 1 Grace CPU with 2 Blackwell GPUs.
- GPU-to-GPU link130 TB/s NVLink for the rack; 3.6 TB/s per Superchip
- GPU memory13.4 TB HBM3E at 576 TB/s
- CPU memory17 TB LPDDR5X at 14 TB/s
- CoolingLiquid-cooled
NVIDIA says liquid cooling "increases compute density, reduces the amount of floor space used." The same page makes performance claims against the H100, including 30x faster real-time trillion-parameter LLM inference, 4x for training and 25x more performance at the same power, and says they are "Projected performance subject to change" and depend on stated test setups. Treat them as the maker's claims, not independent measurements.
The network that ties it together
Inside a rack, NVLink does the work. Between racks, the cluster depends on fabric. On AWS, P5 instances offer up to 3,200 Gbps of EFA networking on the large sizes, and are deployed in EC2 UltraClusters on a "petabit-scale nonblocking network."
NVIDIA's Vera Rubin page lists the same layering for its next platform: an NVLink 6 switch inside the rack, ConnectX-9 SuperNICs and BlueField-4 DPUs, and Quantum-X800 InfiniBand or Spectrum-X Ethernet to scale out.
Renting instead of buying: cloud GPU clusters
On AWS, P5 instances carry up to 8 NVIDIA H100 GPUs and up to 640 GB of HBM3 GPU memory per instance. P5e and P5en instances carry up to 8 H200 GPUs and up to 1128 GB of HBM3e per instance. UltraClusters scale up to 20,000 H100 or H200 GPUs, which AWS says can deliver up to 20 exaflops of aggregate compute.
AWS says P5 can "reduce cost to train ML models by up to 40%" against previous-generation GPU instances. That is a vendor claim for a stated comparison, not a price. The AWS page publishes no dollar rates and points to its EC2 pricing page and to Capacity Blocks for reservations, so check those for current numbers. For provider comparisons, see our guide to GPU cloud providers for AI training.
Power and cooling are the real limit
A single DGX B200 draws approximately 14.3 kW at maximum, and NVIDIA's operating range for it is 10 to 35 degrees Celsius. Multiply that by a hall of servers and the facility becomes the constraint.
The IEA says data centres "accounted for around 1.5% of the world's electricity consumption in 2024, or 415 terawatt-hours (TWh)" and are "set to more than double to around 945 TWh by 2030," with AI "the most important driver of this growth." The page read gives no AI-only share.
What NVIDIA says is next
NVIDIA's own page says its Vera Rubin NVL72 platform, with 72 Rubin GPUs and 36 Vera CPUs, is ramping into full production with server makers shipping systems. It claims more tokens per watt and lower cost per token than Blackwell but gives no numbers for that claim, and no memory capacity or bandwidth figures, so none are repeated here.
What this means when you plan
- Start from the workload. Training, inference and fine-tuning stress memory, networking and power differently. NVIDIA's own comparisons use stated latency and cluster setups, so match them to yours.
- Count the whole system. CPUs, memory, storage, DPUs and switches are part of the published bill of materials.
- Plan power and cooling first. Liquid cooling is part of the NVL72 design, and the DGX B200 lists a kilowatt figure and a temperature range.
- Price from the provider. Cloud GPU rates are on the provider's pricing pages and change; this page quotes none.
- Also read: how to measure and improve GPU utilization and hidden costs of running AI infrastructure.
Questions people ask
What is in a GPU server for AI?
Using NVIDIA's DGX B200 as the example: 8 Blackwell GPUs with 1,440 GB of total GPU memory, 2 Intel Xeon Platinum 8570 processors, 2 TB of system memory, ConnectX-7 network adapters and BlueField-3 DPUs, in 10 rack units.
What is a GB200 NVL72?
A rack-scale, liquid-cooled NVIDIA system that connects 36 Grace CPUs and 72 Blackwell GPUs, with 130 TB/s of NVLink bandwidth for the rack, per NVIDIA's product page.
How much GPU memory does a cloud instance have?
AWS says P5 instances offer up to 640 GB of HBM3 GPU memory per instance, and P5e and P5en up to 1128 GB of HBM3e.
How big can a cloud GPU cluster get?
AWS says EC2 UltraClusters scale up to 20,000 H100 or H200 GPUs.
How much does cloud GPU time cost?
The AWS P5 page read for this article publishes no dollar rates. It points to the EC2 pricing page and Capacity Blocks, which are the places to check.
Why does power matter so much?
The IEA says data centres used around 415 TWh in 2024 and are set to more than double to around 945 TWh by 2030, with AI the most important driver. A single DGX B200 is listed at approximately 14.3 kW max.
Sources
- NVIDIA: DGX B200 · 8-GPU server specifications
- NVIDIA: GB200 NVL72 · 72-GPU liquid-cooled rack
- NVIDIA: H200 · Memory and thermal design power
- NVIDIA: Vera Rubin platform · Next platform, maker claims
- Amazon EC2 P5 instances · Cloud GPU instances and UltraClusters
- IEA: Energy and AI · Data centre electricity use
- Wikimedia Commons: Nvidia DGX GB200.jpg · Hero photo, Pokiiri, CC BY-SA 4.0
GPU infrastructure is servers, racks, networking, power and cooling designed together. Use the makers' published specifications as a starting point and confirm current availability and pricing with your provider. This page is general information, not advice for your specific business.
Top photo: Pokiiri / CC BY-SA 4.0 via Wikimedia Commons