NVIDIA H200 is the best AI inference accelerator: 141GB of HBM3e at 4.8TB/s with the widest system support. AMD Instinct MI350X is #2 with 288 GB at 8 TB/s, and Google Ironwood is #3 as the first TPU built for inference. NVIDIA L4 is the 72W pick.
Answrs VerdictThe safest default for large-model inference.
Memory: 141GB of HBM3e memory at 4.8TB/sPower: Max thermal design power up to 700W (configurable) for SXM and up to 600W (configurable) for the PCIe NVL versionPartitioning: Up to 7 Multi-Instance GPUs at 18GB each on SXM, or 16.5GB each on NVLForm factors: SXM or PCIe; HGX H200 partner and NVIDIA-Certified Systems with 4 or 8 GPUsPrice: Not published on the NVIDIA product page; sold through partner systems
NVIDIA’s H200 is the first GPU to offer 141GB of HBM3e at 4.8TB/s, according to NVIDIA, and that memory is what speeds up large language model serving. It comes as SXM with up to 700W (configurable) or as a PCIe NVL card with up to 600W, and can be split into up to 7 Multi-Instance GPUs. HGX H200 partner systems come with 4 or 8 GPUs. NVIDIA does not publish a price on the product page.
Go ifServing large language models on the most widely supported inference GPU
Skip ifYou need a card under 100W or a published list price
Why #1 over #2: It has less memory than AMD’s MI350X but far broader partner system support.
2
AMD Instinct MI350X
Answrs VerdictThe most memory and bandwidth per GPU here.
Memory: 288 GB of HBM3E at 8 TB/s peak memory bandwidthPower: Typical board power 1000W (TBP), in the OAM module form factor on a 54V UBBSilicon: 185 billion transistors, 256 MB last level cache and 8192-bit memory interfaceComparison: AMD lists 288 GB and 8.0 TB/s for MI350X against 180 GB and 7.7 TB/s for B200 SXM5Price: Not published on the AMD product page
AMD’s Instinct MI350X pairs 288 GB of HBM3E with 8 TB/s of peak memory bandwidth on a 185 billion transistor design with 256 MB of last level cache. It is an OAM module with a typical board power of 1000W. AMD’s own comparison lists 288 GB and 8.0 TB/s against 180 GB and 7.7 TB/s for the B200 SXM5. AMD does not publish a price on the page.
Go ifFitting very large models into one GPU with the most memory on this list
Skip ifYour data center cannot supply 1000W per accelerator
Why #2 over #3: It holds more memory per chip than Google’s Ironwood, but you cannot rent it by the chip-hour from AMD.
3
Google Cloud TPU Ironwood
Answrs VerdictGoogle’s inference-first TPU, rented by the chip-hour.
Memory: 192 GB of HBM per chip with 7.37 TB/s of HBM bandwidthBuilt for: Google calls Ironwood its seventh-generation TPU and the first designed specifically for inferenceScale: Comes in a 256 chip configuration and a 9,216 chip configurationPrice: Cloud TPU on-demand price for Ironwood is $12.00 per chip-hour in us-central1 (Iowa); rented, not soldAvailability: Available as a Google Cloud service rather than a card you buy
Google calls Ironwood its seventh-generation TPU and the first designed specifically for inference. Each chip has 192 GB of HBM and 7.37 TB/s of HBM bandwidth, and it comes in 256 chip and 9,216 chip configurations. It is a cloud service: Google’s Cloud TPU pricing page lists Ironwood on demand at $12.00 per chip-hour in us-central1 (Iowa).
Go ifTeams already on Google Cloud serving models at large scale
Skip ifYou want to own hardware in your own rack
Why #3 over #4: It publishes a rental price, which AWS does not for Trainium2 on its Trn2 page.
4
AWS Trainium2
Answrs VerdictAmazon’s own accelerator, rented as EC2 Trn2.
Memory: trn2.3xlarge lists 1 Trainium2 chip and 96 GB of accelerator memoryInstance: Trn2 instances are powered by 16 Trainium2 chips with 1.5 TB of HBM3 and 46 TBps of memory bandwidthCompute: Up to 20.8 FP8 petaflops per Trn2 instanceUltraServer: Trn2 UltraServers connect 64 Trainium2 chips across four Trn2 instances for up to 83.2 FP8 petaflopsPrice: No per-chip price on the Trn2 page; rented as EC2 instances
AWS Trainium2 powers Trn2 instances: the trn2.3xlarge lists 1 chip and 96 GB of accelerator memory, while a Trn2 instance has 16 chips, 1.5 TB of HBM3 and 46 TBps of memory bandwidth, up to 20.8 FP8 petaflops. Trn2 UltraServers connect 64 chips across four instances for up to 83.2 FP8 petaflops. The page lists no per-chip price.
Go ifAWS customers who want custom silicon for inference and training
Skip ifYou need hardware outside AWS
Why #4 over #5: It scales to 64 chips in an UltraServer, ahead of Gaudi 3’s 8-accelerator OAM systems.
5
Intel Gaudi 3
Answrs VerdictOpen, Ethernet-scaled inference with 128 GB per card.
Memory: 128 GB of HBM2e with 3.7 TB of memory bandwidth and 96 MB of on-board SRAMNetworking: 24 200 Gigabit Ethernet ports for scalable networkingPCIe card: Full-height PCIe add-in card at 600 wattsForm factors: Available in OAM on a UBB with 8 accelerators connected all-to-all, and as a PCIe CEM card for inferencing and fine tuningPrice: Not published on Intel’s newsroom pages
Intel’s Gaudi 3 has 128 GB of HBM2e, 3.7 TB of memory bandwidth and 96 MB of on-board SRAM, plus 24 200 Gigabit Ethernet ports for scaling out. It comes in OAM form on a UBB with 8 accelerators, and as a full-height PCIe card at 600 watts for inferencing and fine tuning. Intel does not publish a price in its newsroom material.
Go ifEthernet-based inference clusters outside the CUDA ecosystem
Skip ifYou need the newest memory type or a published price
Why #5 over #6: It uses standard Ethernet for scale-out, while Rebellions’ Rebel100 relies on UCIe chiplets and PCIe.
6
Rebellions Rebel100
Answrs VerdictA 144 GB HBM3E inference chip from Korea.
Memory: HBM3E 144GB at 4.8TB/sCompute: 1,024 TFLOPS (FP16) and 2,048 TFLOPS (FP8)Power: Power consumption up to 600WArchitecture: A 4-homogeneous-chiplet SoC based on UCIe-Advanced with 2x PCIe Gen5 x16Price: Not published on the Rebellions product page
Rebellions’ Rebel100 is a 4-chiplet SoC with 144GB of HBM3E at 4.8TB/s, 1,024 TFLOPS of FP16 and 2,048 TFLOPS of FP8, drawing up to 600W. It connects through 2x PCIe Gen5 x16 and natively supports PyTorch 2.x, vLLM and Triton. Rebellions does not publish a price.
Go ifBuyers who want a non-GPU HBM3E inference chip with rack-scale options
Skip ifYou need a card you can buy off the shelf today
Why #6 over #7: It matches H200’s 4.8TB/s with more memory, but lacks NVIDIA’s partner ecosystem.
7
Cerebras CS-3
Answrs VerdictWafer-scale silicon, now with a CS-4 successor.
Memory: Can be paired with a single MemoryX unit of up to 1,200 TB of external memorySilicon: Over 4 trillion transistors, 57x more than the largest GPU, and 2x faster than its predecessorScale: Up to 2048 CS-3 systems can be linked togetherSuccessor: Cerebras says CS-4 delivers up to 10x more throughput per watt than CS-3 and shipments begin this quarterPrice: Not published on the Cerebras pages
Cerebras introduced the CS-3 with over 4 trillion transistors, 57x more than the largest GPU, and 2x the speed of its predecessor. Up to 2048 systems can be linked, and one CS-3 can be paired with a MemoryX unit of up to 1,200 TB. Cerebras now markets the CS-4, which it says delivers up to 10x more throughput per watt than CS-3. No price is published.
Go ifOrganizations running very large models who want wafer-scale hardware
Skip ifYou want a card for a standard server or the newest Cerebras system
Why #7 over #8: It scales further than Tenstorrent’s boards but is a rack system, not a card.
8
NVIDIA L4
Answrs VerdictThe 72W card for small-model inference at scale.
Memory: 24GB of GPU memory at 300GB/sPower: Max thermal design power of 72WForm factor: 1-slot low-profile PCIe with a PCIe Gen4 x16 interconnectPrice: Not published on the NVIDIA product page
NVIDIA’s L4 is a 1-slot low-profile PCIe card with 24GB of memory at 300GB/s and a max thermal design power of 72W. It uses a PCIe Gen4 x16 interconnect and goes into standard servers through partner and NVIDIA-Certified Systems. NVIDIA does not publish a price on the page.
Go ifDense, low-power inference in standard servers
Skip ifYou are serving large language models that need more than 24GB
Why #8 over #9: At 72W it fits servers that cannot cool Tenstorrent’s 300W boards.
9
Tenstorrent Blackhole p150a
Answrs VerdictA $1,399 open-stack accelerator you can order.
Memory: 32 GB of GDDR6 at 512 GB/sec memory bandwidthPrice: $1,399 for the p150a on Tenstorrent’s hardware pagePower: Total board power of 300W with 120 Tensix coresNetworking: Four passive QSFP-DD 800G ports for linking multiple cards togetherSoftware: Supported by Tenstorrent’s fully open source software stack
Tenstorrent’s Blackhole p150a has 120 Tensix cores and 32 GB of GDDR6 at 512 GB/sec, with a total board power of 300W and four passive QSFP-DD 800G ports to link cards together. It is priced at $1,399 on Tenstorrent’s hardware page and runs Tenstorrent’s fully open source software stack.
Go ifDevelopers who want an open, buyable AI card with a published price
Skip ifYou need data-center-class memory capacity
Why #9 over #10: It has more memory than the Wormhole n300 (32 GB against 24 GB) at the same $1,399 as the n300s.
10
Tenstorrent Wormhole n300
Answrs VerdictThe older two-ASIC Tenstorrent board at a low price.
Memory: 24 GB of GDDR6 at 576 GB/sec memory bandwidthPrice: $1,449 for the n300d and $1,399 for the n300sPower: Total board power of 300WSilicon: Two Wormhole ASICs and 128 Tensix cores, with 466 TeraFLOPS (FP8)Cooling: n300d includes a 2.5-slot active cooling solution; n300s is passively cooled
Tenstorrent’s Wormhole n300 carries two Wormhole ASICs and 128 Tensix cores, with 24 GB of GDDR6 at 576 GB/sec, 466 TeraFLOPS (FP8) and a 300W board power. The n300d is $1,449 with a 2.5-slot active cooling solution and the n300s is $1,399 with passive cooling.
Go ifCheap experiments with a multi-chip Tensix board
Skip ifYou want the newer Blackhole generation
How we ranked these
We ranked chips with a maker-published memory capacity and bandwidth, a maker that positions them for inference, and a way to buy or rent them, using only each maker’s own pages. The old list also had Google’s earlier Trillium TPU, Groq, SambaNova, Graphcore and Qualcomm entries, which we replaced with Ironwood, Rebel100, L4 and two Tenstorrent boards because we could not source a maker product photo and current spec page for them. Facts are taken from each maker's own published information; check the official site before you buy.
Questions people ask
What is the best AI inference accelerator?
NVIDIA H200 is our top pick: 141GB of HBM3e at 4.8TB/s in SXM and PCIe forms with the broadest partner system support. AMD Instinct MI350X is #2 with 288 GB at 8 TB/s, and Google’s Ironwood is #3 as the first TPU designed specifically for inference.
Which accelerator has the most memory?
Of the chips on this list with a stated on-package capacity, AMD Instinct MI350X leads with 288 GB of HBM3E, followed by Google Ironwood at 192 GB per chip and Rebellions Rebel100 at 144GB.
Can you buy these accelerators at a published price?
Only Tenstorrent publishes card prices: $1,399 for the Blackhole p150a and $1,449 for the Wormhole n300d. Google publishes an Ironwood rental price of $12.00 per chip-hour in us-central1. NVIDIA, AMD, Intel, Rebellions and Cerebras do not publish a price on the pages we used.
What is the lowest-power inference card here?
NVIDIA L4 has a max thermal design power of 72W in a 1-slot low-profile PCIe card with 24GB of memory. The Tenstorrent boards are rated at 300W, and AMD MI350X is 1000W.
Memory per accelerator, in GB, as listed on each maker’s page (Trainium2 per chip; Cerebras CS-3 not shown because its memory is external) at a glance
NVIDIA L424 GB
Tenstorrent Wormhole n30024 GB
Tenstorrent Blackhole p150a32 GB
AWS Trainium296 GB
Intel Gaudi 3128 GB
NVIDIA H200141 GB
Rebellions Rebel100144 GB
Google Cloud TPU Ironwood192 GB
AMD Instinct MI350X288 GB
Memory per accelerator, in GB, as listed on each maker’s page (Trainium2 per chip; Cerebras CS-3 not shown because its memory is external), as published by each maker. Picks without a published figure are left off.
