NVIDIA DGX B300 vs B200 vs H200 vs H100: Matching the Node to Your LLM Workload

Key Takeaways

• GPU memory is the decision axis, not compute. Per GPU: 80 GB HBM3 on H100, 141 GB HBM3e on H200, 192 GB on B200, 288 GB on B300; node totals of 640 GB, 1.1 TB, 1.4 TB and 2.1–2.3 TB. Whether your model plus KV cache fits one node decides your architecture before any FLOPS figure matters.
• DGX B300 is a reasoning and long-context machine. Blackwell Ultra delivers roughly 2× the attention performance and 1.5× the dense FP4 throughput of the DGX B200, with 288 GB per GPU from 12 high-density HBM3e stacks. Buy it for chain-of-thought models, 128K+ context windows, and agent fleets, not for work that a B200 already handles comfortably.
• DGX B200 is the mainstream Blackwell workhorse: roughly 3× the training and up to 15× the inference performance of a Hopper-generation DGX at 1,000 W per GPU, which most upgraded facilities can still cool. For enterprise LLM inference and fine-tuning, it holds the best price-to-capability ratio in the lineup.
• H200 fixes Hopper’s memory problem; H100 remains the value play. Its 141 GB and 4.8 TB/s lift token generation on memory-bound decode well above the H100’s 80 GB and 3.35 TB/s, on the same footprint and the same 700 W-class thermals. H100 nodes, new or certified refurbished, are still the cheapest credible entry into on-premises LLM training.
• Power and cooling gate the Blackwell decision more often than budget. Per-GPU TDP climbs from 700 W to 1,000 W to 1,400 W across H100, B200, and B300, and B300-class density assumes liquid cooling plus 1.6T-capable networking. If your facility cannot deliver that, the honest answer is B200 or H200 today and B300 at the next build-out.
• Decisions outside the server drive total cost of ownership. Energy over three to five years, liquid-cooling retrofit, InfiniBand or Spectrum-X fabric, high-throughput storage, rack and power distribution, and staff time account for a large share of program cost. Compare cost per million tokens or cost per fine-tuning run, never cost per node.

Overview

Choosing among the NVIDIA DGX B300, B200, H200, and H100 is no longer a question of which system is fastest. All four are fast. The question is which one matches the shape of your workload, the ceiling of your data center, and the number on your purchase order. A team serving a quantized 70B model to a few hundred internal users has almost nothing in common with a team fine-tuning a 405B model or running a fleet of long-context AI agents. Yet both teams end up comparing the same four DGX servers, and both sit through the same marketing slide.

One framing note before the specifications. A DGX server is a fixed, validated appliance: eight GPUs, an NVSwitch fabric, integrated networking, NVMe storage, and the full NVIDIA software stack, sold as a single unit. That removes integration risk and configuration freedom together. If your workload is well served by a conventional AI GPU server built from standard enterprise server components, that route is usually cheaper per FLOP. The DGX line earns its premium when in-node GPU-to-GPU bandwidth is the bottleneck.

The Short Answer: Which DGX Server Should You Choose?

Here are some quick answers to find the right DGX Server according to your needs:

• Choose DGX B300 for frontier-scale training, reasoning models, and long-context agent fleets.
• Choose DGX B200 for mainstream enterprise LLM inference and fine-tuning.
• Choose DGX H200 when you need memory headroom without a facility upgrade.
• Choose DGX H100 when cost per training hour matters more than peak throughput.

Hopper to Blackwell Ultra: The Three Shifts That Matter

Three things changed between the H100 and the B300, and each maps to a different kind of workload gain: memory capacity and bandwidth, low-precision compute, and the fabric that connects nodes together.

1.  From HBM3 to 12-High HBM3e Stacks

The H100 shipped with 80 GB of HBM3 at roughly 3.35 TB/s. The H200 kept Hopper’s compute engine but increased HBM3e capacity to 141 GB at 4.8 TB/s, a refresh that targeted the reality that LLM serving is usually memory-bound. Blackwell pushed further: 192 GB per B200 GPU and 288 GB per B300 GPU, using 12-high HBM3e stacks in place of the B200’s 8-high configuration, at around 8 TB/s per GPU. That is a 3.6× increase in GPU memory across three generations, the same way server memory capacity decides database and virtualization hosts, only with less room to compensate in software.

2.  FP4 Becomes a First-Class Precision

Hopper’s Transformer Engine made FP8 practical; Blackwell made FP4 practical, and Blackwell Ultra made it the design center. Dense FP4 runs at roughly 9 PFLOPS per B200 GPU and 15 PFLOPS per B300 GPU, about 67% more compute. At the node level, DGX B300 reaches approximately 144 PFLOPS sparse and 108 PFLOPS dense FP4, with FP8 around 72 PFLOPS sparse. TensorRT-LLM and vLLM now support FP4 quantization with limited quality loss, so production deployments increasingly capture the gains rather than merely project them.

3.  Networking, Interconnect and the Fabric

Inside the node, the DGX B300 uses six NV Switches, providing 4.8 TB/s of bidirectional bandwidth, so all eight GPUs behave as a single accelerator. Between nodes, Blackwell Ultra moves to ConnectX-8 networking with 1.6T optics, up from the 800G class. For single-node inference, this is irrelevant; for multi-node training across a GPU cluster, it is the difference between linear and disappointing scaling, and undersized server networking is among the costliest AI infrastructure mistakes.

1.  DGX H100: The Hopper Baseline

DGX H100 is still rational in 2026 for one reason: it is the cheapest credible way to own eight NVLink-connected data-center GPUs. With 640 GB of node GPU memory and roughly 3.35 TB/s per GPU, it handles full fine-tuning up to around the 13B class, LoRA well beyond that, and production inference on quantized 70B models. Its ceiling is memory, not compute; at 80 GB per GPU, a 70B FP16 model requires tensor parallelism before a single token can be served. Our guide to the NVIDIA H100 and its role in AI workloads covers the architecture in more depth.

The commercial case is strongest for two buyers: organizations building a first AI GPU server capability who need to prove value before requesting a facility upgrade, since Hopper thermals fit existing racks; and cost-sensitive research or batch-training environments where wall-clock time is negotiable. For both, our guide to buying refurbished data center servers sets out where certified pre-owned hardware fits, and Hopper GPU nodes are exactly where the discount is large and the risk manageable.

2.  DGX H200: Memory-Bound Relief on a Familiar Footprint

The practical value of the H200 is that it installs into an existing environment: same rack density, same power envelope, same air cooling, same server networking. For an infrastructure manager tasked with doubling inference capacity without a new build-out, it is often the only option that avoids a capital project. At node level you get around 1.1 TB of GPU memory, enough for a 70B FP16 model with a useful KV cache, or a quantized 180B-class model with room to spare.

The limitation is that you are still buying Hopper compute. There is no FP4 path, so you cannot achieve the throughput multiplier needed for Blackwell economics to work at high volume. Prefill-heavy workloads such as long-document ingestion and large-batch embedding generation for RAG remain compute-bound, and Blackwell would relieve that pressure exactly if your roadmap includes reasoning models or agentic systems within eighteen months; size an H200 purchase as a bridge rather than a destination.

3.  DGX B200: The Mainstream Blackwell Workhorse

For most enterprises, the DGX B200 is the right choice. Against a Hopper-generation DGX, it trains around three times faster and serves inference up to fifteen times faster, backed by 192 GB of HBM3e per GPU and around 1.4 TB per node. Critically, it does so at approximately 1,000 W per GPU rather than 1,400 W, a density many upgraded facilities can still cool, and one that fits power distribution most enterprise rack server environments can be brought up to. At roughly 9 PFLOPS of dense FP4 per GPU, a single node supports very high concurrency with a quantized 70B model.

On fine-tuning, the 1.4 TB node memory pool is the headline. Full fine-tuning of a 70B model with optimizer states and activations becomes feasible on a single node with standard sharding, removing the multi-node coordination that makes such jobs fragile on Hopper. LoRA and QLoRA on 100B-plus models become routine. For context beyond the DGX line, our GPU hierarchy and benchmark comparison show how each generation of NVIDIA graphics card ranks relative to alternatives.

4.  DGX B300: Blackwell Ultra for Reasoning and Agents

DGX B300 is built around a bet that the dominant AI workload is shifting from single-pass generation to extended reasoning. Eight Blackwell Ultra SXM GPUs with Intel Xeon 6776P processors deliver 2.1 TB of HBM3e at the node level, 288 GB per GPU via 12 high stacks, at roughly 8 TB/s per GPU. Against DGX B200, that is roughly 2× the attention performance and 1.5× the dense FP4 performance. Storage matches: eight 3.84 TB NVMe drives at around 50 GB/s peak, plus mirrored 1.92 TB OS devices; comparable high-capacity enterprise NVMe SSDs are what any serious training cluster needs behind the GPUs.

The workloads that justify the premium share one characteristic: they repeatedly hold a very large model and a very long context in memory simultaneously. Reasoning models, agentic pipelines, 128K-plus context windows, and large mixture-of-experts architectures all qualify. Testing shows an eight-GPU Blackwell node sustaining over 250 tokens per second per user on a 671B-parameter reasoning model, throughput that changes what is commercially viable.

A pragmatic reading: buy DGX B300 if you are training frontier-scale models, are commercially exposed to reasoning-model latency, or are building sovereign and regulated AI capacity where denser nodes reduce the compliance surface. If B200 meets your workload today, the incremental capability is real, but the infrastructure cost of reaching it is disproportionate. Read our comparison of 1U, 2U, and 4U rack server form factors first, since the chassis fixes GPU density and cooling capacity long before the GPUs arrive.

LLM Inference: How the Four Systems Actually Behave

·      Prefill vs. Decode

Prefill, which processes the input prompt, is compute-intensive and runs in parallel. It benefits from FLOPS, so Blackwell’s FP4 throughput delivers clear improvements. The B300’s roughly 2× attention capacity compared to the B200 is also evident in longer prompts.

Decode, by contrast, generates output tokens sequentially, and memory bandwidth limits it because each token must read the model weights and the KV cache. Performance hinges on HBM bandwidth, so the H200’s upgrade to 4.8 TB/s delivered tangible real-world gains over the H100 on identical compute.

·      Context Length and the KV Cache

This is the constraint that surprises teams most often. The KV cache scales with the product of the context length and the number of concurrent requests, and it competes with model weights for GPU memory. Moving from 8K to 128K context increases the cache requirement roughly sixteen-fold per request. On a 640 GB H100 node, a 70B FP16 model consumes most of the memory before any cache is allocated, forcing low concurrency or aggressive quantization. On a 2.1 TB B300 node, the same model leaves room for genuine concurrency at long context. That is an argument about capacity, not speed.

·      Concurrency and Cost per Million Tokens

Evaluate an LLM server on cost per million tokens at your target latency, not on peak throughput. A B300 node costs far more than an H200 node, but if it supports four times as many concurrent users within your latency service level, the cost per token can be lower. With modest concurrency and short contexts, the same node remains underutilized, incurring the worst cost per token in the lineup. The AI infrastructure decision is an economics decision wearing an engineering costume.

LLM Fine-Tuning and Training: What Fits Where

LLM fine-tuning has a harder memory profile than inference because you are holding weights, gradients, optimizer states, and activations simultaneously. A useful rule of thumb for full fine-tuning in mixed precision with the Adam optimizer is that you need somewhere in the region of sixteen to twenty bytes per parameter once optimizer states and activation memory are accounted for. That arithmetic, not the FLOPS figure, decides what fits on a node.

1. LoRA and QLoRA: Accessible on Every System Here

Parameter-efficient fine-tuning changes the economics entirely. LoRA and QLoRA update a small adapter rather than the full weight set, cutting memory requirements by an order of magnitude. A DGX H100 can LoRA-tune a 70B model; a DGX H200 can do so comfortably with longer sequences. For most enterprise LLM fine-tuning work, such as domain vocabulary, tone, and structured output formats, this suffices, and buying Blackwell Ultra for it over-specifies the job.

2. Full Fine-Tuning: The Memory Cliff

Full fine-tuning is where the systems genuinely separate. Apply that bytes-per-parameter rule to a 70B model and the job wants well over 1 TB of GPU memory, beyond a 640 GB DGX H100 node, marginal on a 1.1 TB H200, comfortable on a 1.4 TB B200, and straightforward on a 2.1 TB B300.

3. Pretraining and Multi-Node LLM Training

For pretraining from scratch or for continued pretraining on large corpora, single-node capability is no longer the question; cluster scaling begins. This is where ConnectX-8 and the 1.6T optics on Blackwell Ultra justify themselves: gradient synchronization across dozens of nodes is bandwidth-bound, and a fabric that scales poorly converts a 30-day run into a 45-day run. Budget for high-throughput storage feeding the pipeline and a network switching layer sized for the fabric; an underfed GPU cluster is an expensive idle asset.

Where RAG and Agent Workloads Put the Pressure

A RAG pipeline is prefill-dominated: it injects retrieved documents as context, so effective prompt lengths of 16K to 64K tokens are common even for short answers. That makes RAG a memory-bandwidth and KV-cache problem, favoring H200 over H100 and B300 over B200. It is also a storage problem: vector indexes and document corpora require low-latency enterprise storage, and the retrieval stage is CPU-bound in ways worth understanding. Our explainer on how GPUs and CPUs differ in processing workloads is a useful primer for sizing the host side.

AI agents are the workload that most clearly justifies Blackwell Ultra. An agent loop issues many sequential model calls, each carrying accumulated state, and each call adds latency that the user experiences serially. Holding full model weights and long-lived context in memory without offloading or recomputation is exactly what 288 GB per GPU enables. If your roadmap is agentic within 18 months, size for B300 now; retrofitting power and cooling later is more disruptive than specifying it upfront.

GPU Memory, HBM and Memory Bandwidth: The Real Decision Axis

If you take one principle from this NVIDIA DGX comparison, take this: GPU memory determines what you can run, memory bandwidth determines how fast, and compute determines how many users at once, in that order. HBM sits stacked beside the GPU die, which is what allows multi-terabyte-per-second figures that conventional DDR-based server memory cannot approach. It is also fixed at purchase: you cannot expand a DGX node’s HBM later, so the capacity you buy is the capacity you keep.

Networking, Storage and the Rest of the GPU Cluster

A common and expensive failure pattern is buying flagship GPU nodes and connecting them with standard enterprise networking. Multi-node training is bandwidth-bound; a 200 GbE fabric behind a Blackwell cluster wastes most of the compute you paid for. Budget InfiniBand or Spectrum-X-class Ethernet with 800G-1.6T per-node capability, and validate per-GPU bandwidth with engineering rather than assuming datasheet figures.

Storage is the other frequently underspecified layer. DGX B300 ships eight 3.84 TB NVMe drives at roughly 50 GB/s peak for a reason: checkpointing a large training run repeatedly writes hundreds of gigabytes, and slow storage stalls every GPU in the node. Plan high-capacity enterprise SSDs for both the local scratch and shared dataset tiers.

Do not neglect the host side. Data loading, tokenization, and retrieval all run on CPUs, and an undersized host starves the GPUs. Blackwell Ultra nodes pair with high-core-count Xeon 6 processors and very large system memory pools; if you are building an equivalent HGX-based AI server yourself, specify host DRAM and enterprise server components to match rather than treating them as an afterthought.

Rack Power and Cooling: Can Your Facility Host These Nodes?

The per-GPU power trajectory across these generations is the most consequential trend in the comparison: roughly 700 W on H100 and H200, 1,000 W on B200, and 1,400 W on B300. At eight GPUs per node, before networking, storage, and host overhead, that is a fundamentally different rack conversation at each step. Many enterprise data centers were designed around 10 to 15 kW per rack; Blackwell Ultra density pushes well beyond that.

Cooling follows directly. H100 and H200 nodes remain deployable with air cooling in a well-designed hot-aisle environment. B200 sits at the boundary, workable with air in some facilities and better with liquid assistance. B300-class density assumes liquid cooling as the design point; these systems center on direct-to-chip implementations that capture the vast majority of the heat, and air cooling cannot dissipate the thermal output at rack scale. This is a facility project with lead times measured in quarters, not a procurement line item.

The honest procurement sequence is workload first, memory second, facility third, price fourth. Teams that invert it buy a DGX server they cannot cool or a cluster they cannot feed. Decisions made at the rack server level constrain everything above them, so start there and work upward.

DGX Server Cost: The Seven-Line Items That Decide TCO

The DGX server cost is quoted per node, but no one operates a node in isolation. A defensible total cost of ownership model for a three- to five-year horizon needs at least seven components, and the accelerators are only the first.

• Hardware Acquisition: Node cost rises with capability across H100, H200, B200, and B300, and Blackwell Ultra costs notably more than the rest.
• Energy: The highest recurring cost and the one most often understated. At 1,400 W per GPU, an eight-GPU B300 node draws over 11 kW from the accelerators alone, before overhead, when running continuously. Multiply by your electricity rate and your utilization assumption, then multiply again by your power usage effectiveness.
• Cooling and Facility Works: Redesigns include liquid-cooling infrastructure, coolant and power distribution upgrades, and structural modifications. For Blackwell Ultra, this often represents the most substantial non-hardware line item.
• Networking Fabric: InfiniBand or Spectrum-X switching, including adapters, optics, and cabling, scales with the node count and, in the case of B300, also with the transition to 1.6T optics.
• Storage and Data Pipeline: High-throughput NVMe tiers, shared filesystem capacity, checkpoint storage, and backup.
• Software and Support: NVIDIA AI Enterprise, orchestration and management tooling, and vendor support contracts, typically charged annually.
• People: MLOps and infrastructure engineering time. The system that requires fewer workarounds to run your workload is cheaper even at a higher list price, and this is routinely the most underestimated item in the model.

Once those components are in the model, the ranking often changes. DGX B300 has the highest acquisition cost and the best throughput per megawatt, up to roughly 5× Hopper, so at high sustained utilization it can show the lowest cost per token. DGX H100 has the lowest entry cost and the worst efficiency, which is why the certified refurbished market makes it attractive for intermittent workloads rather than 24/7 production serving.

Six Questions That Narrow the Choice to One Node

Work through these in order. The first question that returns a hard constraint is your answer.

• What can your facility deliver? If you cannot supply at least 12 kW per rack with liquid cooling, DGX B300 is not available to you, regardless of budget. This eliminates more candidates than any other question.
• What is your largest model and longest context? Calculate the weights footprint along with a KV cache sized for your target concurrency and context length. If it doesn’t fit within a single node, you’re incurring multi-node complexity, so include the fabric cost right away.
• Is your workload prefill-heavy or decode-heavy? Prefill-heavy RAG and document processing reward Blackwell’s FP4 compute. Decode-heavy conversational serving rewards memory bandwidth and capacity, with H200 being a strong value and B300 the strongest.
• Are you training, fine-tuning, or serving? Serving quantized models at moderate concurrency: H200 or B200. Full fine-tuning at 70B and above: B200 minimum, B300 comfortable. Frontier-scale pretraining: B300 with a properly specified GPU cluster.
• Is your roadmap agentic or reasoning-based? If so, size the system for the largest memory capacity you can justify today. The agent state and chain-of-thought token volume grow more quickly than most capacity plans anticipate, and HBM cannot be added later.
• What is your utilization forecast? A node at 30% utilization has a terrible cost per token, whatever its specification. If you cannot keep a B300 busy, buy a B200 and add a second node when demand justifies it.

Final Thoughts

The NVIDIA DGX B300 vs B200 vs H200 vs H100 decision resolves cleanly once you stop comparing peak FLOPS and start comparing GPU memory, memory bandwidth, and cost per token at your latency target. The DGX H100 remains the affordable entry point to NVLink-connected data-center GPUs. DGX H200 is the memory upgrade that fits existing facilities. DGX B200 is the mainstream Blackwell choice for most enterprises. DGX B300 is the reasoning and agent platform for organizations whose economics depend on long-context throughput.

The recommendation that applies to every reader: define the largest model, longest context, and peak concurrency you must serve in 24 months, then buy the smallest system that meets it with headroom. Verify power and cooling before signing, and size networking and storage to match the GPUs. If you are specifying AI and GPU server hardware, comparing NVIDIA GPU options, or sourcing across the rest of the enterprise range, the components matter as much as the accelerators.

If you are still unsure which NVIDIA GPU server rack form factor is right for your business, contact our expert team for more details, or visit our website to request a bulk quote.

Frequently Asked Questions

1. What is the main difference between DGX B300 and DGX B200?

DGX B300 offers 288 GB HBM3e per GPU versus the B200’s 192 GB, roughly 2× attention performance and 1.5× dense FP4 throughput, at 1,400 W per GPU requiring liquid cooling.

2. Is DGX H200 still worth buying for LLM inference?

Yes. Its 141 GB of HBM3e memory and 4.8 TB/s memory bandwidth substantially improve decode-heavy LLM inference over H100 on existing power, cooling, and server networking infrastructure, without a facility upgrade.

3. Which DGX server is best for LLM fine-tuning?

DGX B200 suits most enterprise LLM fine-tuning, offering 1.4 TB node GPU memory for single-node 70B full fine-tuning. Choose DGX B300 for larger models or longer sequences.

4. How much does a DGX server cost to own?

DGX server cost extends beyond hardware. Total cost of ownership includes energy, liquid cooling, GPU cluster networking, enterprise storage, software licensing, and staffing, often rivaling acquisition cost over three years.

5. Do I need DGX B300 for RAG and AI agents?

Only at scale. RAG runs well on B200 or H200, but AI agents that handle large context windows benefit most from the B300’s 288 GB of per-GPU memory headroom.