NVFP4 is a 4-bit floating point data format that NVIDIA introduced with the Blackwell GPU architecture for AI inference. It stores each value in the compact E2M1 layout and recovers accuracy through two-level scaling: an FP8 (E4M3) scale shared by every block of 16 values, plus a per-tensor FP32 scale. In our benchmarks, this design delivered up to 2× higher LLM inference throughput on an RTX PRO 6000 than on an A100 while keeping accuracy near-lossless.
In March 2024, NVIDIA announced Blackwell—its next GPU architecture—bringing a major update to the [5th-generation Tensor Cores], which now natively support FP4 (and FP6) for ultra-low-precision AI operation. This post focuses only on NVFP4, NVIDIA’s FP4 data format for Blackwell. (Blackwell also introduces “Tensor Memory,” a new on-chip memory tier that changes GEMM kernels, but that’s out of scope here)
Classic FP4 squeezes each value into 4 bits—great for bandwidth and compute—but risks accuracy. NVFP4 tackles that tradeoff with micro-block scaling. With this technique, values are grouped into blocks of 16, each sharing a high-precision FP8 (E4M3) scaling factor, plus an additional per-tensor FP32 scale. In practice, each 4-bit number is a tiny index and the scale is the “zoom level” that restores magnitude during compute;
where:
This two-level scheme preserves dynamic range and reduces quantization error while keeping the data tiny—exactly what large Transformer-based LLMs need at inference.
What is evaluated in this post?
Motivated by the hardware FP4 path and NVFP4’s layout, I measure LLM inference on a Blackwell workstation GPU, the RTX PRO 6000, and share where NVFP4 helps most (and where it needs care). The RTX PRO 6000 is a Blackwell-class pro GPU with 5th-gen Tensor Cores and 96 GB GDDR7, making it a practical target for On-Premise LLM serving.
NVFP4 runs natively on NVIDIA Blackwell-class GPUs and newer. Blackwell's 5th-generation Tensor Cores provide a hardware FP4 (and FP6) compute path, while earlier architectures execute FP4 workloads through emulation on higher-precision kernels, which removes the latency benefit.
Support has also widened since this benchmark: Blackwell Ultra (B300) carries NVFP4 forward, and the Rubin architecture announced at GTC 2026 pushes NVFP4 inference to 50 petaflops per chip through its third-generation Transformer Engine while remaining compatible with Blackwell. NVFP4 is settling in as a standard format across NVIDIA's inference stack rather than a single-generation experiment.
No. The A100 is an Ampere GPU, and Ampere Tensor Cores natively support FP16/BF16 as their highest-throughput compute path (~312 TFLOPS dense, 624 TFLOPS sparse[2]). FP4 and FP8 operations there are emulated through higher-precision kernels, so NVFP4 models load and run but gain no latency advantage over BF16. In our measurements below, the original BF16 model achieved the lowest TTFT on that GPU across all conditions.
NVFP4 differs from classic FP4 in one key way: micro-block scaling. Both keep 4 bits per value, and FP8 keeps 8, but how each format preserves dynamic range determines its accuracy behavior in practice.
MXFP4 is the open-standard (OCP Microscaling) sibling of NVFP4 that drew wide attention when gpt-oss shipped in it. It stores the same E2M1 values but with larger 32-value blocks and a coarser E8M0 scale format, and reports so far consistently favor NVFP4 on accuracy under matched conditions. For NVIDIA-only deployments NVFP4 is the natural default; MXFP4 becomes the option when models need to move across vendors.
Full configuration details are in the tables below.
On Blackwell (RTX PRO 6000), NVFP4 delivered the fastest prefill (TTFT) and the highest overall throughput in our tests, while on the A100 the BF16 baseline stayed ahead (measured on vLLM v0.10.1.1).
Since the Y-axis scales differ across panels (e.g., 3.0 s vs 2.0 s), cross-device absolute comparisons should be avoided; the focus here is on relative performance within each GPU.
TTFT-Batch size graph with fixed input token length (1024). The figure on the left side is results on A100 and right side is that on RTX PRO 6000.
TTFT-Input Token Length graph with fixed batch size (16). The figure on the left side is results on A100 and right side is that on RTX PRO 6000.
On A100 (Ampere), the Original (BF16) model achieves the lowest TTFT across all conditions. This is expected since A100’s Tensor Cores natively support FP16/BF16 as their highest-throughput compute path (~312 TFLOPS dense, 624 TFLOPS sparse[2]). FP8 and FP4 operations, on the other hand, are not natively supported on Ampere — they are emulated through higher-precision kernels,resulting in no latency advantage compared with BF16.
On RTX PRO 6000 (Blackwell), NVFP4-W4A4 consistently delivers the fastest TTFT, followed by NVFP4-FP8-Dynamic. Blackwell’s 5th-generation Tensor Cores natively support FP4/FP6/FP8, allowing NVFP4 workloads to fully utilize low-precision compute and bandwidth efficiency. As batch size or input sequence length grows, the matrix-multiply load and memory pressure increase, amplifying the benefits of native FP4 execution. This demonstrates Blackwell’s strong prefill-phase efficiency compared with earlier architectures.
However, it is important to note that software and kernel maturity (e.g., vLLM runtime and NVFP4/FP8 kernel implementations) can affect the absolute margins shown here. As frameworks and kernels evolve, absolute TTFT values and relative gaps may shift, so these results should be interpreted as relative trends based on vLLM v0.10.1.1.
TPOT-Batch size graph with fixed input token length (1024). The figure on the left side is results on A100 and right side is that on RTX PRO 6000.
TPOT-Input Token Length graph with fixed batch size (16). The figure on the left side is results on A100 and right side is that on RTX PRO 6000.
Across both GPUs, a consistent trend emerges: the lower the numerical precision, the smaller the TPOT (Time-Per-Output-Token). This reflects the inherent nature of the decode phase, where the introduction of the Key–Value (KV) cache reduces the computational weight of matrix multiplications per token, shifting the dominant bottleneck from computation to memory access. As a result, formats that minimize data movement make more efficient use of available memory bandwidth, producing shorter latency per token.
When comparing the two devices, A100 (Ampere) generally shows lower TPOT than RTX PRO 6000 (Blackwell) across most conditions. This is primarily due to the A100’s higher memory bandwidth (≈2,039 GB/s via HBM2e) compared to the RTX PRO’s GDDR7 bandwidth (≈1,792 GB/s), which provides a clear advantage in a memory-bound scenario. However, as the input length or batch size increases, the workload transitions from memory-bound to compute-bound. In this regime, Blackwell’s native FP4/FP8 Tensor Core pathways begin to show their strength, leading to cross-over points where NVFP4-based models surpass higher-precision ones—similar to the pattern observed in the prefill phase.
An intriguing observation is that on RTX PRO 6000, not only AWQ-W4A16 but also NVFP4-W4A16 achieves lower TPOT than fully lower-bit NVFP4 variants such as NVFP4-FP8-Dynamic and NVFP4-W4A4. In the case of AWQ, this can be readily explained by the use of the specialized AWQ-Marlin kernel, which is explicitly beneficial for small/moderate-batch decode workloads, reducing launch overhead and improving scheduling and memory reuse.
For NVFP4-W4A16, however, the explanation lies elsewhere: at the time of this experiment (vLLM v0.10.1.1) it lacked a Marlin-style kernel; instead, it compresses only the weights into FP4 while retaining FP16 activations. This allows it to execute on the mature FP16 accumulation path with well-optimized GEMM kernels.
In contrast, FP4/FP8 kernels—while natively supported on Blackwell—remained, at the time of testing, in relatively early stages of optimization within vLLM and CUTLASS. These kernels can incur non-negligible overhead from block-scale application and dequantization steps. Furthermore, since decoding is dominated by KV-cache activation reads rather than repeated weight matmuls, the advantage of fully quantized activations becomes limited, and the extra dequantization operations may even offset their theoretical gains.
Consequently, NVFP4-W4A16 could outperform fully lower-bit NVFP4 configurations in practical latency measurements under the software conditions of this experiment. Since then, vLLM has been adding dedicated NVFP4 kernel backends (CUTLASS, Marlin, and FlashInfer families), so this gap should narrow as the frameworks mature.
Throughput-Batch size graph with fixed input token length (1024). Solid lines indicate best cases for RTX PRO 6000 and dot lines means best cases for A100.
Throughput-Input Token Length graph with fixed batch size (16). Solid lines indicate best cases for RTX PRO 6000 and dot lines means best cases for A100.
Building on the findings from the Prefill (TTFT) and Decode (TPOT) phases, the throughput results can be seen as a summary metric that unifies both behaviors. While TTFT reflects the latency of reading and processing the input prompt, and TPOT captures the per-token generation efficiency, throughput represents the overall capability of the inference pipeline as a whole. For small batches, kernel launch and memory latency dominate, but as the workload increases, those effects diminish, and the hardware operates closer to its true throughput limit.
On the A100 (Ampere), both Original (BF16) and AWQ-W4A16 configurations show steady but moderate scaling, reaching around 4,000–5,000 tokens/sec. As discussed in TTFT/TPOT analyses, this reflects the lack of native FP4/FP8 execution on Ampere — the GPU simply cannot fully exploit the benefits of ultra-low-bit computation, and its FP16/BF16 path remains the most optimized and efficient.
In contrast, the RTX PRO 6000 (Blackwell) demonstrates clear superiority. All NVFP4-based models outperform the A100 across the board, with NVFP4-W4A4 and NVFP4-FP8-Dynamic reaching 6,000–8,000 tokens/sec, roughly 1.7–2× higher throughput than A100 under identical conditions. This outcome directly mirrors our earlier phase analyses: the Prefill advantages of native FP4/FP8 computation combine with the Decode-phase memory efficiency to produce large end-to-end gains. In short, Blackwell’s 5th-generation Tensor Cores natively accelerate FP4/FP8 math, drastically increasing arithmetic density.
Between the NVFP4 variants, NVFP4-FP8-Dynamic overtakes NVFP4-W4A16 as batch size or input length grows, because quantizing activations to FP8 significantly reduces memory traffic and scaling overhead becomes negligible in large compute-bound workloads. However, in smaller batches, the dynamic-scaling cost can offset those benefits, making NVFP4-W4A16 slightly more efficient.
Through the experiments, Blackwell scales its performance consistently with workload size, showing that its low-precision NVFP4 format delivers a truly scalable inference throughput. Together, these results confirm that the combination of Blackwell architecture and NVFP4 quantization unlocks sustained efficiency across the entire inference pipeline, offering a decisive advantage for next-generation LLM workloads.
NVFP4 shows no practical accuracy loss: across HellaSwag, MMLU, and PiQA, score differences among all precision formats stay within ±0.005–0.02, and only Winogrande shows a slightly larger, task-level variation.
The academic benchmark results complement the latency and throughput analyses by revealing how quantization affects model quality in practice. Across HellaSwag, MMLU, and PiQA, accuracy differences among all precision formats are within ±0.005–0.02, effectively negligible both statistically and practically. Such small variations can easily arise from non-deterministic factors such as random seeds, prompt ordering, or tokenization details.
These findings are consistent with prior literatures which are most practical nowadays. Lin et al.[3] showed that 4-bit AWQ quantization preserves `near-lossless` (0.1-0.7 score drop) performance; Xiao et al.[4] reported <0.3% deviation after 4-bit/8-bit quantization; This confirms that Blackwell’s NVFP4 format achieves up to 2× higher efficiency while maintaining near-lossless accuracy compared to FP16/BF16 baselines. For a hands-on look at how this holds at a much larger scale, see our case study on [quantizing K-EXAONE 236B with NVFP4].
Only Winogrande shows a slightly larger variation, likely due to its linguistic reasoning complexity. Winogrande evaluates commonsense pronoun resolution—one of the most semantically sensitive benchmarks—where small precision perturbations can directly alter contextual understanding. Its relatively small dataset size and sentence diversity also make accuracy more volatile, amplifying minor numerical effects. Hence, these deviations should be interpreted as task-level sensitivity rather than genuine model degradation.
That said, the benchmarks used here are not fully representative of real-world LLM deployment scenarios. While HellaSwag, MMLU, PiQA, and Winogrande capture general reasoning and commonsense tasks, they do not comprehensively cover large-scale knowledge retrieval, multi-turn dialogue, coding, or mathematical reasoning. Still, it is remarkable that NVFP4 maintains almost lossless accuracy even under such an extremely low-bit (FP4) setting, and this suggests strong potential for further optimization—for instance, through better scaling strategies, calibration sets, or structure-aware quantization—to make NVFP4-based compression virtually lossless in practice.
This experiment demonstrated that Blackwell’s architecture and the NVFP4 data format can dramatically improve LLM inference efficiency. Native FP4 computation in the prefill phase and enhanced memory efficiency during decode together achieved up to 2× higher throughput compared to A100 (Ampere), while accuracy degradation remained statistically insignificant. These results confirm that NVFP4 delivers the practical benefits of ultra-low-precision computation without compromising FP16/BF16-level quality. This indicates that low-precision arithmetic is not merely an optimization trick but a viable acceleration technology for large-scale language model inference, offering a clear path toward improved efficiency and cost-effectiveness.
Nota has already been actively developing Vision-Language Model (VLM)–based solutions and is extending the findings of this study to optimize NVFP4 inference for multimodal workloads. The methodology has already moved into production work: the K-EXAONE 236B NVFP4 quantization and the MoE-specific quantization research introduced above are its first results. Because visual features are generally more continuous and densely distributed than textual ones, more nuanced optimization is required.
(1) researching quantization techniques specialized for multimodal architectures
(2) developing NVFP4-optimized GEMM/GEMV compute kernels and scheduling methods
(3) conducting analysis and specialized optimization studies of the Tensor Memory hierarchy, which was not explored in this work
With these possible future works, Nota will lead the way to real-time, high-efficiency inference for next-generation large-scale models.
If you have any further inquiries about this research, please feel free to reach out to us at following 📧 email address: contact@nota.ai.
Furthermore, if you have an interest in AI optimization technologies, you can visit our website at 🔗 netspresso.ai.
