[NetsPresso® AQ Deep Dive] How to Quantize the K-EXAONE-236B Model to 4-Bit (NVFP4)

Open-source quantization tools have made rapid progress recently. But deploying a real production service is a demanding domain, one that rarely tolerates even a 0.1-second slowdown or a 0.1% drop in accuracy. Add fragmented hardware requirements to the mix, and the constraints become tightly interlocked. However strong any single tool may be on its own, controlling all these entangled variables precisely enough to break through the limits is hard.

AQ (Advanced Quantizer), the quantization module of the AI model optimization platform NetsPresso®, is an optimization engine that freely combines multiple quantization techniques to fit the situation and fuses them into a single flow. It is not tied to any particular model architecture or hardware; it weaves scattered open-source techniques together under one configuration. This lets it control complex constraints more precisely, constraints that older approaches could only work around by compromising.

The case in this article covers quantizing 'K-EXAONE-236B-A23B', a frontier-class MoE model from LG AI Research, to NVFP4 (microscaling 4-bit floating point) with NetsPresso® AQ, tuned for a serving environment on FuriosaAI's datacenter NPU (neural processing unit). The result keeps accuracy on par with the original BF16 while compressing the model by roughly 3.5x, and the output is served directly in a vLLM environment with no extra kernels.

Rather than applying algorithms blindly, NetsPresso® AQ diagnosed the model's structural characteristics and fused techniques step by step. We walk through that engineering process one piece at a time below.

※ You can see the full flow NetsPresso® runs, from quantization to deployment.

Why 4-Bit Quantization of MoE Models Is Hard: Outliers in the Final Two Layers

A Mixture-of-Experts (MoE) model holds many expert networks inside and selectively activates only some of them depending on the input. This K-EXAONE-236B-A23B is exactly that, as its name says: an MoE model that activates only 23B of its total 236B parameters per token.

This structure raises inference efficiency well beyond what the model's sheer size would suggest, but at the 4-bit quantization stage it turns into the single largest constraint. Weight distributions differ widely across layers and experts, so limited precision is hard to allocate evenly across the whole model.

This structure raises inference efficiency well beyond what the model's sheer size would suggest, but at the 4-bit quantization stage it turns into the single largest constraint. Weight distributions differ widely across layers and experts, so limited precision is hard to allocate evenly across the whole model.

‍

Here too, a full analysis of roughly 227 billion expert weights showed that the share of outliers with an absolute value above 0.1 was up to about 250x higher in the final two layers (46, 47) than in the earlier layers. Outliers this concentrated distort the NVFP4 block scale and degrade the precision of neighboring weights as well.

In large MoE models, uneven distributions across experts or layers and the outliers that come with them are a common problem that raises quantization difficulty. Where outliers concentrate varies by model architecture and training method, and in the K-EXAONE case they concentrated in the final two layers. Because frontier-class models each carry their own outlier distribution map, how precisely you can diagnose and control it is the shared core challenge of quantization.

The Standard NVFP4 Serving Spec: Quantizing With a Single Format and No Detours

When this kind of concentration shows up, the usual move is to sidestep it: exclude those layers or experts from quantization and leave them in high precision (BF16 or INT8), or reach for a workaround such as mixed precision, which assigns different bit widths by weight sensitivity.

In this case, though, quantization had to follow the standard NVFP4 serving path as-is, with no extra runtime kernels or exception handling. That meant setting aside the usual workarounds, applying one identical optimization policy across all 47 MoE layers, and building the model in the single NVFP4 format alone.

The nature of the NVFP4 data type itself added to the difficulty. NVFP4 has a double-scale structure, a global scale combined with a block scale, so precision is easily damaged without careful control. In the end, the task was to satisfy a demanding format constraint with every workaround off the table.

Offline Fusion of Multiple Quantization Algorithms: NVFP4 Quantization That Holds Precision

To hold precision inside the standard NVFP4 spec with every detour closed off, NetsPresso® AQ performs an offline fusion that combines the several algorithms it needs and ties them into a single flow. Applying complementary transforms separately makes it hard to hit two conflicting goals at once, spec compliance and error control, but interlocking them into one flow satisfies both.

Instead of adding extra operations at inference (runtime) to reduce error, AQ transforms the coordinate system of the weights themselves just before quantization, so the correction is baked into the weights ahead of time. This idea of reshaping the distribution through coordinate-system rotation is close to TurboQuant, which recently drew attention for KV cache compression. The difference is that TurboQuant does this at runtime.

In general, chaining several transforms of different natures in sequence without inference-time involvement risks corrupting the original model's function, or triggering extra operations that push the model off the standard serving path. NetsPresso® AQ is built to control exactly this risk. The three offline transform techniques applied in this optimization are as follows.

• Rotation: Rotates the weight coordinate system so outliers do not pile up in one place. This step effectively mitigates the outliers that NVFP4 is especially vulnerable to.
• Outlier migration: Moves the outliers that remain after rotation onto another axis, lowering the load on the block scale one more time.
• Quantization: Quantizes these conditioned weights to NVFP4.

The output tied together this way is saved just like an ordinary NVFP4 model and runs directly on the native vLLM path, with no extra kernels.

These transforms all dissolve into the weights themselves. That is, no transform runs separately at inference time, as confirmed by the empty runtime-transform field in the public model card.

BF16 Accuracy Held at 4-Bit: The GPQA, IFBench, and AIME 2025 Benchmarks

We compared the accuracy of the quantized model made this way against the original BF16. The three benchmarks evaluate scientific reasoning (GPQA Diamond), instruction following (IFBench), and math problem solving (AIME 2025), respectively.

For all three benchmarks, the difference before and after quantization sits within the margin of error. In other words, the model shrank by about 3.5x while accuracy stayed on par with the original BF16. This carries particular weight because it was achieved under conditions that ruled out any workaround, such as mixed precision or excluding sensitive layers.

This shows that NetsPresso® AQ's combination approach holds up even for frontier-class MoE models. The same approach can extend to other MoE models and to NPU serving.

Compression also cuts the memory needed to load the model, so a model of the same scale can be served with less memory. Inference for LLMs, and MoE models in particular, is memory-bound rather than compute-bound, so simply shrinking the model speeds up inference by roughly 1.5 to 2x. In datacenter infrastructure, it also opens room to handle more requests on the same accelerator memory.

NetsPresso® AQ: An Engine That Quantizes Any Model Into a Deployable Form

AQ (Advanced Quantizer) is NetsPresso®'s optimization engine that weaves together standard quantization algorithms such as AWQ, GPTQ, AutoRound, QuaRot, SpinQuant, and SVDQuant, as needed. It is not tied to any particular architecture and covers a wide range, from LLMs and MoEs to VLMs, audio, and diffusion models. These algorithms are usually scattered across different open-source projects with different interfaces and formats, so using them together means installing each one and hand-stitching the data conversions at every step.

AQ unifies this process into a single flow. Output optimized this way is served in production in the specified format as-is, with no extra kernels. The strength of NetsPresso® AQ is that it handles ordinary quantization under forgiving conditions and, through the same pipeline, cases like this K-EXAONE project where the standard NVFP4 spec must hold and every detour is closed off.

If you want to apply model compression, optimization, and deployment, including frontier MoE quantization, to fit your own environment, get in touch with NetsPresso®.

※ The benchmark figures in this article are results from a specific combination of model, hardware, and settings; actual performance may vary by environment.