How We Compress Large-Scale MoE Models

Highlights
🔹 Why Compression Now Decides Adoption
🔹 Solar-Open-100B: 102B Parameters, 72.8% Smaller
🔹 Solar Open 2: 250B Parameters, 76.5% Smaller
🔹 K-EXAONE-236B-A23B: 236B Parameters, 71% Smaller
🔹 One Pipeline, Ready for the Next Model

Why Compression Now Decides Adoption

This year, Nota AI compressed three large-scale MoE models. It started in March with 102 billion parameters, and continued in July with 250 billion and 236 billion — 588 billion parameters in total. This issue brings that record together in one place.

As high-performing open-weight models such as DeepSeek and Moonshot AI's Kimi series draw growing market attention, the basis of AI competition is expanding beyond raw performance toward practical agentic usability and operating cost efficiency. For many enterprises, the bigger challenge is shifting away from obtaining a model and toward running it within their real infrastructure and cost constraints — and MoE, now the default at this scale, lightens the computation without lightening what has to sit in memory.

In the end, adoption is decided less by benchmark scores than by a single question: does this model fit on the infrastructure we already have? That is exactly the problem Nota AI has focused on this year. The three models differed in structure, and so did the hardware they were validated on — yet every one of these optimizations was built on NetsPresso®, Nota AI's model optimization platform.

Solar-Open-100B: 102B Parameters, 72.8% Smaller

Rerouting, the hidden variable

Nota AI is participating in Proprietary AI Foundation Model Project led by the Ministry of Science and ICT, as part of the Upstage consortium. Solar-Open-100B is Upstage's flagship open model and the project's phase 1 result: an MoE architecture trained on 19.7 trillion tokens.

There is a point that is easy to overlook when standard quantization techniques are applied to MoE as-is. Keeping a decoder block's output close to the original is not enough to preserve performance reliably. In an MoE, the router decides where each token goes by selecting the top-k experts, and when quantization perturbs the router logits, the selected set of experts itself changes — rerouting occurs. Routing can be affected even without quantizing the router layer, purely as a result of quantizing the self-attention layers. Rerouting does not end as a simple continuous-value error: because it changes the very computational path a token travels, its effect on block output is amplified non-linearly.

NMQ: matching the decision structure, not the output

Rather than matching output values, Nota AI chose to match the decision structure itself. NMQ (NotaMoEQuantization) keeps the top-k expert set the router selects, the corresponding probability and score distributions, and the ranking and boundary relationships among experts as close to the original as possible after quantization. It is less a standalone quantization algorithm than a loss function used when selecting or tuning quantization parameters, which makes it possible to plug NMQ into a wide range of quantization methodologies.

The results show up on both fronts, size and performance. Memory usage came down from 191.2GB to 51.9GB — roughly 72.8% — while perplexity landed at 6.81, close to the original model's 6.06. Given that some general-purpose quantization techniques degraded performance more than fivefold, that is a meaningful gap.

The effect was most pronounced in long reasoning, where each step's routing decision continuously shapes subsequent representations and next-token predictions: measuring how closely the models before and after quantization select the same experts, NMQ scored above the baseline across every layer, with the largest gap exceeding 11.29%. On an NVIDIA B300 SXM6 with vLLM, NVFP4 also delivered nearly 2x better time to first token than W4A16 — a difference that grows more noticeable as inputs get longer and the prefill stage takes up more of the work. The model was selected as the official quantized version of Upstage's Solar-Open-100B and released on Hugging Face.

📄 Curious how the loss function was designed and how the experiments were run? Read the full methodology on the Nota AI blog: NotaMoEQuantization: An MoE-Specific Quantization Methodology Applied to Solar-Open-100B

Solar Open 2: 250B Parameters, 76.5% Smaller

Four bits could not get eight GPUs down to two

Solar Open 2 is built for enterprise-grade agentic AI — document processing, coding, tool calling, multi-step reasoning — and activates only 15 billion of its 250 billion total parameters at inference. It is more than twice the size of the phase 1 model, and agentic workloads consume a lot of tokens. Even with the compute efficiency of MoE, running the original model requires eight NVIDIA H100s, and converting weights to 4 bits alone does not meaningfully move that number.

Adding pruning to quantization

Holding to the router-preservation principle validated in phase 1, the approach expanded into three techniques. PASCAL-MoE, a data-driven MoE quantization method, synthesizes calibration data so that every expert module retains stable statistical characteristics while converting weights into INT4 or NVFP4-based 4-bit formats. Router-aware quantization, Nota AI's own algorithm, keeps the router's expert selection steady after quantization. And non-uniform global pruning goes a step beyond quantization-centric optimization, selectively removing the less important expert modules themselves — not just lowering precision, but taking away the experts that are rarely used.

The result: weight memory dropped from 500.6GB to 117.8GB, roughly 76.5%. By memory capacity, a model that required eight NVIDIA H100s now runs on two. That figure is based on the memory required to hold the weights; the number of accelerators an actual deployment needs will vary with context length, batch size, and KV cache configuration. Tool calling, a core capability for agentic AI, remained stable after quantization. This is not simply a smaller model — it is one optimized into a form that can actually be deployed in enterprise environments. The compressed model is available on Hugging Face.

📄 Want the background behind this release and how the compressed model can be used? Read the full announcement → Nota AI Releases a Compressed Version of the Sovereign AI Foundation Model Solar Open 2

K-EXAONE-236B-A23B: 236B Parameters, About 71% Smaller

Outliers concentrated in the last two layers

K-EXAONE-236B-A23B is a frontier-scale MoE model released by LG AI Research, activating 23 billion of its 236 billion total parameters per token. Nota AI's task was to quantize it to NVFP4 (microscaling 4-bit floating point) with NetsPresso® AQ, tuned for a serving environment on FuriosaAI's data center NPU.

A full sweep of roughly 227 billion expert weights revealed an unusual distribution. The share of outliers exceeding an absolute value of 0.1 was up to about 250x higher in the final two layers (46 and 47) than in earlier layers — concentrated enough to distort NVFP4's block scale and drag down the precision of surrounding weights along with it.

The usual response is to route around the problem, excluding those layers from quantization and keeping them at higher precision, or applying mixed precision with bit widths varied by weight sensitivity. Neither option was available here: the model had to follow the standard NVFP4 serving path with no additional runtime kernels and no exception handling, applying an identical optimization policy across all 47 MoE layers to produce a model in a single NVFP4 format.

Three transformations, all completed before inference

Instead of adding computation at inference time to reduce error, NetsPresso® AQ transforms the coordinate system of the weights themselves just before quantization, folding the correction in advance. Rotation turns the weight coordinate system so outliers do not cluster in one place, easing the problem NVFP4 is especially vulnerable to. Outlier migration moves whatever outliers remain onto other axes, lowering the block-scale burden one step further. Quantization then converts these organized weights to NVFP4. All three are absorbed into the weights themselves; nothing extra runs at inference time.

Model size came down by roughly 71%, and even so the model retained about 99.2% of the original's accuracy on average across scientific reasoning (GPQA Diamond), instruction following (IFBench), and mathematical problem solving (AIME 2025) — with every workaround off the table. There is a side benefit as well: LLM inference, and the decoding stage that generates tokens sequentially in particular, is shaped more by memory bandwidth than by raw computation, and in this environment shrinking the model alone made inference roughly 1.5 to 2x faster. The output is stored exactly like any standard NVFP4 model and runs on vLLM's native path without additional kernels, so the complexity cost of deployment and maintenance comes down too.

📄 Want the outlier analysis and the transformation process in full? Read the complete technical deep dive on the Nota AI blog → [NetsPresso® AQ Deep Dive] How to Quantize K-EXAONE-236B to 4-bit (NVFP4)

One Pipeline, Ready for the Next Model

The models were different. The validation hardware was different — some served on NVIDIA GPUs, one targeted FuriosaAI's data center NPU. Yet all three were built on NetsPresso®, Nota AI's model optimization platform. Its quantization engine, AQ (Advanced Quantizer), selects and combines standard quantization and calibration techniques — AWQ, GPTQ, AutoRound, QuaRot, SpinQuant, SVDQuant — according to the model architecture and hardware conditions at hand, and on top of that foundation sit Nota AI's own MoE-specific techniques: NMQ, PASCAL-MoE, router-aware quantization, outlier migration. It is not tied to any particular architecture, covering everything from LLMs and MoE models to VLMs, audio, and diffusion models.

That is what makes the three results add up to more than three results. In the open-weight era, the differentiator has shifted from which model you use to the conditions under which you can run it. Anyone can download the weights; fitting them inside your own infrastructure and budget remains your own problem to solve. Beyond making models smaller, these projects optimized large-scale MoE models into a form enterprises can realistically adopt — and the real outcome is not the individual results, but the fact that the same pipeline is ready for the next model.

Nota AI will keep advancing quantization and lightweighting techniques tuned to each model architecture and hardware environment. The range is wide — from large MoE models in the data center to edge devices — but the goal is singular: to let enterprises adopt and scale AI within the infrastructure and budget they actually have.

🔹 Korea Efficient Days @ ICML 2026 — Nota AI brought the efficient AI community together with Korea Efficient Night on July 7 and Open Office Days on July 8–9, held alongside ICML 2026.

💡 Learn more about Nota AI: https://blog.nota.ai/
📩 Subscribe to Edge Insights: https://bit.ly/44XuDKX

‍