These days, when a new AI model comes out, its announcement often lists two numbers side by side. The Solar Open2 model card is like that, with two cells: "total parameters" and "active parameters."
I knew that parameters tell you the size of a model, but active parameters? At first I had no idea what that meant.
The hint was in the "architecture" cell right before it. The two numbers come from a model architecture called MoE (the "Hybrid-Attention" part in front of it isn't something I'll cover this time). A good share of today's big models, like DeepSeek, Qwen, and gpt-oss, use this architecture, so once you know MoE, news about new models reads a lot more easily. So this time I went and studied MoE properly.
MoE is short for Mixture of Experts. Just as the name suggests, MoE splits part of an AI model into several pieces called "experts," and for each small chunk of the incoming text, it picks only the few experts it needs and puts them to work.
A general hospital makes this easy to picture. Scene 1 shows the reception desk guiding a patient only to the department that fits their symptoms.
A big hospital has specialists in many departments, but a single patient doesn't see every doctor. The reception desk hears the symptoms and points the patient to one or two matching departments. MoE works the same way. A router, which plays the role of the hospital's reception desk, looks at an incoming token and decides which experts to send it to, and only the chosen experts do the computation.
Now let's look again at the Solar Open2 model card specs from the intro. Total parameters were 250B, and active parameters were 15B. Plugging that into what we just learned, all the specialists on duty are 250B, and the ones who actually treat a single patient are 15B.
This time, instead of the hospital, let's look a little more closely at how a real model works. When a token comes in, the router gives each expert a recommendation score for how well suited it is to handle this token. Then only the top few experts by score actually compute. The rest sit this token out. In Figure 1, if you follow the differently colored line for each chunk, you can see the two experts that chunk picked.
In Figure 1, two of the eight experts worked, but a real large model keeps tens to hundreds of experts and has only a few of them work at a time. How many experts you design each token to pick also changes the active parameters. Even with the same 250B total, the active parameters could be 20B if each token picks more experts, or 10B if it picks fewer.
In the past, most large AI models were dense rather than MoE. In a dense architecture, every time a token comes in, the whole model computes together. When models were small, this was fine, but a problem showed up as people grew the parameters to build smarter models. Computation grew right along with the parameters, so processing a single token meant computing even the parts that weren't really needed each time. People needed a way to grow the knowledge without growing the computation, and that's how MoE came into the spotlight.
Scene 2 imagines what Scene 1's hospital would look like if it ran like a dense model.
In a dense model, a single token passes through the whole model. For the hospital, that's like a patient who came in with a stomachache having to stop by internal medicine, surgery, and ophthalmology, one after another. Sure, while looking at the stomachache patient's eyes, ophthalmology might stumble on an unexpected clue, but honestly that's rare. Most of the time, just going to internal medicine gives nearly the same diagnosis as visiting every department. That's exactly what MoE is after: computing only a few matching experts per token without losing much answer quality.
As we saw, MoE's biggest strength is that it cuts the time and resources computation takes. But there's one thing to keep in mind. Even if only one or two doctors are seeing patients, every specialist has to be on duty, because you never know which patient will walk in.
A real MoE model is the same. To run an AI model, besides computation you need memory, the space where the parameters sit. Since you can't know in advance which expert the next token will call, every expert has to stay loaded in memory at all times. So computation scales with the active parameters, but memory scales with the total parameters. In other words, using MoE doesn't shrink the model's overall size. To run Solar Open2 as-is, storing the parameters alone takes 500.6GB, which needs eight H100 GPUs.
In my earlier quantization explainer, I mentioned that what Nota AI is good at is quantization, making big models smaller while keeping their performance. MoE models are also large overall, so to actually use them, they often have to be made smaller with quantization. But MoE models are much trickier to quantize than ordinary models.
Do you remember what plays the role of the reception desk in MoE? The router. The router has to pick the right expert for each token. But when quantization makes the numbers inside the model slightly less precise, the recommendation scores the router gives shift a little too. That can make it pick a different expert than it originally would. For the hospital, it's like a reception clerk who dieted to get lighter and lost some energy, sending a stomachache patient who should go to internal medicine to surgery instead. A surgeon can handle a stomachache to some degree, of course, but probably not as well as an internal medicine doctor.
There's one more reason MoE quantization is hard. Before quantizing, you run calibration, feeding in sample data to measure what range the numbers inside the model fall into. But if you only feed in ordinary data, some experts almost never get called.
That got a little technical. Putting it back in hospital terms, patients with colds are common, so internal medicine specialists get called often, but a burst appendix is rare, so the surgeon who performs appendectomies almost never gets called. Since there's no way to measure how a barely-called specialist works, that specialist may not treat an appendix patient properly after quantization.
So when quantizing an MoE model, the key is to keep the router picking the same experts as before, and to properly measure even the experts that are barely called. Nota AI is solving these two problems this way.
And the case where Nota AI used both of these methods together is Solar Open2, which I mentioned at the start. By adding pruning (a technique that trims away less important parts) on top of quantization, Nota AI shrank the original 500.6GB down to 117.8GB, so a model that needed eight H100 GPUs now runs on two. It also kept more than 99% of the original score on hard reasoning tasks (average score: 83.99 → 83.39).
We started with just two numbers, but from the router to memory to quantization, there turned out to be more to know than I expected. Still, this is all you need to remember. Computation scales with the experts you pick; memory scales with all of them! And one more thing: the place that shrinks this complicated MoE while keeping its performance almost intact is, of course, Nota AI!
No. MoE is a design choice you make when you first build a model. The compression techniques that make a model lighter are quantization and pruning. If anything, MoE models have large total parameters and use a lot of memory, so in practice they often need compression even more.
Not necessarily. What an expert takes on isn't assigned by a person; the model splits it up on its own as it learns. So rather than landing neatly as "the math one" or "the coding one," they often split along lines that are hard for a person to name, like words of a similar shape or similar sentence structures. The internal medicine and surgery in the hospital analogy are just an example to show that "the work is divided up."
It varies by model. Mixtral 8x7B picks 2 of 8 experts, gpt-oss-120b picks 4 of 128, Qwen3-235B picks 8 of 128, and DeepSeek-V3 picks 8 of 256. The more experts there are, the smaller the fraction it picks tends to be.
It does get faster as computation drops, but not by exactly that ratio. You have to keep all the experts loaded in memory and pull out each needed expert's parameters on the spot, and when they're split across several GPUs, it also takes time for the GPUs to exchange data.
Yes. An ensemble takes several separate, finished models, has each one solve the same problem in full, and then gathers their answers, so computation grows with the number of models. MoE splits part of a single model into experts and has only a few of them handle each token, so computation doesn't grow much.
The upside is that you can grow the amount of knowledge (total parameters) a lot while keeping the computation for a single token small. The downside is that you have to keep all the experts loaded in memory, so it uses a lot of memory, and shrinking it with quantization is trickier than for an ordinary model too.
Further reading
![[Nota AI Server] Building an On-Premises LLM: What to Check Before You Buy GPUs](https://cdn.prod.website-files.com/69d764b1be3398a43fb536b5/6ad71055243908f9649e9759_en_thumbnail.webp)
