🔹 Agent Operating Costs and Guardrails
🔹 Ultra-Large MoE Models on Four GPUs
🔹 MetaM: Extending Optimization to Agent Operations
🔹 Optimizing Model, Serving, and Infrastructure Together
Enterprise adoption of AI agents is accelerating. Salesforce analyzed monthly data from companies running Agentforce agents in production between February 2025 and April 2026: the average number of active agents per organization rose from five to 13, close to triple the starting number. Gartner, meanwhile, projected in June 2025 that more than 40% of agentic AI projects would be canceled by the end of 2027, citing high costs, unclear business value, and inadequate risk controls.
What stands out is that none of the reasons Gartner cites has to do with model performance. They are operational: cost, business value, risk control. Deploying agents in real services is not only a matter of making models smarter. It also means running many agents at predictable cost, and keeping their behavior and outputs reliably under control.
In this issue of Edge Insights, we examine why agent operating costs climb so quickly, and how Nota AI lowers the infrastructure burden of ultra-large MoE (Mixture-of-Experts) models such as Kimi K3 and Qwen3.8-2.4T-A95B. From there, we trace how that optimization capability is extending into multi-agent operating environments and integrated AI infrastructure, through the joint project with MetaM and through Nota AI Server.
Agents consume far more tokens than chatbots. A chatbot mostly answers questions. To complete a task, an agent plans, retrieves information, acts, checks the result, and retries when needed. Each loop adds model calls and tokens. According to Gartner, a single agent task requires 5 to 30 times more tokens than a typical generative AI chatbot request.
Nota AI's own experiment showed the same pattern. When the decisions involved in model quantization were handed to an agent end to end, with no guardrails in place, a single experiment consumed roughly 500 million tokens and $1,074 in API costs. The agent tried 13 quantization techniques. The conversion itself succeeded for most of them, but eight had no dedicated kernel to run efficiently on the target device. Actual inference ran at one to two tokens per second, and the system still logged all of them as successes.
The experiment shows that controlling an agent matters as much as expanding its autonomy. Without a clearly defined decision scope, success criteria, target-device compatibility, and external verification, tokens and GPU resources go to attempts that lead nowhere, and results that cannot realistically be deployed are recorded as successes. Running agents in production therefore calls for a system that validates cost, quality, and deployability alongside autonomy.
📄 Read the full experiment on the Nota AI Tech Blog
Running the ultra-large models that carry out long-horizon tasks and complex reasoning takes considerable compute. Kimi K3, the open-weight MoE model released by Moonshot AI, has 2.8 trillion parameters. Under the deployment configuration Nota AI published, serving the original model requires eight NVIDIA B300 GPUs.
Nota AI applied its own non-uniform expert pruning to build two lighter versions. The 25%-pruned model runs on six B300s; the 50%-pruned model runs on four. In Nota AI's internal evaluation, the 50%-pruned model scored higher on three of four public benchmarks than a REAP model pruned at the same ratio.
The significance here is not simply that the model is smaller. Cutting the required B300 count from eight to four halves the infrastructure needed to operate a model at this scale, which puts ultra-large models within reach for companies whose budget or infrastructure previously ruled them out.
Not every ultra-large model responds to the same technique, though. The right combination of methods changes with the model's training precision, architecture, and expert configuration. Qwen3.8-Max (Qwen3.8-2.4T-A95B), released by Alibaba in August, illustrates the difference. It is an MoE model with 2.4 trillion parameters that activates 95 billion per token; serving the BF16 original takes 24 B300s in Nota AI's deployment environment.
For Qwen3.8-Max, Nota AI combined NVFP4 (microscaling 4-bit floating point) quantization with non-uniform global pruning. Of the 47,104 experts across 92 MoE layers, 27,968 were kept and roughly 40% removed, bringing the GPU requirement from 24 down to four. Measured against the NVFP4 model without pruning, the AIME score held at 90.0, while GPQA-Diamond and IFEval came in higher, at 84.9 and 92.1.
Kimi K3 and Qwen3.8-Max started from different places. Kimi K3 was trained at low precision from the start, which left limited room for further quantization, so expert pruning alone cut its GPU requirement in half. Qwen3.8-Max, built on BF16, allowed quantization and pruning to be combined, taking it from 24 GPUs to four.
Together the two cases show that optimizing ultra-large models is not a matter of applying one compression technique uniformly. Model architecture, precision, and target hardware have to be analyzed together, and the combination of techniques designed for each set of conditions. Only then do GPU and memory requirements come down meaningfully without giving up performance. This is how Nota AI turns data-center-scale MoE models into infrastructure configurations that customers can actually adopt and operate.
📄 See Nota AI's ultra-large model optimization work in detail
On August 13, Nota AI began a collaboration with MetaM, a leading AICC (AI Contact Center) company, to build a next-generation AI customer service system. It is Nota AI's first commercial project in the enterprise AI agent operating infrastructure market. MetaM is a BPO (Business Process Outsourcing) specialist with more than 20 years of contact center operations experience.
The two companies are jointly building an "AI-native multi-agent customer service system," in which role-specific agents each take a part of the work: customer intent analysis, information retrieval, customer response, conversation summarization, knowledge management, and quality management. Nota AI is responsible for the shared AI engine that runs the agents and for the orchestration and RAG (Retrieval-Augmented Generation) layer, and will apply its own compression and optimization technology to the large language models and the inference engine. MetaM handles domain knowledge, the specialized functions of each role-specific agent, and AICC build-out, deployment, and operations.
.png)
Customer service is a prime example of a production environment where multiple agents must handle large volumes of real-time traffic. As the number of agents and model calls grows, compute load, GPU cost, and memory use grow with them. Accuracy alone is not enough. Compute efficiency, response speed, and memory footprint all have to be optimized for the conditions of the actual service.
The core of this project is reducing the compute and memory demands of the large language models and the inference engine, while holding response times steady even when many agents run at once. Through it, Nota AI's optimization scope extends from individual models to a live multi-agent operating environment.
Agent operating efficiency is not determined by model compression alone. For the same model, the GPUs and server specifications required change with the workload — request volume, concurrent users — and with serving settings such as batch size and KV cache. At the adoption stage, model, serving environment, and infrastructure have to be designed together.
Nota AI Server is an on-premise AI package that delivers optimized AI models, the serving environment, and GPU infrastructure as one. It comes with validated recommended configurations by model size. Accuracy, throughput, and operating cost are then verified through a PoC before the system moves into production.
For reference, models up to roughly 35B run on two L40S or RTX PRO 6000 cards; models in the 100B–250B class on two to four RTX PRO 6000s; Tera-scale models such as Kimi K3 on four to six B300s. These are representative recommendations. GPU configuration and server specifications can be adjusted to the customer's AI model, concurrent user count, response-time SLA, and operating environment. Custom models can also be optimized and supported as needed.
The cases in this issue point to the same conclusion: running agents efficiently requires control and optimization together. Guardrails and verification reduce wasted executions and false success signals, while optimizing the model, inference engine, serving environment, and hardware lowers the compute and infrastructure each call requires.
With Kimi K3, Nota AI brought the B300 configuration from eight cards down to four. With Qwen3.8-2.4T-A95B, combining quantization and pruning took it from 24 to four. The joint project with MetaM carries optimization of large language models and inference engines into live multi-agent operations, and Nota AI Server unifies model, serving, and infrastructure into a single operating environment.
They look like separate cases. The goal is the same: to let customers run high-performance AI agents quickly and reliably, within the infrastructure and the budget they can afford.
Nota AI's optimization scope is widening from model compression to inference engines, serving configurations, and hardware. At the same time, multi-agent operations require a control layer — orchestration and guardrails — to be designed alongside it. In the age of AI agents, the deciding factor is not only how capable a model is, but how quickly, reliably, and cost-efficiently it can be operated in a real service.
Looking for an optimization approach that fits your AI adoption and operating environment? Nota AI designs for a balance of performance and cost efficiency, considering model, serving, and infrastructure together.
💡 Learn more about Nota AI: https://www.nota.ai/
📩 Subscribe to Edge Insights: https://bit.ly/44XuDKX
