Running Llama 8× Faster on a Raspberry Pi: The KCC 2026 On-Device AI Optimization Challenge

This past June, Nota AI hosted the "On-Device AI Optimization Challenge with NetsPresso®" at KCC 2026 (the Korea Computer Congress) in Jeju. Out of 20 teams that applied, the winning team ran Llama-3.2-1B on a Raspberry Pi and pushed response speed up by 8× or more. They cut memory use by 8× or more as well. And they did it while holding to the quality bar.

What stands out, though, is less the result than who the participants were. The winning team, and most of the teams that reached the finals, had little formal experience with large language model (LLM) optimization. Even so, after just 30 minutes of training they picked up how to use NetsPresso, and on top of it they optimized three models (Llama-3.2-1B, Qwen2.5-0.5B, and SmolLM2) and passed a demanding quality gate. Here we look at how teams with little optimization experience reached production-level metrics in such a short time, and the role NetsPresso played underneath it all.

The Real Reason On-Device Optimization Is Hard

There was a clear reason for framing the challenge as "running an LLM on a Raspberry Pi at real-usage speed": the rise of on-device AI, where the intelligence runs on the device itself. Because on-device AI never goes through the cloud, there is no response lag, data privacy is preserved, and no network communication costs are incurred. These strong advantages are now extending quickly into the territory of large models like LLMs.

But you cannot take a server-grade LLM and drop it onto an edge device like a smartphone or a Raspberry Pi as-is. These small devices have no powerful server GPU, and their available memory is only a small fraction of a server's. If the model is too heavy, it will not even load into memory; and if you force it to run, token generation is so slow that it cannot be used in a real service. In the end, you have to carefully combine compression techniques such as quantization with a device-optimized runtime to close the physical gap between a large model and a small device.

Compression algorithms, execution runtimes, kernels, and performance measurement tools each come from their own fragmented open-source projects. Change one, and a compatibility conflict shows up somewhere else, and tracing the cause of an error is difficult. In short, the experience of seeing the whole pipeline through, from model quantization to token generation on a real device, is itself a steep barrier to entry. This competition was designed so that teams would break through this daunting end-to-end process themselves.

30 Minutes of Training, One Connected Pipeline

In a typical setting, just building the pipeline to clear this barrier eats up enormous time and trial and error. But for teams with little model-optimization experience, using NetsPresso for the first time, all it took to cross that threshold was NetsPresso and 30 minutes of training.

Inside NetsPresso, the scattered work of quantization algorithms, graph optimization, quality validation, and export for on-device execution connects into a single pipeline. With the work of stitching tool to tool gone, teams could focus on the optimization experiments themselves.

I didn't really know concepts like quantization, but it handled everything from the start all the way to deployment. When it comes to ease of use, I don't think anything else comes close.
— DKE, Excellence Award

The qualifier task the teams faced was far from easy. They had to optimize three small LLMs and pass a per-model quality gate: a perplexity (PPL) retention of at least 0.75 and an accuracy retention of at least 0.80 relative to the original. If even one metric fell below the bar, that model scored zero; only models that cleared the gate were then scored on response speed and memory efficiency.

Demanding as it was, most teams used NetsPresso to find a recipe that got all three models through the gate. In the usability survey, the question of whether the tool was intuitive scored 4.25 out of 5, the highest of the 15 quantitative items.

One Line of Config, 22 Combinations Explored

An intuitive pipeline also raised the speed of experimentation. In NetsPresso, a single line of config swaps the algorithm combination, and at each optimization step you can immediately gauge whether the quality gate is met. Below is part of a config in the same form as the one used in the qualifier.

# run.yaml: the same form of config used in the qualifier iterations (excerpt)
steps:
  - advanced_quantize:
      algorithm: AWQ           # swap to explore: AUTOROUND, SMOOTHQUANT, GPTQ
      scheme: W4-A8_dynamic
  - graph_optimize: null       # runs automatically, no extra config
  - graph_quantize:
      scheme: W4-A8_dynamic    # relax to W8-A8_dynamic if quality drops

By simply modifying config values within a single environment and immediately verifying the results, teams could conduct systematic engineering even under tight deadlines. Three cases stand out for how efficiently they ran the NetsPresso pipeline.

• Case 1: Cross-testing 22 algorithm combinations
  One team tested a full 22 algorithm combinations during the qualifier. They built a systematic search grid, cross-applying various quantization algorithms such as AWQ, AutoRound, SmoothQuant, and GPTQ together with graph optimization. They first secured baseline quality at 8-bit, then pushed the compression ratio to its limit at 4-bit, and when quality dropped they recovered it with a different algorithm, running a refined, staged scenario.
• Case 2: A flexible compression strategy sized to each model
  Another team tailored compression to each model's weight class. When they applied the same default options to all three and only the comparatively large Llama cleared the quality gate, they quickly saw that each model tolerates a different compression strength. They then quickly adjusted only the per-model compression strength in the config file and brought all three past the bar.
• Case 3: Debugging the culprit stage using pipeline visibility
  One debugging case pinned down the cause of a quality drop. When a model's quality fell below the bar, the team inspected the pipeline stage by stage and found that quality held through the quantization algorithms but broke at the final graph-quantization step. Relaxing only that step to 8-bit brought the quality retention, which had been under 0.75, straight back up to 0.97 or higher.

This practical value was also reflected in the participant survey too: the "fit for iterative experiments" item scored 4.25 out of 5, a high level of satisfaction.

Finals Results Built on a Proven Pipeline

Having adapted to the tool through the online session and the qualifier, teams enjoyed greater freedom in the finals. The task narrowed to running a single specified model (Llama-3.2-1B) on a Raspberry Pi 5 (quad-core Arm Cortex-A76) with the ExecuTorch runtime and the XNNPACK backend. Retraining the model or changing its structure was not allowed, but broad optimization across every layer was, from quantization settings to the inference runner and kernels. Final performance was evaluated by measuring time to first token (TTFT) and time-per-output-token (TPOT) on the actual device.

In the finals too, each team carried over the NetsPresso-based pipeline they had built in the qualifier. Because that pipeline gave them a proven baseline, teams could spend the remaining time entirely on the finals-specific optimization. As a result, the winning team, while holding to the quality bar, generated tokens 8× or more faster, cut time to first token by 3.8×, and reduced memory use by 8× or more.

A big part of [winning the Grand Prize] was that we made full use, in the second round, of the insights and strategies we had gained in the first.
— Ecopruner, Grand Prize

As the grand-prize team put it, the insights and the extra time gained from working with the integrated pipeline in the qualifier became the foundation for trying deeper optimization techniques in the finals. An environment that cut the time spent setting up fragmented tools and focused on the optimization search itself led directly to the teams' better performance metrics.

New to Optimization, Deployment-Grade Results

The results drew academic interest as well. After watching participants achieve meaningful performance gains in a short time, faculty members reached out about bringing NetsPresso-based practice into their regular coursework.

Survey numbers from participants who worked with the pipeline hands-on back this up. The item asking about willingness to use it again in real research or work averaged 4.12 out of 5, and willingness to recommend it to a colleague scored 4.0.

These results were validated in a controlled competition, but the problems the teams handled and solved sit exactly where real-world on-device deployment does. That makes it clear what value NetsPresso offers in an enterprise setting.

Even without specialized optimization staff or large R&D resources, a pipeline that unifies fragmented tools sharply reduces the trial and error of standing up the initial environment. On top of that, intuitive visibility into step-by-step results aids debugging and shortens the experiment cycle. This competition proved that even teams with little experience can reach deployment-grade optimization metrics quickly and reliably.

※ Watch the competition highlights and winner interviews on YouTube