The Sovereign AI Foundation Model Project is a government-led initiative in South Korea aimed at developing AI models optimized for the Korean language and national context. The Upstage consortium, which includes Lablup, was selected as one of four teams in the second phase of a highly competitive process. Within the consortium, each organization contributes domain-specific expertise, with Upstage leading model development, Lablup providing the infrastructure layer, and Nota AI focusing on model compression to enable efficient deployment with reduced GPU requirements.

In a previous interview with Upstage, we covered how large-scale training was reliably executed using over 500 GPUs during the first phase of the project. In contrast, Nota AI operates on a different axis within the same consortium, running rapid, iterative cycles of quantization and evaluation on models produced by Upstage. This article examines how Nota AI leveraged Backend.AI and Backend.AI FastTrack 3 to streamline these workflows, and how Solar Open 2 was ultimately optimized to run on just two H100 GPUs.

Nota AI: Model Compression and Optimization Specialist

 

Nota AI specializes in AI model compression and optimization. Through its proprietary platform, NetsPresso®, the company has developed capabilities to optimize models across a wide range of hardware targets, from GPUs to NPUs, with a strong focus on on-device AI deployments. Within the Sovereign AI Foundation Model Project consortium, Nota AI’s role can be summarized as ensuring that Solar models maintain performance while minimizing GPU resource consumption.

Modern AI models tend to scale in performance with increasing parameter counts, which is why foundation models are typically developed at massive scale. However, organizations deploying these models face a different constraint: operating them with limited infrastructure. In practice, the most common requirement Nota encountered was straightforward: reducing GPU usage.

This challenge becomes more complex with Mixture of Experts (MoE) architectures, where quantization errors can alter the router’s expert selection. To address this, Nota synthesized calibration data to ensure balanced expert activation and developed techniques to preserve routing decisions after quantization through parameter adjustment. In addition, they applied non-uniform global pruning, selectively retaining different numbers of experts per layer to further reduce model size.

These optimizations directly translated into lower GPU requirements at serving time. Large-scale models are typically deployed using tensor parallelism, distributing weights across multiple GPUs, usually in powers of two such as 2 or 4. To reduce a 4-GPU configuration to 2 GPUs, the combined footprint of weights and KV cache must fit within the aggregate memory of the two GPUs; otherwise, the deployment still requires 4 GPUs.

Leveraging these techniques, Nota successfully enabled the Solar Open 100B model to run on a single H100 GPU during the first phase of the project, effectively halving the typical GPU requirement for a model of that scale. In the second phase, they applied NVFP4, INT4 quantization to Solar Open 2 250B, reducing the serving configuration from 8 H100 GPUs down to just 2.

Building a Flexible and Accessible Environment with Backend.AI

Hancheol Park, Tech Lead at Nota AI, highlighted flexibility as one of the most valuable aspects of Backend.AI in their workflow.

Nota was allocated three 8-GPU nodes through the consortium, using two for quantization algorithm development and one for calibration data synthesis. This allocation was continuously adjusted based on workload requirements. For tasks such as profiling, which required loading the full model into memory and exceeded a single node’s capacity, Nota leveraged Backend.AI’s multi-node cluster sessions to aggregate up to 12 GPUs across nodes into a single job.

At the core of Nota’s R&D workflow is rapid iteration. Without a flexible environment, time spent reconfiguring infrastructure quickly becomes a bottleneck. Backend.AI addresses this with GPU virtualization, dynamic resource allocation, and real-time visibility into available capacity at session launch. As a result, the team was able to provision GPUs on demand and immediately proceed with the next experiment without reservation delays or idle wait times.

For quick file edits, the team used a browser-based VS Code session, while lightweight checks were handled directly in the web terminal. To manage experiment environments, Nota created a shared uv virtual environment on network storage and configured the package cache to use the same location. This allowed each session, even when launched in a fresh container, to reuse an identical environment without additional setup, effectively reducing bootstrap time before experiments could begin. Park noted that being able to instantly launch VS Code, especially for urgent tasks that did not warrant opening a full editor setup, was particularly useful.

Reducing Quantization Pipeline Time by 25% with FastTrack

Nota’s quantization workflow exhibits significant variation in GPU requirements across stages. The profiling phase, which collects statistics for quantization, requires loading the full model into memory and thus demands the highest GPU capacity. In contrast, the quantization step operates at the decoder block level and typically requires only a single 80GB GPU. The evaluation phase, used to measure model accuracy, usually runs on one or two GPUs, with multiple benchmarks executing independently and benefiting from parallel execution.

According to Geonho Lee, each benchmark had distinct Python dependencies, making environment isolation essential, while the workflow itself had to be repeatedly executed with different algorithms and configurations. Managing such a pipeline manually, with varying GPU requirements and inter-step dependencies, would introduce significant overhead.

To address this, Nota automated the entire workflow using Backend.AI FastTrack. Built on Backend.AI’s session and resource management framework, FastTrack enables chaining multiple sessions into a pipeline, automating GPU allocation and dependency handling at each stage. It also provides templating capabilities, allowing the same pipeline to be reused across different configurations, which is critical for iterative experimentation.

On FastTrack, Nota constructed a pipeline that performs profiling, quantization, and vLLM validation, followed by parallel execution of benchmarks such as HumanEval, AIME, and GPQA, along with inference latency measurements. By defining GPU requirements per stage, each step was automatically scheduled based on available resources, reducing the need for manual intervention. Compared to manual orchestration, this approach reduced total pipeline execution time by approximately 25%. In addition, parameters such as model variants, algorithms, and bit precision could be dynamically injected via environment variables, enabling efficient reuse of pipeline templates.

“After an initial learning curve, it became clear how to structure and run the pipeline. Compared to sequential bash scripts, FastTrack made GPU control much more manageable,” Lee noted. Although this was his first time using a pipeline orchestration tool, he emphasized that FastTrack allowed the team to focus more on output quality rather than environment setup and management.

*Based on Nota’s internal measurements: 10 iterations of a quantization and evaluation pipeline for a 100B-scale MoE model on an 8× H100 setup.

Running a 250B Model on Two H100 GPUs

Solar Open 2 is an agent-focused open-weight LLM with a Mixture of Experts (MoE) architecture, featuring 250 billion total parameters and 15 billion active parameters per token. It is designed for real-world agent use cases such as document processing, coding, and tool use.

The original BF16 model requires 520.6GB of memory, including 500.6GB for weights and 20GB for KV cache (at a 256K context length), necessitating 8 H100 GPUs for serving. Nota reduced the model size to 142.9GB through INT4 quantization, and further down to 117.8GB by applying expert pruning. Including KV cache, the total footprint becomes 137.8GB, which fits within the combined 160GB memory of two H100 GPUs. This reduces the serving requirement from 8 GPUs to just 2.

Despite the aggressive compression, performance degradation was minimal. Across 15 benchmarks covering agent tasks, mathematics, coding, and Korean language understanding, the original BF16 model scored 81.57, while the INT4 quantized model achieved 81.17. With additional expert pruning, the model maintained comparable performance, scoring 83.39 versus 83.99 on a separate three-task average. This demonstrates that GPU usage can be reduced by up to 75% while preserving model quality.

Toward Data Center–Scale Optimization with Backend.AI and FastTrack

---

Just as Nota enables large-scale language models to run on fewer hardware resources through quantization, Lablup empowers organizations to focus on their core business without the burden of infrastructure and environment management. If your workloads involve iterative experiments with varying resource requirements, explore how Backend.AI and FastTrack can streamline and accelerate your operations.