Two days after Upstage released Solar Open 2, we had it running on a pair of DGX Sparks and brought it to a service-ready state, despite this combination being considered unsupported. This post explains why DGX Spark was seen as an incompatible target for Solar Open 2 NVFP4, and what changes and workarounds were required at the FlashInfer and vLLM levels to make it work.

TL;DR

To run Solar Open2 NVFP4 model using two DGX Sparks, follow Step 1-6 through NVIDIA's multi-node vLLM guide, while replacing every occurrences of export VLLM_IMAGE=... lines with export VLLM_IMAGE=cr.backend.ai/stable/ngc-vllm:26.06-cuda13.3-ubuntu24.04-solaropen2. After that, start the vLLM server by:

Solar Open 2 in brief

Upstage released Solar Open 2 on July 22, 2026. It is a 250B-parameter Mixture-of-Experts model with approximately 15B active parameters, using 320 routed experts and 1 shared expert with top-8 routing.

The architecture uses hybrid attention, alternating 12 GQA softmax layers with 36 KDA (Kimi-Delta-Attention) linear-attention layers. No positional encoding is used, and the model supports a 1M token context window. Only 12 out of 48 layers maintain a KV cache, reflecting the properties of the linear-attention design.

At BF16 precision, the model requires about 500 GB of memory, and Upstage recommends at least four H200 GPUs for serving.

Alongside the release, Nota AI published an NVFP4-quantized checkpoint using the compressed-tensors format (nvfp4-pack-quantized). Since this format is native to vLLM, it can be loaded directly without NVIDIA ModelOpt.

With NVFP4 quantization, the memory requirement drops from 500.6 GB to 153.3 GB. Across 15 benchmarks, the average score decreases only slightly from 81.57 to 81.35 (−0.22), with 4 benchmarks showing improvements, indicating negligible performance loss.

Why DGX Spark was considered unsupported

DGX Spark uses a unified memory architecture with 128 GB capacity. In practice, only about 110 GiB is available for model loading due to system overhead. With two nodes, this gives roughly 220 GiB of usable memory, which is sufficient for the 153.3 GB NVFP4 model.

However, even with sufficient memory, the model fails to load correctly. The issue is not memory, but compute architecture.

The GB10 GPU in DGX Spark differs from datacenter Blackwell GPUs like B200 or B300. While NVIDIA brands all of them as Blackwell, they have different compute targets and capabilities.

Target	Hardware	Core feature
sm_100	B200, GB200 (Datacenter)	tcgen05 + Tensor Memory, 228KB smem/SM
sm_120	RTX 5090, RTX PRO (Desktop)	warp-level mma.sync, ~100KB smem/SM
sm_121	GB10 / DGX Spark	Same ISA as sm_120, different functional levels

For a deeper dive, see our previous blog post: Inside NVIDIA DGX Spark: Is DGX Spark Actually Blackwell?

The b12x kernel tree

This is where FlashInfer’s b12x tree comes in. It is a separate, functionally limited kernel tree for SM 12.x, ported in April 2026 from the community b12x project through multiple PRs (#3051, #3066, #3080). It uses NVIDIA’s CuTe DSL to JIT-compile kernels.

However, the b12x tree does not support all features of the mainline FlashInfer implementation¹. Acceleration paths such as trtllm-gen attention, CUTLASS, FHMA, and DeepGEMM are not yet available. Still, it appeared to include sufficient MoE functionality to run NVFP4 weights.

In practice, two major issues arise.

Lack of Expert Parallelism

Serving a 320-expert model across multiple GPUs requires Expert Parallelism (EP). However, upstream FlashInfer b12x does not support EP.

Checkpoint format incompatibility

Even without EP, b12x cannot load Nota’s NVFP4 checkpoint because it only supports ModelOpt format or on-the-fly BF16 quantization. The compressed-tensors format is rejected.

No viable alternative backend

Other vLLM MoE backends also fail for this setup (SM 121 + NVFP4 + EP=2):

• FLASHINFER_TRTLLM / CUTEDSL / CUTEDSL_BATCHED only support Hopper or datacenter Blackwell
• FLASHINFER_CUTLASS (selected by vLLM AUTO on SM 121) has reported issues such as garbage outputs and illegal address crashes on GB10
• VLLM_CUTLASS does not support Expert Parallelism
• Marlin works but falls back to W4A16, losing FP4 tensor core benefits

The only viable solution was to extend FlashInfer directly.

The fix

Adding Expert Parallelism to b12x

We implemented EP support by introducing a local_expert_offset parameter into JIT-compiled kernels.

Each node holds a contiguous shard of experts. For example, one node handles experts 0–159 and the other 160–319. The kernel remaps global expert indices to shard-local indices and filters out-of-shard pairs using CTA-uniform guards.

This applies across all four fused-MoE backends in b12x, including the micro path (Triton compact pre-pass), static and dynamic NVFP4 W4A4 paths, and W4A16.

Additional changes include:

• Relaxing scale tensor alignment from 16 to 4
• Adjusting backend selection heuristics to local workloads
• Moving workspace buffers from per-layer allocation to a shared module-level pool

Without this shared pool, 48 MoE layers would each allocate about 0.5 GB of scratch memory, exceeding available memory.

Supporting real NVFP4 checkpoints

We added prepare_b12x_nvfp4_packed_weights, which converts pre-quantized checkpoints into the format expected by the kernels.

This includes:

• Supporting both ModelOpt and compressed-tensors formats
• Converting Nota’s reciprocal global scales and folding them into FP8 block scales with re-rounding (clamped to 448)
• Swizzling into the kernel’s 6D MMA tile layout
• Handling fused gate/up projection ordering and EP shard-based scale slicing

Deployment

We provide a ready-to-use Docker image:

cr.backend.ai/stable/ngc-vllm:26.06-cuda13.3-ubuntu24.04-solaropen2

Based on NGC vLLM 26.06, it includes:

• Upstage’s vLLM fork (branch v0.22.0-solar-open2) for model implementation, reasoning/tool parsers, and KDA prefill autotune warmup fix
• Upstage’s transformers fork (branch v5.14.1-solar-open2) registering the solar_open2 model type
• Lablup’s FlashInfer fork replacing stock 0.6.12 with 0.6.15-based wheels

What this means

As of July 24, 2026, upstream FlashInfer still does not support Expert Parallelism or compressed-tensors W4A4 checkpoints in the b12x tree (see commit 817e4bd1). vLLM also does not enable EP for b12x (see commit dd72658e).

Our fork represents the first public demonstration of native W4A4 NVFP4 expert-parallel MoE serving on DGX Spark.

You can find the patch here:
https://github.com/lablup/flashinfer/commit/1c623dea557a51e5b92b20ea8a342fd546cc5bf9

Footnotes

1. https://github.com/flashinfer-ai/flashinfer/issues/3170 ↩