Optimizing Small Language Model Serving with Fractional GPUs

NVIDIA MIG (Multi-Instance GPU) is a hardware partitioning feature available since the Ampere generation. It divides a GPU into profile units such as 1g.10gb and 1g.20gb, assigning each instance dedicated compute and memory, with memory bandwidth QoS, memory protection, SM performance isolation, and error isolation guaranteed in hardware. Profile-based partitioning keeps layouts fixed and predictable; changing the layout involves recreating the instances, and support covers datacenter and workstation GPUs. Backend.AI's fGPU implements the same partitioning in software at the CUDA API layer. It assigns ratios freely using fractional slot values like 0.1 and 0.25, reclaims slots when a container terminates so they can be reused at different ratios without a GPU reset, and divides memory and compute themselves rather than taking turns like time-slicing, running across NVIDIA GPUs including GeForce, regardless of MIG support.

Choosing a partitioning approach depends on where you place the weight between isolation requirements and resource utilization. Environments that must guarantee workload isolation at the hardware level and environments that frequently reshape partitions to raise utilization call for different approaches, and Backend.AI supports both partitioning methods, so you can choose per workload. See the full whitepaper for the benchmark methodology, complete results for all 19 models, and decision criteria by isolation requirements.