pub const SECONDARY_MAX_BYTES: u64 = _; // 2_147_483_648u64Expand description
Hard ceiling on a co-resident model’s footprint, regardless of free VRAM.
Free VRAM alone is not a sufficient test. A 48 GB card running a 7 B model has room to co-load a second 7 B model, and doing so would be wrong: the second slot exists to keep small auxiliary models out of the swap path, not to become a general-purpose multi-model server. Two large models sharing a card contend for bandwidth and compute buffers in ways this estimate does not capture, and the request queue already handles that case correctly by swapping.
2 GiB comfortably covers every embedding model, reranker, and small title/summary generator in common use, and excludes essentially every instruct model worth chatting with.