Trait ModelRuntimePort 

pub trait ModelRuntimePort:
    Send
    + Sync
    + Debug {
    // Required methods
    fn admit<'life0, 'life1, 'async_trait>(
        &'life0 self,
        model_name: &'life1 str,
        num_ctx: Option<u64>,
        default_ctx: Option<u64>,
        overrides: LaunchOverrides,
    ) -> Pin<Box<dyn Future<Output = Result<Admission, ModelRuntimeError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait;
    fn current_model<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Option<RunningTarget>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn stop_current<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Result<(), ModelRuntimeError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided methods
    fn admission_snapshot(&self) -> AdmissionSnapshot { ... }
    fn list_running<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Vec<ProcessHandle>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait { ... }
    fn pinned_model(&self) -> Option<String> { ... }
    fn set_pin(&self, pin: Option<PinnedSpec>) -> Result<(), ModelRuntimeError> { ... }
}

Port for admitting requests to a running model.

This is the primary interface the proxy uses to get a running model server. Implementations handle:

• Model resolution (name → file path)
• Process lifecycle (start, stop, health check)
• Context size management
• Admission control: queueing, batching, and the VRAM resident set

Required Methods

fn admit<'life0, 'life1, 'async_trait>( &'life0 self, model_name: &'life1 str, num_ctx: Option<u64>, default_ctx: Option<u64>, overrides: LaunchOverrides, ) -> Pin<Box<dyn Future<Output = Result<Admission, ModelRuntimeError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Admit a request to a running model, launching or swapping if needed.

This method:

1. Resolves the model name to a database entry
2. Admits immediately if the model is already resident
3. Otherwise queues until the model can take a VRAM slot — either by co-loading alongside what is already there, or by swapping once the outgoing model has no requests left in flight
4. Waits for the health check to pass
5. Returns the routing target and a lease on the slot

The returned Admission::lease must be held for as long as the request is being served. Dropping it early tells the runtime the slot is free and permits a swap out from under a live generation. Callers that only want the model launched — not served — use Admission::into_target, which drops the lease deliberately.

Arguments

• model_name - Name or alias of the model to run
• num_ctx - Optional context size override from request
• default_ctx - Default context size if not specified
• overrides - Per-call launch options layered on the runtime’s standing template, so one shared runtime can serve callers with different launch needs (a GUI start carrying --mlock, a benchmark that must never gain a prompt cache). LaunchOverrides::default means “no opinion”.

Errors

Returns ModelRuntimeError if the model cannot be started, or ModelRuntimeError::AdmissionTimeout if the request never reached the front of the queue.

fn current_model<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Option<RunningTarget>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Get information about the currently running model, if any.

Returns None if no model is currently running.

fn stop_current<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), ModelRuntimeError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Stop the currently running model.

This is primarily for cleanup/shutdown scenarios.

Provided Methods

fn admission_snapshot(&self) -> AdmissionSnapshot

What the admission queue and the VRAM resident set look like right now.

Synchronous for the same reason Self::pinned_model is: it is a single read of plain shared state, not a query against live process state. The dashboard publisher calls it on every tick.

Defaults to empty for runtimes with no resident set to report (test doubles, remote backends).

fn list_running<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Vec<ProcessHandle>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Every llama-server process this runtime currently owns.

Sibling of Self::current_model for callers that need process-level detail — pid and start time — rather than routing information; the GUI server list is the motivating case.

Defaults to empty for runtimes that do not track individual processes (test doubles, remote backends). Returning nothing is always safe here: callers treat it as “no servers to show”.

fn pinned_model(&self) -> Option<String>

The one model this runtime is pinned to, if any.

Some(name) means every other model is refused with ModelRuntimeError::PinnedModelMismatch rather than swapped to — the mode gglib serve runs in. None is the ordinary auto-swapping runtime.

Synchronous because the pin is plain shared state, unlike Self::current_model, which reports live process state. Owned rather than borrowed because the pin can change at runtime (see Self::set_pin) — a borrow could not outlive the lock guarding it.

Defaults to unpinned so test doubles and remote backends need not implement it. Callers use it to avoid offering a model that would only be refused — /v1/models being the motivating case.

fn set_pin(&self, pin: Option<PinnedSpec>) -> Result<(), ModelRuntimeError>

Pin this runtime to a single model, or clear the pin.

Some(spec) makes every request for another model fail with ModelRuntimeError::PinnedModelMismatch instead of swapping; the spec’s launch overrides are layered onto the runtime’s standing template for the pinned model’s launches. None restores ordinary auto-swapping. This is how gglib serve reaches the daemon’s shared runtime: the pin travels over POST /api/proxy/start rather than being fixed at construction.

Errors

The default refuses, so a runtime that cannot honour a pin (test doubles, remote backends) fails loudly instead of silently serving every model against the caller’s explicit instruction.

Implementors

impl ModelRuntimePort for NoopModelRuntime