Module model_runtime 

Model runtime port for proxy model management.

This port defines the interface for admitting a request to a running model. It abstracts the process management details from the proxy layer.

Admission, not “ensure running”

The entry point is ModelRuntimePort::admit, and it returns an Admission — a routing target plus a lease. The lease is what makes request batching possible: the runtime cannot decide whether it is safe to swap models unless it knows how many requests are still being served by the one currently loaded. Holding the lease for the life of the request is therefore not bookkeeping, it is the mechanism.

A caller that only wants a model up and does not care when it goes away (the GUI’s “start model” button) drops the lease immediately; the model stays resident until something else wins admission.

Modules

error_type

Canonical error.type discriminants, shared by every surface.

Structs

Admission

A granted admission: where to send the request, and the lease that keeps the model loaded while it is in flight.

AdmissionLease

Proof that a request is being served by a resident model, and that the runtime must not evict that model until the request is done.

LaunchOverrides

Per-call launch overrides layered on a runtime’s standing configuration.

NoopModelRuntime

A ModelRuntimePort that never has anything running.

PinnedSpec

A runtime pin: the one model a runtime will serve, plus how to launch it.

RunningTarget

Target information for a running model instance.

RuntimeErrorEnvelope

Structured, serializable view of a ModelRuntimeError.

Enums

ModelRuntimeError

Errors that can occur during model runtime operations.

Traits

AdmissionRelease

The runtime side of an AdmissionLease: what to call when a request that was holding a VRAM slot is finished with it.

ModelRuntimePort

Port for admitting requests to a running model.

Functions

is_retryable_error_type

Whether a wire error.type discriminant denotes a retryable condition.