Expand description
Short-lived snapshot of application settings.
Every chat-completion request needs settings — global inference defaults,
and now the configured inference profiles. Reading them per request meant a
full SELECT over the settings_kv table on the hot path, twice on the
upstream-death retry path. Profiles are needed earlier in the handler than
the defaults were, so without a cache the cost would have doubled again.
This wraps the repository in a snapshot that is reused for DEFAULT_TTL
before being refreshed, turning a per-request query into roughly one query
per TTL window. Callers hold the returned Arc for the life of the
request and borrow from it, so nothing is cloned per request either.
§Why a TTL rather than invalidation on write
The obvious alternative — clear the cache whenever settings are saved —
cannot work here. The CLI writes the same SQLite file from a separate
process, so an in-process invalidation hook would never observe
gglib config profile set while the proxy is running. A TTL bounds
staleness uniformly no matter which process did the writing, at the cost of
settings changes taking up to DEFAULT_TTL to take effect.
§Failure behaviour
A failed load never fails the request. The last good snapshot is served if
there is one, and Settings::default otherwise — matching the previous
.ok().and_then(...) behaviour at the call sites. A failure does not
refresh the expiry, so the next request retries rather than serving a stale
value for a whole TTL window; during a sustained outage that means one
attempt per request, which is what the code did before this cache existed.
Structs§
- Settings
Cache - A settings snapshot refreshed at most once per TTL window.
Constants§
- DEFAULT_
TTL - How long a snapshot is served before it is refreshed.