A laptop today can run a surprising number of different inference engines, and none of them agree on anything. One speaks to a background service over HTTP. One loads weights straight into the process. One is tuned for Apple Silicon and refuses to exist anywhere else. A speech model shells out to a command-line tool. A cloud vendor wants an API key and a network round-trip. If every part of the system that needs an answer from a model has to know which of these it's talking to, the system rots from the inside: every new engine touches every caller.

So we did the boring, durable thing. There is exactly one abstraction for inference, and everything registers behind it.

The registry is the only door

Our design note states the rule plainly: the provider registry "is the single abstraction for inference," and "new inference surfaces add provider implementations, never new abstractions." That second clause is the load-bearing one. When someone wants to add support for a new engine, the answer is never "add a new way to call models." The answer is "write a provider and register it." The door does not move.

What lives behind the door today is a genuinely mixed crowd:

• a background-service runner with retry and a circuit breaker, for the most common local case;
• an in-process runner that loads model files directly into memory;
• an Apple-Silicon-specific runner that only registers when it can actually run;
• a speech-to-text runner that wraps a command-line transcription tool;
• a couple of text-to-speech paths, including a fallback to the operating system's built-in voice;
• a high-throughput server runner for a machine with a real GPU;
• a remote-fleet runner that borrows another device's models;
• the cloud vendors, each shipped as its own self-contained package that registers itself into the same registry;
• and a mock, for tests.

That is a lot of variety. From the outside, it is one surface.

The router dispatches by purpose, not by name

Behind that single door sits a second idea that matters just as much. Callers do not ask for a specific model. They ask for a purpose. Our note describes a model router that "dispatches a request to the right backend by purpose (chat / coding / agent / transcription / vision) and device capability."

This is the inversion that keeps the codebase sane. A feature that wants to summarise a conversation does not say "use this particular local model." It says "I need a chat answer," and the router decides — based on what's installed, what the device can handle, and what's available right now — which backend actually serves it. A later design pass, as the note records, "unified the consumer surface so callers target purposes, not concrete backends."

The payoff is concrete. The same line of feature code does the right thing on a phone, on a laptop, and on a GPU workstation, because the decision of which engine to use was lifted out of the feature and into the router. When we later added a first-class server runner for the GPU machine, no chat feature changed. It just started being chosen for agent work on the host that had it.

Device capability is a first-class input to that decision, not an afterthought. The router consults a hardware-capability inventory and a thermal manager — because on a phone, the binding constraint isn't whether a model can run, it's whether running it will cook the device. Our note is honest about the number: roughly 44% throttling has been observed on an iPhone under sustained load. The router treats that the way a good scheduler treats heat — as back-pressure, not as a thing to ignore.

Boot stays instant because nothing loads at boot

There's a trap waiting in any design like this. If the system registers ten engines at startup and each one eagerly loads its weights to prove it works, startup takes forever and most of that work is wasted — you were only ever going to use one or two models this session.

We refuse that cost by design. Boot registers every provider eagerly but loads zero weights. The only thing a provider does at startup is a cheap availability check — our note is exact about this: the availability probe "only stats files," meaning it asks the filesystem "do your model files exist?" and nothing more. The actual, expensive load happens on the first real inference call, and not a moment sooner. The standard has a name for the invariant — boot does no weight loading — and a structural check guards it.

The result is the property you actually want from a daemon: it comes up immediately, advertises everything it could do, and only pays for a model the instant something genuinely needs it. A user who never touches the vision path never pays to load a vision model. A test that only needs the mock never warms a real engine.

Why this shape, and not the obvious one

The obvious shape — let each feature talk to whichever engine it prefers — is faster to write on day one and unmaintainable by day thirty. Every engine becomes a tendril reaching into every feature. Adding the GPU server means editing every caller. Supporting a phone means special-casing thermal logic in a dozen places.

The single-door shape front-loads a little discipline and buys back a lot of freedom. New engine? One package, one registration, zero caller changes. New device class? The router already asks about capability. Cloud vendor? It ships as its own package and slots in beside the local engines without the core ever importing it.

The honest caveat, recorded in the same note: there was an older, more ambitious vision — distributing model weights peer-to-peer as first-class packages over the network — and that one did not ship. Today, model management still leans on a local command-line tool and files on disk. We name that gap rather than paper over it. But the part that did ship — one abstraction, a purpose-driven router, lazy loading — is the part that has held up under every new engine we've thrown at it.

One door. The rest of the system knocks once and gets an answer, and never has to ask who's behind it.

Related: Plan With the Big Model, Work With the Local One · The Protocol That Lets an AI Drive Your Daemon.

Written by AI agents from real project logs; owned and edited by Mujo.