How running a model on your own machine is supposed to work

When you run an AI assistant on hardware you own, you make two choices, not one, and almost every explanation of local AI only mentions the first.

The first choice is the model — the trained weights, shipped as a single large file you download: some flavour of qwen, llama, whatever you trust. The file is inert. It is a pile of numbers describing a network, and on its own it does nothing at all.

The second choice is the runtime: a separate program whose job is to load that file into memory and actually run it, turning your text into the model's text. Ollama is one such runtime. llama.cpp is another. MLX, built for Apple silicon, is a third. There are several, they are independently written, and the same model file can be handed to any of them. The runtime is the engine; the model file is the fuel.

On top of that sits the behaviour that makes an assistant an assistant rather than a chatbot: tool calling. A tool is a concrete operation the assistant is allowed to perform on your behalf — read this, write that, send this — and the runtime offers the model a list of them. When you ask the assistant to do something, the correct outcome is that it picks a tool from that list and invokes it. The failure that looks superficially similar, and is not the same thing at all, is the model writing a fluent paragraph describing the action it would take, and taking none.

The moment that matters most is the first turn — the first thing the assistant does after you ask. If the first action lands, everything downstream has something real to build on. If the assistant instead narrates, you are already in a conversation about work rather than in the work. So the property worth engineering for, and the one worth measuring, is narrow and testable: when asked to do a thing, does the assistant take the action on its first move?

The design goal underneath all of this is that your two choices should both be free ones. You pick the model you trust and the runtime that suits your hardware, and the assistant behaves the same way either way. That is what makes "run it yourself" a real option rather than a slogan.

What was wrong

We used to file the runtime under "plumbing detail." Then we measured it.

We took that one specific property — asked to do something, does it take the action — and measured how often a model got it right across different runtimes. Same model file. Same prompt. The only thing that changed was the program running it.

The spread was startling. The identical model file took the action 96% of the time on one runtime and 20% of the time on another — a 76-point cliff. A third sat around 40%. Which runtime you happened to pick mattered as much as, sometimes more than, which model you picked.

flowchart LR
  M["the same model file"] --> A["on one runtime →
takes the action 96% of the time"]
  M --> B["on another runtime →
takes the action 20% of the time"]
  B -. "the runtime, not the model,
made the difference" .-> M
  A --> F["our fix: hand it exactly one tool
on the first turn"]
  B --> F
  F --> R(["takes the action 100% —
on every runtime we tested the lever on"])

This is exactly the kind of thing that can quietly wreck the promise. The whole point is that your AI runs on your own machine, answering to you, on whatever model you choose to trust. But if whether it actually does the thing secretly depends on an invisible runtime choice, then "run your own model" comes with a hidden reliability lottery — and the person holding the ticket is a user who never knew they were playing. From the outside there is nothing to see: no error, no crash. Just an assistant that is talkative on one machine and effective on another.

How it was fixed, and what is still rolling out

So we closed it, at least for the most important moment: the first action. The fix is almost embarrassingly small once you see the failure. On that first turn, when the assistant should do exactly one thing, we hand the model a list with exactly one tool on it — nothing to get distracted by, nothing to talk itself out of. With that, first-action reliability went to 100% on every runtime we tested the lever on — four of them, twenty-five trials each. The fifth, our cloud command-line assistant, was already at 100% without it and was never put through the mechanical arm. The runtime stops being a lottery for that first, decisive step.

This sits underneath the same idea as giving you one door for every model you run and letting you reach for the big model or the local one as the job demands: the machinery should make your choice of model and runtime work, not quietly punish it.

The discipline here is to never dress a draft up as finished. So: the finding is solid and measured, and the first-action fix is real and proven at 100% on the four runtimes it was mechanically tested against. What's not done yet — the full multi-step case is closed on two of those five runtimes, with the rest being measured now; and the fix is something we can switch on, not yet the default for everyone while we finish extending it. The hard part — knowing the runtime is a first-class variable, and having a mechanism that flattens it — is in hand. Turning it on everywhere, by default, is the work in front of us.

The lesson we're keeping is bigger than the patch: when you run models locally, the runtime underneath them isn't a detail. It's a variable in whether the thing works at all — and we're measuring across runtimes now, not just across models, so that "it did what you asked" stops depending on a choice you didn't know you were making.

Written by AI agents from real project logs; owned and edited by Mujo.