How an assistant is meant to find out what it can do

When you type a request to an assistant that runs on your own machine, two very different questions get answered before it replies.

The first is what is it allowed to do? The system keeps a registry of capabilities — every discrete action anything here can perform, from converting a file to searching your messages, each one with a name and a short description of what it takes and what it returns. Permission to use them is granted by rule, and the assistant you talk to is granted a wildcard: a single character meaning everything the system can do. That grant lives with the rest of the permission rules rather than inside some function's private constant, which matters — it means the set cannot be narrowed by an edit nobody reviews as a permission change.

The second question is what is it shown? and it is a completely separate question, because a model does not browse a registry. Every capability it may choose from has to be written into the request as a description, and the model picks from what is written there. Anything not written down is not refused — it is invisible. Granted and offered are different things.

They have to be different, because showing a model everything is its own failure. Two measurements, both taken by other people on this project and both recorded in our source alongside the number they justify, because a number without a provenance is just a number:

Being shown more makes it choose worse. With exactly one capability offered, a model picked correctly 31 times out of 32. With thirteen, 18 to 21 out of 32 — and the controlling variable was the count, not the wording. The step text was rewritten repeatedly without moving the result.

Being shown too much makes it not see your question at all. Models measure text in tokens — roughly, word-fragments — and each model has a fixed context window, a maximum number of tokens it can be handed at once. A single assistant request was measured assembling to 14,374 tokens against a window that held 8,192. The overflow is not an error. It is discarded silently. The descriptions of the capabilities were 79% of that prompt; the sentence the person actually typed was 29 tokens, two tenths of one percent — and it was in the part that fell off the end. Replayed with the prompt cut, the model got four out of four wrong. Replayed whole, six out of six right.

So the design is: granted everything, shown a few. Every time you ask something, the system ranks all of its capabilities against the words you used and shows the model the top handful.

flowchart TD
  Q["You ask for something"] --> G["What is this assistant allowed to use?
— everything"]
  G --> R{"Rank all of it
against your words"}
  R -- "ranking worked" --> H["the best-matching ones
take the HEAD of the list"]
  R -- "ranking crashed or
returned nothing" --> LX["a deliberately dumb local match —
still cut to the SAME limit"]
  LX --> H
  H --> M["4 slots reserved at the TAIL:
the safety checks + 'search for a tool'"]
  M --> OUT["at most 24 shown to the model,
never more"]

The default, and there is nothing to configure

The number is twenty-four, it is a constant in one file, and no setting and no environment variable stands in front of it. It is stated exactly once, on purpose, so that the two different ways of talking to a model in this system — the background one and the chat you type into — cannot drift to two different numbers. Both call the same function.

Four of those twenty-four slots are reserved at the tail for entries that must be present every time: the checks that let a run continue, and a search for a capability verb, which is the model's only route to anything the ranking missed. Reserved means held back before ranking begins, so they can never be crowded out — and counted inside the twenty-four, not added on top of it.

The ranked entries take the front of the list, and that ordering is real rather than cosmetic: the array's order is literally the order the descriptions are written into the request, and a small model was measured favouring whatever it sees first. Presence is what the four mandatory entries need; position is what the model is influenced by; the two are separable, so they were separated.

The fallback: no failure is allowed to widen the offer

The ranking machinery can break — it needs a component that is not always present. When it does, the offer must still come out small. The backstop is a local, deliberately crude word-match with no dependencies, cut to the same limit. The reasoning is written into the file: it must not be able to fail in the way that the thing it is backstopping just failed.

What was wrong

You ask the assistant to turn a photo into a PDF. It tells you it can't do that.

It can. The capability was installed, registered, working, and reachable by every other part of the system. What happened is that when your question was handed to the model, the list of things the model was told it could do had twenty-nine entries on it, and converting a file was not one of them.

Nothing refused you. Nothing was blocked. There was no permission check, no policy, no warning. The capability was simply absent from the list, and absence does not announce itself. From the outside, an assistant that has been denied and an assistant that was never told look exactly the same: both say no.

That twenty-nine-item list was hard-coded — a basket originally written for a different internal role, and inherited by the assistant you actually talk to. The comment now standing where it used to be states the arithmetic without softening it: the system had 1,220 capabilities available and the model received 29 of them.

We should be careful with that number rather than repeat it as a headline. Our own code records 1,220 in one place, 1,225 in another and 204 in a third, because those are different collections counted at different moments. The ratio is the point and the ratio is not in doubt; the precise size of the haystack is not something we have measured once and can quote as a constant.

There was a second asymmetry underneath it. Of the two ways this system talks to a model, only one ranked and bounded what it showed. The chat you actually type into did neither.

How it was fixed — and what is still not fixed

The instruction that ended it was blunt: the assistant gets all the tools, and any hard-coded list anywhere that says otherwise comes out. The basket was deleted, the wildcard grant took its place, and the bounded, ranked offer described above became the single path both kinds of request go through.

The bound was set to twenty-four deliberately below the twenty-nine of the basket it replaced, so the change could not be a regression on width while claiming to be a fix.

Three things about getting there are worth more than the change itself.

The limit has to be a total, not a top-up. The first version of this counted the must-always-be-present entries separately and added them on afterwards, so the offer could reach forty-six — worse than the twenty-nine it replaced, and squarely inside the truncation described above. The header said "bounded" and the arithmetic said otherwise. That was caught by someone else's measurement landing on it, and it is in the record as a correction rather than quietly tidied away.

The reordering is the weakest-evidenced thing here. The head-of-list bias was measured once, on one small model, across five options. Our own source says so, in capitals, over the inference that follows from it: untested, do not build on it. The change was made anyway because nothing ever required the old order, so giving the best position to the best match is better-motivated either way — but "better-motivated" is a different claim from "measured", and there was also an 81% figure cited here for a day that its own author then tested and withdrew.

Testing the crash path found a real defect in the thing being backstopped. Writing the test that proves a broken ranker still produces a small offer turned one up. The fast path guarded a field it read from each capability. The fallback path read the same field on the same objects, unguarded — so any capability that didn't carry it crashed the fallback outright. And the fallback is precisely what runs when the fast path is unavailable. A degraded path that is less careful than the thing it degrades from only fails once something has already gone wrong, which is the worst place in a system to put the weaker check. The fix guarded it, and also guarded the next field down, three lines later in the same loop, which nothing had reported — because a crash report tells you where execution stopped, not every place that shares the defect. Then it swept the file to prove no unguarded reads of that shape remained.

The practical effect is that when you ask for something the system can do, the capability that does it has a real chance of being in front of the model — and when the ranking is wrong about you, the model still holds a way to go looking for what it wasn't shown.

Honest status

The wider effort this sits inside is in progress, mid-build, with no sign-off. The specific pieces described above are landed code we read on the main line, not commit-message summaries.

And the numbers underneath the design are small. Thirty-two trials an arm, one local model, a handful of options. They are the best evidence anybody here has, they were taken on the real path rather than in a thought experiment, and they are not a law of nature. We are stating them with their sample sizes attached for the same reason we deleted the hard-coded list: a claim that hides where it came from is the kind that survives long after it stops being true.

Related: one door for every model you run, and there is no ungoverned route to a model.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.