We run the local models on our GPU machine on llama.cpp, with one person's work in between: Lumina Nao, who keeps two repositories on Codeberg — llama-hdd.cpp and llama-launcher. This piece is the thank-you we owe.

Start with what they stand on, because honouring a fork means honouring its parent. llama.cpp is the C++ project that made running large language models on ordinary hardware normal, and it is generous engineering by any measure: fast, portable, MIT-licensed, and moving quickly. Much of what follows is careful work laid on top of that.

Two things you want when coding, both made of memory

Coding with a model is not a chat. It is a long, accumulating working state: the files it has read, the errors it has seen, the decisions already made. The more of that it can hold at once, the better it codes — a model that can see the whole shape of what you are doing stops asking you to re-explain it, and stops making changes that contradict something three files away. A large context is not a luxury here; it is most of the difference between an assistant that helps and one you spend your day correcting.

The second thing you want is more than one of them. Real work runs several sessions side by side — one building, one reviewing, one chasing a failure — and their value is that each keeps its own accumulated state while you are away from it.

Both of those are made of the same substance, and it is memory. When a model reads a long prompt it does not merely read it, it encodes it, turning the tokens into the internal state it will consult from then on. That work is called prefill, and on a long prompt it is the expensive part. The encoded state lives in a slot: one session's working memory inside the server.

So the two things you want pull against each other, and against the model itself. Each session wants a slot, each slot holds its whole encoded history, and those are large. On a box with memory to spare this is somebody else's problem. On a box where the model weights and the session state draw on one pool — a single GPU, or a machine where system and graphics memory are the same thing, which the field calls unified memory — it is the whole problem. Every session you keep warm is memory the model does not get, which means a smaller context for the session you are actually using.

There are two usual answers and neither is good. Run fewer sessions and keep each context small, or throw the idle ones away and re-encode from scratch when you come back — which on a long working history means paying the expensive part again, from the top, while you wait.

The trade: take the cache out of memory

The launcher offers a third answer, and the shape of it is one flag. Its disk-cache mode turns on a disk-backed slot cache and, in the same breath, forces the in-memory prompt cache to zero. That second half is the interesting one. It is not a durability feature that happens to use a disk; it is a deliberate decision to stop spending memory on idle sessions at all, and to spend the disk instead. Where the model and the cache draw on one pool, the memory this stops spending is memory the model keeps — which is what buys back both the larger context and the extra session — which is the pressure the fork's README names from the other side, where holding large caches in memory eats into the space the model needs.

What happens when you switch

Between the model server and whatever is talking to it, the launcher runs a proxy, and the proxy is where the idea becomes a mechanism.

Each conversation gets its own file, matched by a session identifier the client sends — one stable id, one file, exact match. The conversation file is not rewritten after every response: the proxy writes it at deliberate moments — when you switch away, and when it shuts down cleanly.

A switch, then, is two steps and a tidy-up: the outgoing conversation is written out when it has anything new to keep, the incoming one is read back in, and the disk is tidied on the way past. What comes back is the encoded history, so the model does not re-read the conversation — it picks it up.

flowchart LR
  A["conversation A
(live in the slot)"] -->|"switch away"| S["save A:
slot file + checkpoint file"]
  S --> D[("disk:
one file set
per conversation")]
  D -->|"switch back"| R["restore B:
reads slot file +
checkpoint file"]
  R --> B["conversation B
(live in the slot)"]
  D -.->|"while past a limit"| E["retire the oldest,
not the two in play"]

What the fork adds beside the cache

Here is what the fork adds, and it is a small thing.

Upstream can already write a slot to disk. What it writes is the encoded cache. Alongside that cache, a slot also holds a list of checkpoints — snapshots of the parts of the state that cannot simply be rewound to an earlier position — and upstream leaves that list in memory. On the model architectures that lean on those checkpoints to pick a conversation back up, a slot restored without them re-encodes from scratch anyway, which the fork's README says in one clean sentence defeats the point. On those models, that is the difference between a switch that costs a file read and a switch that costs a re-encode.

So the fork writes a companion file beside the cache, the same filename with .ckpt on the end. Eight fixed bytes identifying the format first, then a count, then each checkpoint's position range, token count, and its saved state bytes. Nothing has to be switched on: a server already started with a slot directory gets the companion file on its next slot save. The README calls the fork's divergence from upstream a small set of patches centred on that companion file, and the fork adds exactly one flag of its own — off by default — a dial that thins how many checkpoints a save writes, keeping the newest and the earliest and widening the gaps in between. Change nothing and every checkpoint is written.

Two companion files, two deliberate policies

There is a second companion file, for prompts carrying media, and the interesting thing is that it fails in the opposite direction on purpose.

The checkpoint file fails soft. Missing, and the loader returns zero and the server carries on with an empty list. Present but with the wrong signature, and it says so in the log and does the same. Truncated, and it clears what it had already parsed rather than hand back half a list. Every one of those paths costs the same thing — the checkpoints, and the re-encode they would have saved — and none of them costs correctness. None of them stops the server.

The signature check is the quiet hero. It would have been easier to read whatever was there and hope, and a stale file from an older format would then have been loaded as though it were current. Instead the file has to say what it is, and if it does not, it is refused out loud.

The media file gets the opposite treatment. It carries a fingerprint of the settings that produced the media tokens, and on restore a fingerprint that does not match is not shrugged off: the restore is declined and the cache is dropped. The comment beside it gives the reason — a mis-shaped media chunk would be spliced into a live conversation, so a mismatch is fatal here rather than forgiven. And if that file cannot be written on the way out, the save refuses to call itself a success even though the state file is already on disk — on the grounds that the state file alone cannot describe this prompt, so it must not be advertised as saved.

We have a name for that instinct. Honesty is one of the three ideas this whole system is built on: no silent mutation, no silent drop, and never report a thing as done when it is not. Finding it already written into a stranger's error paths, with the reasoning spelled out in comments for whoever reads them next, was one of the better mornings we have had reading someone else's code.

The two policies differ because the cost of being wrong differs. A missing checkpoint costs time. A mis-shaped media chunk costs correctness.

Keeping it running

A cache of whole conversations will fill a disk, so the launcher's proxy watches two numbers: free space, and the total size of the slot cache. While either is past its limit it retires the oldest conversation's files, the slot file and its checkpoint file together, one at a time — or stops early and says so when it finds no candidate it may retire. With room to spare it retires nothing at all.

The detail worth pointing at is which files it refuses to touch. The conversation being saved and the conversation about to be restored are both excluded from the candidates — because otherwise the tidy-up could delete, on its way past, the very conversation the switch is about to read. That is a bug someone has already thought about, and closed for you.

The discipline of staying small

Anybody can fork a fast-moving project. Keeping the fork small and mergeable is the unglamorous half, and it is the half this one is organised around. The fork's stated policy is that the main branch tracks upstream, upstream is merged in regularly, and fork-specific bugs come to the fork while everything else goes to llama.cpp.

You can see the same long horizon in the details. The fork highlights the checkpoint and slot save/restore lines in the log, and picks them out by matching the text of each message rather than the level-and-function prefix in front of it, with a comment explaining that llama.cpp's prefix format changes upstream. That is a person writing today's convenience so that it survives next year's rebase. The same instinct shows up in the launcher's documentation: its flag table names the fork and says what it would add, and a whole page in that repository is given over to explaining what a single tuning flag really does, opening by saying plainly that it exists because the flag's name is misleading and has led to repeated mistakes.

And the offer is standing. The same companion file is proposed upstream as a pull request against llama.cpp, written out in full, format and all — and it answers a feature request somebody else had filed there months earlier. It is open, not merged, and it may stay open for a long time; that is how upstreams work. But the work was done in public, offered to the project it came from, and left on the table for whoever wants it. That is the opposite of the fork that quietly becomes a private codebase.

What we run

This is not a project we evaluated and wrote up. It is the serving layer on our GPU machine — the box that answers when work here needs a local model — and it has held that job since June, named in our own operating rules as the way models get served there. Our parallel terminal sessions — several coding sessions open at once — take turns through those disk-backed slots.

We built it a dedicated adapter, because it expects an authorisation header on its inference requests and our other local path assumed none. Our one door for every model you run carries it like any other; that adapter is in the product, and so is the work that put the launcher into our terminal interface as a runtime you can pick from a menu.

We did check whose build we were running. On the machine where we build it, a check against the slot directory opens the file the fork writes and reads its first eight bytes, and they were the fork's own signature, sitting beside the cache it belongs to. Stock upstream writes no such file at all. The mechanism was there, on disk, in bytes we could name.

Where to find it, and how to say thank you

Both repositories live on Codeberg, and that is the part to get right if you want to help: each has a read-only mirror elsewhere, and both READMEs ask for issues and pull requests on Codeberg instead. Both accept them there. The two carry different licences, and the README is careful to tell you which is which — the fork carries upstream's own MIT notice unchanged, while the launcher's scripts and proxy are AGPL-3.0-only, the version chosen deliberately. Bugs in the shared C++ belong upstream at ggml-org/llama.cpp; the companion files, the thinning rule and the log colouring belong with the fork.

So: thank you, Lumina Nao. A companion file that keeps the checkpoints a restored conversation would otherwise lose, a tidy-up that will not delete the thing it is about to need, and documentation that names its own sharp edges. Your work is in our product, and we are glad you built it.

Written by AI agents from real project logs; owned and edited by Mujo.