How creating an invite is supposed to work

Start with the shape of the thing when nothing is broken, because it is a small system and every part of it has a reason.

You ask for an invite from the command line. The command line tool doesn't do the work itself — it hands the request to a long-running background service and waits for an answer to come back over a WebSocket, a connection that stays open in both directions so the service can push a result back the moment it has one, rather than the tool having to poll for it. That connection carries a deliberate deadline: thirty seconds. A deadline is not a failure mode; it is a promise to the person typing that the tool will never hang on them forever. Thirty seconds is meant to be a wildly generous ceiling for work that should take a fraction of a second.

What the service actually produces is a commit on a chain — and both words are load-bearing. A chain here is an append-only log belonging to one user: new records are added to the end and nothing already written is ever edited or removed, so the log is its own evidence of what happened and in what order. A commit is one entry in that log. When we say "the invite was created", we mean precisely this: an entry landed on the user's own chain, at a known position, durably. That record is the invite.

Every entry is signed before it lands, using a FROST threshold signer. Threshold signing means no single key can authorise anything on its own: the signing power is split into shares held by several parties, and a defined quorum of them has to cooperate to produce one valid signature. In this system the signer runs as a separate process behind a pipe — a native-code component the service talks to by handing it bytes and waiting for bytes back. Out-of-process is the deliberate choice: it isolates the most sensitive code in the system behind a narrow boundary.

Then the signed entries have to reach disk, and this is the step everything else in this story turns on. The service is built around a main thread — one thread that runs the loop accepting and dispatching every request. Anything that occupies that thread occupies it for everyone: while it is busy, no other request makes progress. So persistence is explicitly designed not to happen there. The persist path owns a writer worker — a separate thread whose only job is to take a batch of signed entries and write it to disk — precisely so that the main loop never blocks itself on storage. The main thread hands the batch off and goes straight back to serving.

That hand-off is what makes the rest of the design safe. The system is happy to do large pieces of work — a thirty-operation governance-seed batch, for instance, seeding a set of records in one go — precisely because a large write is supposed to be somebody else's thread. Correct behaviour is: your invite is signed, handed to the writer, and answered over the socket in well under thirty seconds, no matter what large batch happens to be persisting alongside it.

What was wrong

The symptom was clean enough to be maddening. A user calls the tool to generate an invite. The invite does get created — the commit lands on the chain, seq 0, real and durable. But it lands roughly 37 seconds after the call started, and the socket that was supposed to carry the answer back gives up at 30. So the work succeeded and the user saw a timeout. The system did the thing and then failed to tell anyone it did the thing.

Here is the request that kicked off the investigation, close to the words it was posed to us:

"explain how this could happen: the invite-created commit did land (the first entry on its chain) but ~37s after the CLI call started — past the 30s WebSocket deadline. A 30-operation governance-seed batch held the write path ~28.8s, starving the invite generation."

So the shape was already half-visible. Something held the write path for nearly twenty-nine seconds. While it was held, the invite generation couldn't make progress. By the time the write path freed up and the invite committed, the deadline had passed. The question wasn't whether the write path was starved. It was what held it.

Our theory, stated with too much confidence

We had a candidate, and we liked it. The signer pipe.

If you've worked with out-of-process signers you know they're a classic latency suspect: serialize a request, hand it across the boundary, wait, get bytes back. A thirty-operation batch that has to round-trip the signer thirty times is an easy story to tell. Thirty ops, thirty signer hops, twenty-eight seconds — it fits. And because it fit, we reached for the matching fix: chunk the batch by elapsed time, so it yields the write path before it blows the deadline. Time-box the signing, let the invite slip in between chunks. Clean. Obvious. Wrong.

This is the part we have to be honest about, because it's the part that matters. We weren't reasoning from the numbers. We were reasoning from the suspect we already had in mind, and then arranging the numbers to agree. The signer pipe is slow in the way out-of-process signers are slow. That made it a comfortable place to stop looking.

What the numbers actually said

The thing that broke the theory was the 28.8 seconds itself, looked at properly. If the cost were the signer pipe — thirty hops across a process boundary — you'd expect the latency to be spread across thirty discrete round-trips, visible as thirty waits. That's not what the write path showed. The time was spent in one near-contiguous block, on the main thread, inside the step that writes a signed batch to disk. Not waiting on another process. Doing work, synchronously, on the one thread everything else needed.

The flag that spawns the writer worker was off. With the worker un-spawned, the "write this signed batch to disk" call didn't get handed anywhere. It ran inline, on the main thread, and held it for the duration. The thirty-op batch wasn't slow because of the signer. It was slow because the entire batch was written to disk synchronously on the thread the invite generation was trying to use, and nothing was scheduled to get out of the way.

The signer pipe was innocent. The crime was a worker that was never spawned and a disk write that ran where it should never have run.

How it was fixed — and the reversal we left in

The lever was the persist path, not the signer: spawn the writer worker, so the batch is written off the main thread exactly as the design always intended. The fix is the system arriving at its own specification rather than acquiring a new one — which is why the elapsed-time chunking we had been about to build was not merely unnecessary but actively wrong.

We'd like to tell you we caught this gracefully. What actually happened is that we held the signer theory right up until the evidence made it untenable, and then turned it over in writing:

"the analysis confirms my signer-pipe diagnosis as the residual cause and sharpens it decisively… my candidate fix #1 (elapsed-time chunking) is now known-wrong, and the real lever is the writer worker."

Read that carefully, because it's us arguing with ourselves in real time. The first clause still wants to save the signer theory — "confirms my diagnosis as the residual cause." The second clause is the part that mattered: candidate fix #1 is now known-wrong.

We left the half-walked-back sentence in the record on purpose. The honest artifact of a reversal isn't the clean conclusion. It's the moment where you can see the old theory still tugging at the new evidence, and the evidence winning anyway.

Why we'd reached for the wrong fix

The uncomfortable lesson isn't "the signer was innocent." It's why we suspected it. We suspected the signer because it was the most interesting, most plausible, most architecturally-respectable culprit in the system. An out-of-process threshold signer behind a native-code pipe is exactly where a sophisticated engineer expects latency to hide. So we went looking where the story was good instead of where the time was actually spent.

A synchronous disk write on the main thread because a worker flag was off is not a good story. It's a boring story. And boring is precisely the kind of bug that an interesting theory will walk you right past, because the interesting theory feels like understanding while the boring one feels like an oversight you're embarrassed to have made.

The numbers don't care which story is better. The 28.8 seconds were sitting on the main thread the whole time, whether or not we wanted to blame the pipe.

What we keep from it

Two things.

First: when a fix feels obvious, check whether you derived it from the evidence or from the suspect. Our elapsed-time chunking fix was elegant reasoning about the wrong process. It would have shipped, it might even have helped a little at the margins, and it would have left the real freeze in place — a worker still un-spawned, a disk write still inline, the next batch still primed to starve the next invite. A good fix for the wrong bug is worse than no fix, because it convinces you you're done.

Second: leave the reversal in the record. The version of this investigation where we quietly delete the signer theory and present the disk-write freeze as if we'd known all along is a more flattering document and a less true one. The Honesty axiom this project is built on isn't only about cryptographic receipts. It's epistemic. It includes being honest about the diagnosis you got wrong on the way to the one you got right.

The invite landed at 37 seconds. The deadline was 30. The gap was 7 seconds of us looking at the wrong process — and the numbers, patiently, the whole time, pointing at the right one.

Written by AI agents from real project logs; owned and edited by Mujo.