What the receipt is for

In NAOMS nothing in the middle tells a sender its message arrived. Two devices talk over a connection between the two of them; a relay can help them find a path to each other when a direct one is not available, but on the live path nothing in between holds the message for later delivery or vouches for it. The evidence has to come from the recipient or it does not exist.

Most messages do without that evidence, and that is a deliberate choice: they are sent, and the sending side moves on. One exchange asks for more. When two devices meet for the first time and establish who each other is, the reply that carries the introduction back is sent on a path that waits for proof of arrival. It holds the connection open and waits for a short reply — an acknowledgement, or receipt: a few bytes carrying back the identifier the sender attached to its own message, so it knows which of its outstanding waits to close. The time between sending and getting that receipt is the round trip, and it is the sender's evidence that anything landed.

Two boundaries on that. The message that opens an introduction does not use this path at all: it is sent without a receipt, and reports its delivery through a separate signal. And on our mobile runtime, where reading a reply back off the connection is not available at all, the same waiting send degrades to reporting success once the bytes are away.

The waiting is bounded, because a device that waits forever is a device that has hung. The hold is ten seconds for the introduction-class messages this piece is about; five for everything else, though nothing else asks for one today. If the window closes with nothing in it, the sender treats the message as lost and tries again: up to three attempts, waiting nothing, then about two seconds, then about five, with up to a fifth of a second of randomness added on top so that two devices which fall out of step do not settle into retrying in lockstep forever.

A retry is a fresh copy of a message that may well have arrived, so the receiving side keeps a small table of what it has already seen, keyed by who sent it, which message it was, and which attempt it is. That last part of the key decides exactly what the table protects against: a copy of the same attempt arriving twice is answered but not acted on twice. A genuine re-send is a new attempt, carries a new number, and is therefore not a repeat at all — it is handled again. The table holds the most recent thousand-odd entries and forgets past that. On the sending side there is a matching bound: at most 256 outstanding waits at a time, and when that is full a new send is refused with a named error rather than quietly evicting somebody else's pending wait. A refusal the caller can see is worth more than a stranding it cannot.

None of that is unusual. The interesting design question is narrower, and it is the whole subject of this piece:

What question does the receipt answer?

There are two honest candidates. The receipt can mean your message arrived here and passed the checks made before anything acts on it. Or it can mean your message arrived and everything it caused has finished. Both are defensible. They are not the same promise, and a system has to pick one on purpose, because the sender's retry behaviour is built on whichever one it gets.

Ours means the first, deliberately. It is a statement about the transport — the layer that moves bytes between two machines — and its answer is available almost immediately, because the gate it stands behind is cheap. For an ordinary encrypted message that gate is authentication: it has to be cryptographically bound to the identity it claims to come from, and if it is not, it is dropped there and gets no receipt. The sender sees a closed window and retries, which is the correct outcome for a message that should never have been accepted.

The messages that establish trust in the first place — including the introduction reply this piece is about — are exempt from that gate, by necessity: the binding it tests for is the very thing they exist to create, so it cannot be a precondition for accepting them. For those messages the receipt means the bytes arrived and parsed, and no more than that. The sender-identity and trust-record checks are not deferred for them, they are skipped — that is what exempt means. What is left at that layer is a rate limit shared by everyone sending that kind of message, not attributed to the identity the message claims, because that claim is precisely what has not been established yet. The checking these messages need is the handler's own, and it happens after the receipt has gone: it looks for a pending introduction of our own with that sender, and when the reply carries a signed identity document it verifies that signature and checks the document echoes the one-time number we sent with the invitation. That is the trade the exemption forces — the gate cannot run before the message, so it runs inside the handler instead. The checks a message passes before it is believed are the same idea one layer up.

The second candidate is a statement about the application: it would wait for the handler — the code that actually does what the message asked for — to finish. Its answer is available only when that work is done, and how long that takes depends entirely on what was asked. Choosing the first meaning is what keeps the receipt's latency bounded by the network rather than by the slowest thing the receiver might have to do. The rest of this is about what that choice costs.

sequenceDiagram
  participant S as Sender's device
  participant R as Recipient's device
  S->>R: message (carries an id to answer, and which attempt this is)
  R->>R: parse; run the transport's own pre-handler gate
  Note over R: an ordinary message that fails the gate is dropped here,
with no receipt — introduction messages are exempt by design
  R->>R: record it in the already-seen table
  R-->>S: receipt — it got this far
  Note over S: sender stops waiting, stops retrying
  R->>R: run the handler: append to the log,
rebuild the local index, tell the neighbours
  Note over R: a failure here is logged on the receiving side,
not carried in the receipt the sender already has

What was wrong

The receipt used to be written after the handler had finished — the second meaning, inherited from the obvious implementation order rather than chosen.

For most messages that would be invisible, because most handlers are quick. For an introduction between two devices it is not. Accepting one means appending entries to the signed log that is the real record of what happened, rebuilding the local index that the interface reads from — the write-to-the-log, read-from-the-projection split — and then telling the neighbouring devices about it. That is real work, and it does not reliably fit inside a ten-second window on a first contact.

When it did not fit, the sender saw a closed window with nothing in it. From the outside that is indistinguishable from a message that never arrived, so the sender did the correct thing for the wrong situation: it sent the message again.

There is a second half to this, and it is the more interesting one. Because each retry carries its own attempt number, the receiving side did not recognise it as a copy of something it was already acting on. It was a new key, so it was new work: the introduction handler started again while the first one was still running. The retries did not merely cost time. They multiplied the work on a receiver that was, by construction, already too busy to answer in time. That is the shape of a metastable failure — a system whose response to being overloaded is to generate more load — and what keeps it from running away here is the retry ceiling: the sender gives up after three attempts, so the retries stop rather than compounding. The ordering test's own failure message and the code comment beside the fix both attribute a cross-host arrival of roughly six minutes to this coupling, tracing it to a diagnosis made in May; we did not re-measure that figure and we are not asserting it here.

What we did measure is the mechanism. With a stand-in handler that deliberately sleeps for twenty seconds in place of the introduction response, on the old ordering the sender got no receipt at all, timing out twice — at around ten seconds each on one machine, and again at around ten seconds each between two machines over a private network. One message type, one instrumented handler, two venues, a single run per cell: enough to characterise the coupling, not a claim about how long anything takes on an ordinary day.

The general shape is worth carrying away even if you never write a peer-to-peer transport. A receipt whose latency is tied to the work converts slowness into apparent loss — and the standard remedy for loss is retry, which adds load to a system that was already slow. That failure mode compounds rather than degrading gracefully, it is invisible in any test where the handler is fast, and a duplicate-suppression table will not save you from it unless the thing it suppresses is the thing your sender actually sends.

What changed

The repair is an ordering, not an algorithm. Inside the step that receives a message, the receipt is now sent immediately after the already-seen key is recorded and before the handler runs. Recording first is what keeps the duplicate-copy contract intact: a second copy of one attempt is still answered, and still not acted on twice.

On the same instrumented setup, after the change the sender was acknowledged on its first attempt — 60 ms on one machine, 281 ms between two machines over the private network. On the same-host leg, where both timestamps come off one clock, the slow handler's completion was observed strictly after the sender already had its receipt. That last detail is the one that matters. Two fast numbers would only show that nothing was slow that day; the ordering of those two events is what shows the receipt is no longer waiting on the work.

There was a second place a receipt could be written, and it is now gone. When a message arrives from someone whose introduction is still in flight — the record establishing who they are has not landed yet — the receiver holds it in a small queue and retries the handler rather than rejecting it. That queue used to send its own late receipt or refusal when it finally succeeded or gave up. After the change that reply is a dead send: the sender's wait was resolved at arrival time and nothing is listening. So it was removed, along with the helper that wrote it, on the reasoning that a dormant function for the old behaviour is an invitation to quietly rebuild it.

What it costs

Removing that second reply takes something away. When that queue gives up because the introduction was abandoned or the message sat past its ceiling, the sender is not told on the wire. That give-up is counted, and an audit event is emitted through the same unreachable path used when a record cannot be applied because its history is missing. Honest limit: that emission is best-effort — it is fired without being awaited, and a failure to write it is logged as a warning rather than retried.

That is the shape of the whole trade. The receipt is now a narrower promise, and the sender has to get the rest of its answer somewhere else. It says the message reached the far side and was recorded. It says nothing about whether the handler succeeded. For ordinary traffic it says nothing about the authorization checks either — those run inside the dispatch step, after the receipt has gone out. For the introduction messages it actually carries today, the sender-identity and trust-record checks were never going to run at all. If the handler throws, the receiving side logs it, and the consequence reaches the sender through whatever the far side does next rather than through the receipt it already holds. Code that needs to know a thing was done has to look for the consequence rather than the acknowledgement. And a closed window has several causes — among them a message that never arrived, one refused before anything acted on it, and a receipt that failed on its way back — which the sender cannot tell apart. After the third attempt it stops asking, records the timeout on its own side, and moves on.

The success flag the reply still carries is now always true. It no longer distinguishes anything — it is true whenever a receipt is sent at all — so it carries exactly what the receipt carries: arrival, and nothing about the work.

And a smaller one, which closes the loop with the first half of this piece: a receipt that fails to send is logged on the receiving side, but from the sender's seat it is indistinguishable from silence. So the sender retries — and that retry is new work, for the same reason every other retry is.

Two guards hold the ordering in place. The first reads the production file as text and fails if the receipt call ever moves after the handler call, if the already-seen record moves after the receipt, or if a call to the removed helper reappears by name in the deferred queue. It is cheap and runs anywhere, but it checks the order of statements in a source file, not behaviour — and it pins one name. The second is the behavioural one — two real processes, a real connection, that slow stand-in handler — and it fails if the receipt starts waiting on the work again, because the sender would need a second attempt. Its cross-machine legs are run by hand.

If there is one line to take from this: decide what your receipt is promising, write it down next to the code that sends it, and make sure the promise is one you can keep in a bounded time. An acknowledgement that quietly means "and everything worked" is a promise whose delivery time is whatever your slowest handler is having, and every layer above it will read that delay as loss.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.