Some things are worth remembering and some aren't. A signed agreement is worth remembering. So is a message you sent, or a member you let into a group. Where someone's cursor is on a shared canvas right now is not. Nor is whether a friend is online, or which way a person on a live map is heading. The value changes constantly, and only the latest reading means anything.

NAOMS gives these two kinds of state names. The permanent kind are vows: things someone commits to, signed, kept, and replayable to anyone who was away when they happened. The passing kind is weather. This is the story of weather being written down, and the ruling that it shouldn't be: what happens when the line between the two gets blurred.

How it is supposed to work

Vows go on a chain. A chain is an append-only log belonging to a space (you, a friendship, a group), where every entry is signed by whoever wrote it and linked to the entry before it. Members replay each other's entries, so a device that was offline catches up when it comes back. The searchable tables the app reads from, which NAOMS calls the graph, are not a second original. They are a projection, rebuilt at any time by replaying the chain. The chain is the record. The graph is a view of it.

Weather is different. It should never need replaying, because by the time anyone replays it, it is out of date. What it needs is to be cheap, fast, and able to merge. If two of your devices both update your presence, they should settle on one answer without a server deciding.

The classic tool for that kind of merging is a CRDT, a "conflict-free replicated data type." It is a data structure designed so that several copies can be changed independently and then merged, with every copy arriving at the same result whatever order the changes came in. NAOMS uses a CRDT library called Loro. The design documents give it a deliberately narrow role: "a narrow tool for documents whose merge semantics benefit" from it, such as short-lived scratch state during a ceremony (a multi-step exchange such as pairing a new device), while "chains are the primary substrate."

A CRDT also has a property worth knowing. When something changes, you don't need to send or store the whole document. You send the change, a few bytes that describe what moved, and every other copy applies it.

Merging weather doesn't even need that much machinery. When two updates disagree, the newest one wins. The store NAOMS ended up building for weather is exactly that: a plain map in memory with a latest-wins rule.

So the intended picture is simple. Vows go to the chain, and the graph is rebuilt from the chain. Weather stays small and moves as small changes, and nothing about it needs to be kept.

What was actually happening

It started with the owner asking why a monitor kept showing the service on one of our machines as slow to answer. That machine runs our own continuous-integration system on NAOMS, and its database holds hundreds of thousands of chain entries. The numbers below are from that one machine.

The measurement behind that question was stark. Over a three-hour sample on that machine, the service's main loop, the single thread that answers every request, was blocked for 79.2% of wall-clock time. The worst single block lasted 68 seconds. Nearly all of the recorded blocks were database calls.

Several things contributed. When the CRDT turned up in the numbers, the owner asked: "why is loro writing anything? it should only operate on weather which we hardly use."

By default, every time anything was written to the graph of any chain, the code also copied that change into a CRDT document holding the whole graph of that chain. That meant every node and every link, not just weather. Then, on the main path, each change exported the entire document and wrote it to the database, replacing the previous copy.

That is the opposite of what a CRDT is for. The largest document had grown to 37.3 MB and had been rewritten 50,949 times. Across all 144 stored CRDT documents there had been 183,972 full rewrites. By our estimate, averaged over the database's life, that machine wrote between 46 and 92 GB a day of these rewrites, while the whole database grew by only about a tenth of a gigabyte a day. That is roughly 385 bytes written for every byte the database actually gained. On that main path the rewrite ran synchronously, on the thread that answers requests. We counted it as the leading candidate for the part of the blocking where the thread sat waiting on disk rather than computing. We didn't prove that by lining up individual writes against individual stalls.

And the sync all this was meant to serve did not work. The receiving side of a peer (another device) accepts at most 64 KB per message, and the largest document was about 570 times that. In the log we checked, all 1,402 attempts to push these documents failed before getting that far, while being packed for sending ("buffer exceeds maximum length").

So a tool for passing state was doing three things wrong. It was holding a copy of the graph, which can already be rebuilt from the chain. It was storing that state by rewriting all of it after every change. And it was trying to sync it in a form that could never arrive.

The ruling

The obvious fix was to make the CRDT smarter: store changes instead of whole snapshots, or batch the writes. The owner went further. Asked what to do with the service's records about its own activity, he answered:

"drop entirely" … "if it is needed, then lets see if we can store it in 'Weather' — if weather becomes not written to the database"

That answer also settled what weather is: something that is not written to the database. The ruling makes two decisions, and it takes in a third that was already planned:

1. Weather is not written to the database at all. It lives in memory. If the service restarts, current weather is gone, and that's intended. It was never a record.
2. Observational telemetry is dropped, meaning records the service wrote about its own routine activity, such as an entry every time an internal helper ran. If one of those turns out to be needed, it may become weather, and only because weather is no longer stored.
3. Vows stop feeding the CRDT. Permanent state already has its record on the chain and its view in the graph. A third copy, in a different format, does nothing for it.

The trade-off is real and it is accepted. Weather that only lives in memory can't be rebuilt from anything after a restart. For presence, typing indicators, or a live location, that's the correct behaviour. Those are exactly the things that should start fresh.

flowchart LR
  E["A change happens"] --> Q{"Vow or weather?"}
  Q -- "vow" --> C["Signed onto the chain"] --> G["Graph rebuilt from the chain"]
  Q -- "weather" --> M["Held in memory only"] --> P["Sent to whoever is online"]
  M -. "restart" .-> X["Gone, by design"]

What happened next

The ruling was made on 15 September. The code followed in steps.

• 16 September: a gate stopped vows from being copied into the CRDT. The gate is written so that a kind of event nobody has classified yet keeps its old behaviour instead of silently losing data. The next day, the largest document on that machine was sampled six times over twenty minutes. It was not rewritten once, while 175 new chain entries landed in the same window. That was one window on one machine, and the bytes already written were still on disk. The gate did narrow the set of documents still being written, from 144 to 4, but it did not stop the rewriting: over the next four days that same largest document was rewritten another 767 times. That is why the next step removed the stored copy rather than filtering what went into it.
• 17 September: the service stopped writing, by default, its most common kind of record about itself. On the machine we measured, that record type was the largest in the database by count.
• 21 September: the stored copy was removed. The code that created and wrote the table, the code that wrote whole snapshots, and the copy of chain data into the CRDT as each entry was processed were deleted. Existing databases keep the old table, but nothing writes to it. The replacement for weather is an in-memory store with no database access of any kind. The store has no way to write to disk, so there is no setting to get wrong. Loro stays, but it no longer keeps documents of its own on disk; its main new job is inside individual records: on the desktop and phone services, an edit to a note is now stored as a small change, with a full copy written only now and then as a new starting point.

We are deliberately not quoting a before-and-after figure for the blocked thread. Other fixes landed the same week and moved that number too, and the machine we measured had not yet restarted onto the final deletion as of 22 September.

The general lesson

Two lessons, and both apply outside NAOMS.

Mirroring derived state is a cost with no owner. The graph could already be rebuilt from the chain. A second copy in a CRDT added no information, only another thing to keep in step and store. Before you replicate something, ask whether it can be derived. If it can, a second copy needs a reason, such as faster reads. This one had none: the graph was rebuilt from the chain anyway, and the sync the copy existed for never worked.

Storing a snapshot per change turns small writes into huge ones. Writing the whole document every time one field changes costs as much as the whole document, not as much as the change. It is easy to miss, because on day one the document is small. Here the cost grew with the document for weeks before anyone asked why a tool for passing state was writing to disk at all.

The permanent record is one kind of thing and passing state is another. They need different storage, not one store tuned to suit both. For where the permanent record's own durability costs are paid, see Where durability actually gets paid, and for the three kinds of sync that sit on top of it, Three kinds of sync, one engine underneath.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.