How a new kind of record gets filed

Almost everything that happens in NAOMS is written down twice.

The first place is the log: an append-only sequence of small entries — each one signed by whoever wrote it, each one pointing at the entry before it, and never edited after the fact. That is the record of what happened, and it is the thing that has to be right. The second place is an ordinary local database holding a graph of things and the links between them, so the system can answer a question instantly instead of re-reading its own history. That database is a projection: a re-arrangement of the log that can be deleted at any time and rebuilt by replaying the log from the beginning. We have written about that split before; this is about the machinery in the middle of it. What that machinery writes into did not change: the same graph, the same shape of table.

Because something has to do the re-arranging. An entry arrives saying a task was created, and something has to decide that this means a row of kind task, with these fields as properties and that field as a link to a person. For most of the project's life, that something was a hand-written function per kind of entry: add a feature, write its filing rule. Two dozen files of them had accumulated, and every one was a place where the log and the projection could quietly disagree.

The reason it took a function per kind is worth sitting with, because it is not laziness. A plain data payload has no self-description. If an entry says its identifier is abc and its status is open, a general-purpose filing engine cannot tell whether abc is the thing's name or a pointer to something else, whether status is a property or a relationship, or what kind of thing the entry is about at all. The meaning lived in the reader, not in the data — so every new kind of data needed a new reader.

Facts in three parts

What landed instead is a change to the shape of what gets written. Entries now carry their content as triples: facts split into exactly three parts — a subject (what the fact is about), a predicate (which property or relationship), and an object (the value, or another subject). "This task has the title Ship the thing." "This task was created by that person." It is the data model behind RDF, the Resource Description Framework — a long-standing web standard for describing things in a way that other people's software can read without a private agreement with yours. We write triples using JSON-LD, which is ordinary JSON with a small header explaining how its short names expand into full, globally unique identifiers.

The reason this matters for filing is one specific triple. Among the facts an entry carries is a type fact — this subject is a task — expressed with a standard predicate any RDF reader already understands. The kind is no longer something the reading code has to know. It is in the data.

So the filing engine becomes one algorithm, short enough to state in full:

1. Parse the payload into triples.
2. Group them by subject — each subject becomes one row.
3. The type fact gives the row its kind.
4. Plain values — a piece of text, a number, a date — become properties of the row.
5. Values that are identifiers pointing at other subjects become links.
6. Merge the result into the existing row if one is already there, so a later entry can update a few fields without restating everything.
7. Stamp where it came from, and write.

flowchart TD
  A["A feature writes an entry"] --> B{"Is this entry eligible
to be wrapped as triples?"}
  B -- "no" --> C["Written to the log as plain data.
No row is projected."]
  B -- "yes" --> D["Wrapped as self-describing triples"]
  D --> E["Signed, appended to the log"]
  E --> F["Validating handlers,
which may refuse the write"]
  F -- "refused" --> G["No row. The refusal is the point."]
  F -- "allowed" --> H["One general engine:
group by subject, type to kind,
values to properties, identifiers to links"]
  H --> I["Row written to the local graph
(authoritative)"]
  I --> J["Handlers for what a generic rule
cannot infer: links, cascades,
computed fields"]
  I --> K["Optional query index
(off in a default build,
best-effort, read-only)"]

That is the engine. It does not know what a task is, or a shared group, or a credential. It is also forbidden to create a short list of reserved kinds — the rows recording who is a member of what, or what authority has been granted — and when it meets one it refuses, logs the refusal, and steps aside so that the handler with authority over that kind can claim the entry instead.

A feature that wants a new kind of record writes triples under a name-space of its own — a prefix that makes its property names globally unique, declared once in the small file where a feature states what it is — and the rows appear. No filing code. The name-spacing does quiet work there: two features can both have a title without arguing about it, because each one's title is a different full identifier underneath.

What the data is not allowed to say about itself

Self-describing is not the same as self-attesting, and the engine draws that line hard. After it has read the triples, it stamps three things the payload does not get a vote on: which log the entry came from, which entry it was, and who signed it. That third one is resolved rather than copied — some kinds of row are classed as private and record a pseudonym or nothing at all, and a sensitive kind that nobody declared resolves to nothing rather than to a name. If a payload arrives carrying its own origin tag claiming to be one of the system's own internal sources, the claim is thrown away and the rejection is logged. So is anything oversized: an entry may carry at most five hundred triples and a megabyte of payload, and constructs the parser does not implement are rejected rather than half-interpreted.

This is the part worth stealing if you take nothing else. The moment you let data describe itself, you have to be explicit about which parts of the description are the writer's business and which are the host's. Content: the writer's. Provenance — where a record came from: never, and the answer the system gives is a pointer rather than an attribution, for a reason the next paragraph gets to.

The merge in step 6 decides the other half of that: what a later entry can do to an earlier one. Later values overlay earlier ones on the same subject. A small set of creation facts is held back from that — which log the row belongs to, when it was created, who created it — and stays as first written. The rest of the stamp does not: on a row several entries have contributed to, the pointer to the entry that wrote it and the signer that entry resolved to both track the newest write, not the oldest. Which settles what the stamp actually means: not who is responsible for this row, but which entry last wrote it. A different sentence, and worth knowing before you read anything off it. Empty values are skipped, which is what makes a partial update non-destructive, and equally means clearing a field is not something the generic path expresses; a row that needs a value removed is written by a handler with authority over that kind. What decides which entries may overlay your rows at all is a question about replication and trust, and not one this piece takes on.

The default, and the fallback

The default on the write path is automatic, but conditional, and the conditions are ordered. Several cases are decided before anything else — an unparseable payload, an already-wrapped one, a sealed envelope whose encryption must stay outermost, an entry its feature declared as deliberately not projected. Each passes straight through unwrapped, whatever the caller asked for. After that, a caller may pass the format explicitly and that wins; otherwise the wrapping happens only if the entry's kind has a name-space registered. Anything that falls through is still signed and still in the log. It simply produces no row.

That last case is the failure mode worth naming, because it is silent by construction: the feature works, the entry lands, and the screen that reads the graph shows nothing. It is logged on both sides — once where the wrapping was skipped, and once where an unwrapped entry reached the engine with a name-space that was registered, which means somebody clearly wanted a row. Both say so once per kind of entry per start. Deliberately a warning rather than a refusal, because unprojected entries are a legitimate thing to write and blocking them would break real callers.

The fallback on the read path is that the fancy layer is optional and nothing depends on it. Because entries are now triples, they can also be fed to a triple store answering SPARQL queries — SPARQL being to triples roughly what SQL is to tables, and worth wanting for questions that cut across features rather than sitting inside one. That index is behind an opt-in build flag, so a standard build does not contain it at all. Where it is present, the feed is best-effort at every step: if it fails to start, if an insert throws, the failure is logged and the write goes on. The local graph database remains the authoritative store and the triple index is a read-only overlay holding no truth of its own.

Where the exceptions went

A general rule always meets cases it cannot express, and pretending otherwise is how general rules die. Here the escape hatch is a small pipeline of handlers around the write. The first phase runs before the row is written and may refuse the write outright — that is where a validation says no. The second runs after and does what a generic rule cannot infer: drawing a link only a domain rule knows about, cascading a change, computing a field.

Handlers that ship with the system run in the same process. A handler that arrived as an installed plugin runs in a separate worker and reaches the graph through a gateway the system owns rather than the plugin. That placement is the point, and it is the same secure-by-default instinct the runtime gives us, applied one level in: whatever policy governs what an extension may touch has to live on the system's side of that boundary, because the extension's side of it is not ours to trust.

The accounting

Twenty-two filing files, a little under three and a half thousand lines, deleted in the landing change. The engine that replaced them is around seven hundred and seventy lines.

Those two numbers do not belong in the same sentence as a net saving. The parser, the name-space registry, the wrapping path, the handler pipeline and the query feed are all new, all load-bearing, and none is counted in that seven hundred and seventy. Add them up and the machinery comes out larger than what it replaced, by a margin that has only grown as edge cases landed. This was never a line-count win, and the honest headline number is not a number.

Nor is the interoperability payoff banked. The reason to prefer a standard is that other people's software can read our entries without a private agreement — but a standard nobody has yet read our data through is a bet, not a result, and the one concrete thing RDF buys, cross-feature queries, sits behind that opt-in flag. Which leaves an obvious question: why not a plain typed envelope — a kind marker, an identifier, some properties, some references, with reverse-domain names to keep them from colliding? That is close to what we have, it is far smaller, and it needs no JSON-LD parser. The difference is the bet, and it is worth stating as a bet rather than as an argument won.

What is not a bet is where meaning lives. It used to live in code: each kind of record had a reader that knew what it was, and the two could drift apart without anything noticing. Now it lives in the record. The projection cannot disagree with the log about what a thing is, because it is not making that judgement any more — it is reading it. And the projection stays genuinely disposable: rebuilding one replays the log's entries through the same step, in a fixed, deterministic order — all of them but the ones it has been asked to forget, which are deliberately never re-folded back into the projection, because a system whose forgetting undoes itself on the next rebuild has not forgotten anything. A rebuild is therefore a function of what the log still holds, not of everything it ever held, and that is the intended reading rather than a lapse in it. That guarantee covers the engine's own output; the second-phase handlers hanging off it are side effects, and side effects are never what a replay guarantee covers — which is a good reason to keep as much as possible in the part that is replayable. There is no migration story to go with that, deliberately: while the system is pre-release, a stale projection is fixed by rebuilding it rather than by shipping migration code.

The price is the part this piece has tried not to bury: you must decide, up front, which parts of the self-description the data is not allowed to make, and you must make the silent skip visible. Self-describing data is not a way of having fewer rules — it is a way of having the rules in one place, where they can be read. Which is the same one-idea-under-everything move the signed log itself is, and the precondition for a schema travelling between groups, which is the next thing we want it for.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.