The River Remembers: Building a Workflow Engine That Survives a Crash
How a multi-step agreement remembers exactly where it was โ even if you pull the power cord halfway through
Historical note: this describes the workflow engine as it stood in March 2026, in the pre-Rust, graph-database-backed era of NAOMS. The concept evolved substantially afterward; treat this as a design retrospective, not documentation of the current system. Every claim below is traceable to a commit subject from the week of 2026-03-16.
There's a commit in the log titled simply "The River Remembers (workflow engine celebration)" โ one of the project's milestone cards, dated 2026-03-20. The day before it, another one landed reading "Workflow Engine โ COMPLETE โ " (2026-03-19). Those two lines bracket about a week of work, and they name a design problem worth slowing down for: how do you run a multi-step agreement between parties โ possibly between an agent and a human, possibly across machines โ such that pulling the power cord halfway through doesn't lose the thread?
The "river" metaphor in that card title is doing real work. A workflow is a current: it flows from state to state, it has direction, and โ the hard part โ it has to remember where it was even if you walk away and come back. This piece is about how the March engine tried to make the river remember.
The shape of the problem
A workflow in NAOMS isn't a script you run top to bottom. It's a negotiation. Two or more participants agree to a sequence โ propose, review, accept, enact โ and each step might pause for a human, for an agent, or for a condition that isn't true yet. You can't model that as a function call stack, because the stack evaporates the moment the process dies. You need the position in the workflow to be a durable, inspectable fact, not a transient in-memory frame.
That single requirement โ position must be durable โ is what pushes you toward a statechart, and then toward writing that statechart's current state somewhere that outlives the process.
For builders
Phase 1 โ compile the agreement into definitions
The engine landed in five phases, and you can read the architecture straight off the commit subjects. The first one is titled "Phase 1 โ Agreement compiler + 8 workflow definitions."
The key word is compiler. The system didn't hand-code each workflow as bespoke logic. It took an agreement โ a declarative description of who agrees to what sequence โ and compiled it into a workflow definition: the set of valid states and the transitions between them. Eight of these shipped in that first phase. Compiling rather than hand-writing means the rules of "what's a legal next step" are data, not control flow buried in an if-tree. That matters enormously for the later phases, because data can be persisted and inspected; a control-flow tangle can't.
Phase 2 โ the statechart runtime
Phase 2 is "Statechart engine + runtime protocol." This is the heart.
A statechart is a state machine with structure โ nested states, guards on transitions, the discipline that you are always in exactly one well-defined place. The runtime's job is narrow and therefore trustworthy: given the current state and an incoming event, decide whether the transition is legal, and if so, move. No "sort of in two states." No implicit progress. The position is always a single nameable thing.
That narrowness is the feature. Because the runtime only ever does "legal transition or refuse," the entire meaning of an in-flight workflow collapses to one value: its current state. And one value is something you can write to disk.
The rest of this article โ Neo4j persistence and crash-resume (Phase 3), workflow-aware enforcement (Phase 4), federation across machines (Phase 5), and an honest account of what we'd keep and what we'd leave behind โ is behind the paywall. The free preview ends here.
Phase 3 โ persistence and resume (the part that earns the title)
(paid)
Phase 3 is "graph-database persistence + resume," and this is where "the river remembers" stops being a metaphor.
Because Phase 2 reduced an in-flight workflow to a single current state plus its definition, persistence becomes almost boring โ and "boring" is exactly what you want from the component that has to survive a crash. The engine writes the workflow's position into Neo4j, the graph database that was NAOMS's source of truth in that era. When the process restarts, resume reads the position back and reconstitutes the runtime exactly where it left off. No replay-from-scratch, no guessing, no "I think we were about here."
The graph choice is not incidental. A workflow's state, its participants, and its agreement are all relationships โ and a graph stores relationships as first-class edges rather than as foreign keys you reassemble at query time. Persisting a statechart into a graph means the "where are we" question and the "who's involved" question are answered by the same traversal.
Phase 4 โ workflow-aware enforcement
Phase 4 is "Workflow-aware enforcement + 3 agent tools." Persistence lets you remember the workflow; enforcement lets the workflow constrain what's allowed to happen next.
The idea: the policy layer can ask the engine "given where this workflow is, is this action legal right now?" Enforcement becomes contextual. An action that's fine in one state is refused in another, because the engine is the authority on position and position determines permission. Three of these checks were exposed as tools the agents could call, so that they โ not just the core โ could query and advance workflows through the same governed door.
Phase 5 โ federation
The final phase is "Federation, meta-workflow, visualization." Once a workflow's state is a durable, transferable fact, it can move between machines. Federation is the claim that the river can flow across a device boundary without losing its place โ the same resume mechanism, applied across a network seam instead of across a restart. "Meta-workflow" (workflows that coordinate other workflows) and visualization round it out.
What we'd keep, and what we'd leave
Honest retrospective, since the engine itself evolved past this design:
Keep: the discipline of compiling agreements into data. The single best decision here was refusing to express workflows as code. The moment position is data and transitions are data, persistence, resume, enforcement, and federation all fall out of the same property instead of each needing its own bespoke machinery. That principle outlived the implementation.
Keep: statechart narrowness. "Legal transition or refuse, always exactly one state" is a small contract that buys an enormous amount of reliability downstream.
Leave: the hard coupling to Neo4j as the source of truth. Tying durability to one external graph database was right for the bootstrap era and wrong for a local-first system that has to be whole on a single offline device (the Wholeness axiom). The later architecture had to unwind that dependency. (We're describing the direction of that change from the project's stated axioms, not the specific later work that carried it out โ that's a separate article.)
The card title got it right, though. A workflow engine's one real job is to be the thing that remembers the current โ so that when the machine forgets, the river doesn't.
Related: Write to the Log, Read from the Projection ยท Killing Raw SQL.
Written by AI agents from real project logs; owned and edited by Mujo.