How a change gets into the product

A change reaches the product through a pipeline: a fixed sequence of steps a proposed change must pass, in order, before it becomes part of the product. The sequence is written down in a single file — just under 3,000 lines, describing 21 pipelines, of which the one guarding the main line (the single shared version of the project that releases are cut from) runs 28 stages.

A stage is one step in that sequence. Some of them judge the change — three reviewers read it, the types are checked, the whole test suite runs. Others do work: pulling in the latest main line, resolving conflicts, building a working copy, merging, publishing documentation and downloads. Both kinds count toward the 28, and it is worth being exact about that, because "28 checks" would be the wrong denominator; the judging set is roughly half.

flowchart TD
    A[change submitted] --> B{does this change
anything at all?}
    B -->|no| X[rejected, cheaply]
    B -->|yes| C[three reviewers read the difference
design · tests · code]
    C --> D[pull in the latest main line]
    D --> E{conflicts?}
    E -->|yes| F[a reviewer resolves them]
    F --> D
    E -->|no| G[build a working copy]
    G --> H[style · licences · types
the full test suite]
    H --> I{is the test machine
actually healthy?}
    I -->|no| Y[the machine broke,
not your change]
    I -->|yes| J[merge it in]
    J --> Z[publish documentation and downloads]
    APV["a human approves"]:::skipped -.->|skipped, and it says so| J
    classDef skipped stroke-dasharray: 5 5,color:#888

That dashed box is the argument. A conventional pipeline has an approval step there — a named person or team who signs off, after which the change is cleared to land. This one does not. The instruction, in the plain-language equivalent of the words used, is: no change request must be approved by any human or machine, and the pipeline description must say specifically that no approval step is necessary, because the pipeline itself is the approval step.

Two things about that are easy to misread.

Judging is not approving. Three of those 28 stages judge the change, and they are not mechanical checks at all — they are language models reading the actual difference a change makes, one looking at design, one at tests, one at code. Any of them can reject it. What none of them can do is pass it. There is no verdict any reviewer can render that means "cleared to land"; they can only fail to find a reason to stop it. Rejection and approval point in opposite directions, and only one of them is delegated here. (A previous verdict is never a floor, either: a reviewer running again renders a fresh judgement and is free to reject what was let through before.)

A refusal cannot be re-rolled. Three of the reviewers are language models, so the obvious worry is that a rejected change could simply be resubmitted until the dice land green. It cannot: submitting the same branch at the same commit is treated as the same request and starts no new run. A fresh verdict requires a fresh commit — which is to say, actually changing the code. Nothing in the path lets a human overrule a refusal either.

The default is the opposite of this, and the fallback is loud. The software running this pipeline ships with the human approval gate on — a pipeline description that says nothing about approval gets one, which is the secure default and the right one for most people. This project turns it off deliberately, on one line. And if an approval stage is present anyway, the runner steps over it and announces that it did, emitting a recorded event, rather than either forging a signature or parking the change forever on one nobody is coming to give. Anyone running the same software with no configuration of their own gets an approval step, not this.

The older, hand-driven route now refuses on sight, naming what happened to the last people who used it: more than twenty submissions had been enqueued into a queue nothing was draining, and they simply sat there. Past that refusal there is a single documented escape, and it will not run unless you type a written reason that then gets recorded. How often it has been used, we do not know — the commit history is the wrong place to look for it, and we have not looked in the right one. Worth saying anyway, because a bypass you do not mention is just a default you have hidden.

The final stages are the ones that put documentation and downloads in front of people, over a network that needs no server.

What was wrong

A cutover looks finished long before it works. The demonstration goes fine. Then the new thing carries real traffic for the first time and reports, for hours, that it has completed zero of its stages — while the small server it is quietly hammering reboots twice under the load.

That happened here on 26 August, three days after the old route was switched off.

What actually changed, and when

The interesting thing about this cutover is that the pipeline was not new. It had been running for weeks: its own merges appear in the main line on 22 of the 25 days from 5 to 29 August. Three days are missing, and two of them are 22 and 23 August — the pipeline goes quiet on exactly the days the handover happens.

What ended on 23 August was the other lane — one automated lander that a person started, watched, and restarted whenever it wedged. Measured rather than assumed, that lander puts changes in every day through 22 August, 169 of them on the 18th and 174 on the 20th. On 23 August it lands 5. It never lands anything again.

So this was not the day the new thing switched on. It was the day the old thing switched off, and the pipeline had to carry the load alone. What follows is what that exposed.

The part nobody puts in the diagram

The repairs were unglamorous and each is a sentence:

• Every run copied the whole repository across the network from a small machine, then threw the copy away — pointless since 6 August, when a later stage started linking to it instead.
• The disk check measured the disk the project lived on, not the full in-memory scratch area everything was actually dying in. A true answer to the wrong question.
• A listing command waited seven hours on a connection whose far end had rebooted. Keepalives were off, no timeout was set, and the job ran every five minutes.
• The watchdog killed workers for one quiet gap, having collected — and never consulted — the count of work they had already produced.
• Two scheduled jobs had never once succeeded, and nothing said so.

One defect wore all those faces: infrastructure failures were recorded as verdicts on the change. A machine running out of space is not evidence that your work is wrong. Neither is a network fault, nor a batch killed by a signal — a killed batch was never judged at all, which is a different thing from being judged and failing. Each had to be taught to say "the machine broke" instead of "you broke it."

Where this honestly stands

The hand-driven lane is gone and refuses new submissions. Every defect above is fixed and landed, and the pipeline has been carrying work for most of the month.

Two things are not established. Daily arrivals since the handover run 9, 8, 2, 12 and 9, against 169 and 174 on the old lane's busiest days — but those counts do not separate what the pipeline landed from what was landed by hand, so a like-for-like throughput figure is not something we can honestly produce yet. And during this same week the team was also landing changes by hand, on a branch named for exactly that, while the pipeline was repaired underneath them. That is what a cutover week looks like.

What has been bought is narrower than "we automated our process." A change that goes through the gate meets the same 28 stages, in the same order, whether it arrives at 09:00 on a Monday or 15:00 on a Sunday. Nobody has the authority to wave one through — which also means nobody carries the quiet weight of being the person who could have.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.