You open one of your own photos and find a word attached to it that you never typed. It says dog. It is a wolf.

Something looked at the picture and had an opinion. What matters is what your photo library did with that opinion before you ever saw it: whether it wrote the word down as a fact, whether search now believes it, whether you can take it back, and whether the screen is about to tell you how sure it is in a way that sounds more precise than it is.

One thing up front, because the rest is written in the present tense. Taking a label back is something the system knows how to do and something you cannot yet do from the screen. The rules below are built and running; the buttons are not.

flowchart TD
    P[Your photo] --> N{Can a model
even run?}
    N -->|model or libraries missing
image too large| Z[Zero labels
never a fabricated one]
    N -->|yes| M[Labels proposed,
each with a confidence]
    M -->|no confidence number| R[Rejected outright
never invent one]
    M -->|below the bar for
this kind of analysis| A[Declined, nothing stored]
    M -->|at or above the bar| Q[Stored UNCONFIRMED,
number kept exactly as reported]
    Q --> S[Search: confirmed labels
outrank unconfirmed guesses]
    Q --> C[You correct it] --> V[New CONFIRMED label wins,
old one kept and marked superseded]
    Q --> F[You forget it] --> X[Gone from search,
record and signed entry kept]

How it is supposed to work

What the machine is allowed to say about your picture

Some statements about a photo you asserted — you named an album, you typed a caption. Others are derived: a program read the pixels and proposed one. The difference has to survive being written down, because only one of them has a person behind it.

The program doing the deriving is a vision model — trained on a large pile of labelled images and, shown a new one, returning candidate labels each with a confidence, a number between zero and one for how strongly that label fits by its own reckoning. That is not a probability that the label is true in the world, and two models' numbers are not comparable. One model here sorts a picture into a thousand everyday object categories; another into hundreds of kinds of scene — a beach, a mountain, a beer garden.

Only two categories of statement are accepted at all: things present in the frame, and places — and every analysis that actually runs emits the first kind, so the second is permitted rather than populated. Identifying people from their faces is prohibited outright. On top of that, one short list of grooming and cosmetic labels — wigs, baldness, salons, makeup, surgery — is dropped from your search results and from the photo's detail panel, at the point the data is prepared rather than in either screen. The list matches whole labels exactly rather than by fragment, deliberately: bald eagle is a different label and survives.

Note the size of that guarantee, because it is smaller than it sounds. It is a refusal of two categories plus one exact list of grooming words, applied at two places that read labels and — as this piece comes back to below — not at a third. Clothing is not on the list either, which is why the swimwear terms further down are not caught by it, and nothing here vets the rest of a model's vocabulary.

The bar that decides what gets written down

Below some strength a proposal is noise, and noise filed next to signal is worse than nothing, because nothing can tell them apart later. So each analysis has an admit bar — a confidence a label must reach to be stored at all.

There is no single number here, and the defaults matter. The object pass runs at 0.32, the scene pass at 0.12, and a shared floor of 0.60 applies only to an analysis that supplies no bar of its own. Those low-looking numbers are the arithmetic of the thing: a model choosing between a thousand categories spreads its confidence thin, and the tail of correct answers reaches a long way down — in calibration against real photographs a rightly-identified suit scored 0.23 and a rightly-identified ocean 0.152. A single shared bar would have discarded those exact right answers. Both bars can be overridden by configuration for calibration work.

The fallback has a sharp edge. A supplied bar is honoured only if it is above zero and at most one; anything else falls back to the shared floor. So an analysis that has already done its own admission cannot say "do not gate this again" — the natural way to say that is zero, zero is out of range, and it silently gets the 0.60 floor instead. One of ours is in exactly that position. The effect errs strict rather than permissive, but the lesson generalises: if a legitimate state is expressible only as a value your validator rejects, callers will express it anyway and be silently overruled.

A proposal under the bar is declined rather than stored, and counted in that run's result rather than silently dropped — though be honest about that count: it lives in one debug line, and no part of the product reports your decline rate back to you.

Then the sharper rule. A label that arrives with no confidence number at all is rejected outright. It would be easy to supply a default — half, say, a shrug in numeric form — and that is what an earlier version did. Once written to storage, a synthesized 0.5 is indistinguishable from a 0.5 a model actually reported, and there is no later moment when the difference can be recovered. A missing value has to stay missing.

When nothing can be looked at

The most common real state on a fresh machine is not a wrong label. It is no labels at all.

If the machine-learning runtime or the model files are not installed, the classifier fails and the photo ends up with zero labels. There is no fallback to a cruder method — a wrong label produced by a worse one is not a consolation prize. An image too large to load whole is skipped, also with zero labels. A photo whose bytes have not finished arriving is retried later rather than recorded as empty.

Which creates a problem most systems have and never notice: a photo examined and found empty looks exactly like a photo never examined. So examination itself is recorded — a coverage record, one per photo, written before the found-nothing exit, and only when examination is evidenced by a named model or by output produced. If that record fails to write, the failure is logged loudly and the labels are kept anyway; the photo then reads as not indexed until something looks again. Two limits: the record has a slot for which kind of examination happened and only ever writes one value into it; and one exit — an analysis returning malformed output — precedes the record, so that case still reads as never examined.

Admitted is not trusted

A label that clears the bar is not thereby true. It is stored explicitly as unconfirmed and stays that way however high the number was. Clearing a threshold is a statement about the model's output. Being right is a statement about the world. Only a person moves a label from unconfirmed to confirmed.

The default leans cautious: a stored label with no trust marking at all is read as unconfirmed. Absence is handled differently in the two places on purpose — a missing confidence is a broken promise by whatever produced the label, so it is refused at the door; a missing trust marking is a question about our own records, where the safe reading is the suspicious one.

Search ranks accordingly: a confirmed label outranks an unconfirmed guess, so the machine's opinion can help you find things without being allowed to outvote you.

Correcting is not overwriting

You correct dog to wolf. What does not happen is the obvious thing — the old word being replaced. Wolf is written in the confirmed state, attached to the photo, preferred by search. Dog is marked superseded and stays: out of results, not out of history. (If the old label has vanished meanwhile, the new one and the audit entry still land, and the failure to mark it is logged rather than swallowed.)

Both halves are load-bearing. The correction has to win in search or the feature is pointless. The superseded guess has to survive because a record of what a model claimed, and what a person had to overrule, is the only material from which anyone could ask whether that model is any good. Nothing computes that; throwing the wrong answers away would foreclose it permanently, and that asymmetry — cheap to keep, impossible to reconstruct — is the argument.

The correction is written as a signed entry in your own history: an append-only record, new entries only and nothing edited in place, carrying who made the change and when. What we checked is that the entry is written and signed with your key; whether anything re-verifies those signatures on read is not established here. The idea behind that shape is write to the log, read from the projection.

Forgetting is a different verb

Correcting says that is wrong. Sometimes a label is right, and private, and you want it gone anyway. So forgetting is its own operation: the label is marked forgotten and disappears from search entirely — not ranked lower, not surfaced as a weaker match, skipped. The record persists, and a signed entry says who forgot what and when.

Be clear about the trade. This makes a label unfindable, not erased. For a label you wanted gone because it was private, "still on disk, marked forgotten" is not the same as gone, and anyone needing the stronger guarantee should read it as not yet provided. What is bought in exchange is that the system can never quietly lose track of what it once said about you — the difference between unfindable and unaccounted-for, which is the argument in forgetting is a feature.

Two things follow. Nothing described here removes anything: unconfirmed guesses, superseded ones and forgotten ones all accumulate, and each fresh examination adds more — whether anything ever prunes them is not established here. And what a forgotten label looks like on a second device of yours, or to someone you share the photo with, is outside what we checked.

There is also a fragility worth admitting, and it is not hypothetical. The exclusion of superseded and forgotten labels is written out three times: once where search reads labels, once where the photo's detail panel reads them, and once more in the list of terms the search box offers you as suggestions. Three copies, deliberately mirrored rather than shared. And the third copy took the superseded-and-forgotten rule and did not take the grooming-label rule — so a word suppressed everywhere you would look at a photo can still appear in the suggestions the search box shows you. That is exactly the failure a duplicated rule invites, and it has already happened here: copies agree only until someone writes a fourth, and the one who writes it copies the rule they noticed.

What the screen is allowed to say

It surfaces in a small receipt in the photo's details. For each derived label the receipt shows a coarse band — high at 0.85 and above, otherwise medium — and the name of the model that produced it. The decimal is not in the visible text; it stays attached to the row for anyone who wants to audit it.

That is a deliberate refusal to print 0.87 at a person, since a decimal invites arithmetic the number does not support. But read the band as a convenience rather than a measurement: 0.85 is one fixed line drawn across models whose numbers are not comparable, so a scene label admitted at 0.12 prints the same word as one at 0.84. Two coarse words are less wrong than a spurious decimal. They are not precise. Naming the model carries the real information — whose opinion you are reading — with one exception we will not paper over. Alongside a strong label the system stores a few synonyms (swimwear terms, indoor and outdoor), and those rows inherit the parent label's number and the parent's model name, so the model is credited with a word it never emitted.

Three kinds of label never reach the receipt: corrected-away ones, forgotten ones, and any label with no confidence number, which renders no row rather than a row with an invented value. Your own corrections are not in the receipt at all; they appear beside the labels themselves, as their own kind of marker tagged as coming from you. A surface that says what it could not determine instead of quietly rendering a blank is a habit we keep arriving at from different directions; the first time was the receipt that tells you what it couldn't find.

Why the recorded number is never rewritten

An earlier version of this pipeline did not gate on confidence. It clamped: anything above 0.60 was written down as 0.60. The reasoning is almost sympathetic — these numbers are not comparable across models, so do not let a big one shout; flatten everything and nothing can over-claim.

What it did was destroy the model's output and store a fabrication in its place. A model that said 0.93 was recorded as having said 0.60, nothing downstream could recover the difference, and no display could be honest about it.

The replacement separates the two jobs the clamp confused. Gate on the value: decide whether to store the label at all. Then persist it verbatim. A threshold is a decision about admission, not a licence to rewrite the evidence you admitted.

Two things about that change we would rather say than have found. Nothing retroactively repaired labels written earlier — they keep the flattened number they were given, and only a fresh examination replaces them. And the repair was made on the path that looks at images, not everywhere: the sibling path that derives entities from text, in the same file, still clamps, at 0.70. Naming that is the point. A principle applied to the code you were looking at is not the same as a principle applied to the system.

What is real, and what is not

The mechanics above are landed, read directly in the shipped code that sits inside the redesigned photos surface. Specifically: the bars and their defaults, the refusal to synthesize a missing confidence, the unconfirmed-by-default state, the ranking, the correction and forgetting operations with their signed entries, the coverage record, and the receipt.

What is not true is that you can go and use it. There is no control anywhere in the interface for correcting or forgetting a label. Both exist as operations the system accepts and performs correctly, reachable only by a program talking to it. There is also no way to clear every derived label on a photo at once, because no such operation exists — and a greyed-out button promising one was deliberately deleted rather than shipped as a promise.

One question we will not answer from guesswork: what a later re-examination does to a correction you already made. Whether the corrected-away word can return as a fresh unconfirmed guess depends on how the storage layer identifies a repeated label, and we have not verified it. What hangs on that is worth being blunt about — if re-examination can propose dog again, then correcting is durable for the record and not necessarily for what you see.

Which is the ordinary condition of a thing being built: the rules settle before the surface does, and the rules are the part that is hard to change later. A photo library where guesses are stored as guesses can grow a correction button in an afternoon. One where guesses were stored as facts cannot grow the distinction back at all — the information that would have made it possible was discarded on write.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.