Photo apps usually treat recognition as a one-shot job. The picture goes in, some labels come out, and that is the end of it. The more useful pattern is the one a person follows. You notice there is a bird in the picture, and then you look more closely at the bird.

On 3 and 4 September that second step got built. What a photo shows can now start a follow-on plan aimed specifically at that kind of thing.

Plans that wake up for a category

A flow plan in NAOMS is a saved pipeline: a list of steps the system runs over your data when something happens, such as a photo being imported. Until now, plans were started by events like "a photo arrived". They could not be started by what was in the photo.

The trigger reads the short prose description that a vision model on your own machine writes for each photo. The companion piece Show me the photo where… covers how that description is produced and why it never leaves the device. When the description mentions something a plan cares about, that plan runs on that photo.

flowchart TD
  A[Photo imported] --> B[Local vision model writes a description]
  B --> C{Does the description mention
a term an enabled plan lists?}
  C -- "bird, heron, sparrow…" --> D[Bird plan runs on this photo]
  C -- "dog, fox, deer…" --> E[Mammal plan runs on this photo]
  C -- nothing listed --> F[No follow-on work]
  D --> G{Species step:
identification model installed?}
  G -- no --> H[Abstains: nothing claimed]
  G -- yes --> I[Species, or just 'bird', or nothing]

The matching is deliberately plain, and each rule has a test that fails if it is loosened:

• Whole words only. A birdcage is not a bird. Substring matching would send every photo of a birdcage to the bird plan.
• More than one category can fire. A bird eating an insect wakes both plans. The first match does not win.
• A switched-off plan stops answering, and a plan that lists no terms can never fire.
• Each photo triggers each plan at most once. A category plan cannot be started by the ordinary "new item" triggers either, so one plan cannot set off another in an endless loop.

Categories are data

The request that started this ended in "etc.": an animal plan, an insect plan, and so on. So the list of categories could not be written into the code. It is not. The trigger's code holds no list of categories. A category exists because some enabled plan says it does. That plan carries two things: a tag naming its category, and the words it answers to.

Adding "insect" means adding a plan, not changing code. The words live on the plan that uses them, which keeps them in one place: an operator can read them, edit them at runtime and switch them off with the plan's own on/off flag. A separate table of terms somewhere else in the code would drift out of step with the plans it was meant to describe.

A day later the owner pushed the idea one step further: "the design must be that ANY package can add flow plans." A package, NAOMS's unit of installable functionality, can now ship its own plans. They are picked up when the app starts. Species identification became its own package, carrying its own category plans, instead of living inside the photos code.

Which categories, and why only two

The owner named four categories: "mammal, bird, insect, plant as long as license permit distribution by us." The last clause did the work. A survey of available identification models found candidates whose model files were clearly licensed for redistribution for mammals and for birds. For insects and plants, the licences on the model files themselves could not be confirmed.

So two category plans exist, mammal and bird. Insect and plant were deliberately left out rather than added as plans that would never do anything. Shipping a trigger for a category reads as "we picked a model", and that choice had not been made.

The species step inside those plans bundles no model at all. It uses whatever identification model is installed for that category. If none is, it abstains and logs why. Unless an identification model has been installed, that is what happens: the plan runs, and nothing is claimed.

One model, many label lists

The identification design avoids a separate large model per category. It uses a family of models trained to place pictures and short text descriptions in the same space, so "how much does this image look like a photo of a house sparrow?" becomes a measurable closeness. The image model is loaded once and kept; switching category swaps only a small precomputed list of descriptions for its species. The lists committed so far are demonstration sized, eight species each, and exist for birds and insects; there is no mammal list yet. The model used in the real-data test below is a research model, not one of the licence-cleared candidates from the survey.

Every label also carries its full family tree, from kingdom down to species. That makes the most useful behaviour fall out naturally. The chances roll up the tree from a single look at the image. A blurry bird can be confidently a bird while being no particular species. When the model cannot be sure which bird, the answer is "bird", and no species is claimed. Each level that fails to clear its bar is left unanswered and counted, rather than being rounded up into a guess. The result is stored as a machine's claim with its confidence attached, not as something you said; the guess-versus-fact piece covers that rule.

The red square

The first version of this passed its synthetic tests. Then it was run against real image data from a real model, on five real pictures, and it produced this:

• The insect list called an office building a seven-spot ladybird, at 93% confidence.
• It called a solid red square the same ladybird, at the same 93%.
• The bird list called a car park a bird, at 92.5%.

Two unrelated inputs getting the identical answer at the identical confidence is the tell. The model was not telling anything apart there.

The synthetic tests could not see this because their numbers were tidier than reality. In them, a picture of nothing scored very differently from a picture of an animal. On real data, the closeness scores for every picture bunched into a narrow band whatever the picture showed. The safety floor the tests had checked never came into play. The score in the middle of the process always adds up to 100% across the labels, so a picture of nothing still gets some winner.

Two fixes came out of it. The second is the one that teaches something:

1. Compare against "nothing", not against zero. Each list now includes background descriptions, such as a parked car, a building, a person or a plate of food. The measure that counts is how much closer the picture is to an organism than to background. That gap tracks whether an animal is actually present, where neither raw score does. The gap threshold is required; a list without one is refused rather than given a default.
2. A specialist cannot tell you when it is out of its depth. The model used here was trained heavily on species names, and it rated the office facade as more like a ladybird than like "a photo of a building". No threshold on its own output separated the office from a real bee with any margin worth trusting. So whether there is an organism at all has to be decided by the general-purpose vision step that comes before it. The species step can only refine within a category it has been handed. The requirement is written into the step's inputs, so a caller cannot forget to supply it.

That second point is why the design above has the shape it does. The general description decides there is a bird here and starts the bird plan. The specialist only answers which bird.

Two honest limits were written down rather than hidden. The office-building photo still fools the insect list on its own. A test asserts exactly that, with instructions to flip it only on evidence from a proper held-out set. And the gap threshold was set from five images. The label files call that value a placeholder, not a calibration: five images is an anecdote, and it has to be redone for each model and category before shipping.

Status

The trigger, the category plans and the species step were written and tested on 3 and 4 September. The species package and its two plans reached the main line of the codebase on 18 September. The description-based trigger arrived on 23 September. A device picks up the two plans when its default plans are seeded, at onboarding or in a one-time recovery step at startup. Without a local vision model there is no description, so nothing triggers. No identification model ships with it, so even when a plan runs, by default the species step abstains.

Written by AI agents from real project logs; owned and edited by Mujo.