Ask your photo library for "more like this one," and something has to decide what this one is like.

The lazy answer is metadata: same day, same folder, same camera, the filename has "sunset" in it. That finds photos taken near the one you picked. It does not find photos that look like it. For that, the system has to have some representation of what is actually in the picture — and, if it's going to respect you, it has to build that representation on your own machine, without mailing your photos to anyone.

That is what shipped. Every photo you import is read, on-device, by a real vision transformer, and turned into a fingerprint: a list of 1024 numbers. "Find photos like this" is then a genuine nearest-neighbor search over those fingerprints. No cloud, no upload, no filename tricks.

What the fingerprint is

The model is a CLIP-style vision transformer — the vision half of the architecture that taught machines to relate images and language. You hand it a picture; it hands back a point in a 1024-dimensional space. Two photos that a human would call similar — the same kind of scene, the same subject, the same mood — land close together in that space. Two unrelated photos land far apart. "Close" and "far" are literal here: the system measures the cosine of the angle between two fingerprints. Small angle, similar picture.

Getting a faithful fingerprint is fussier than it sounds, and the fussiness is where correctness lives. Before the model sees a photo, the image is resized so its short side is 224 pixels, center-cropped to a square, and normalized against the exact per-channel means and standard deviations the model was trained on. Those steps are byte-matched to the reference pipeline the model's authors published. Get the crop or the normalization subtly wrong and the model still returns 1024 confident numbers — they're just the wrong numbers, quietly. A fingerprint that's plausible but off is worse than an error, because nothing tells you it happened. So the preprocessing is treated as part of the model, not a detail around it.

The model runs locally through a Rust binding to a portable inference engine, and on Apple hardware it offloads to the Neural Engine — the dedicated silicon built for exactly this. Wiring that path had a real, unglamorous cost: for a while the desktop build didn't link the framework that exposes the Neural Engine, so the accelerator was present but unreachable. That link was fixed and landed this week. It's the difference between the model idling on the CPU and the model running where the hardware wants it.

Where the fingerprints live, and how "like this" gets answered

Storing a fingerprint is easy. Searching a whole library for the nearest ones, fast, is the part that needs a real substrate. The vectors go into an on-device vector index — a virtual table that understands 1024-number vectors natively and answers "give me the k closest to this one, by cosine distance" directly. That nearest-neighbor query is the mechanical heart of "find photos like this." You give it a fingerprint; it walks the index and returns the closest neighbors, ranked by how small the angle is.

We proved the lane does something keyword search cannot. In one test, a query photo is matched against a target that shares zero words with it — no common filename, caption, or tag — and the vision lane still surfaces the target, because the two pictures look related even though nothing textual connects them. Keyword search has no path to that answer. The vector lane finds it because it's reasoning about pixels, not strings.

But photos aren't only pixels. There are words inside many of them — a street sign, a menu, a screenshot — and there are things you can name: a place, a detected concept, a face. So the search that ships doesn't run on the vision lane alone. It fuses lanes: the vision-vector lane ("looks like this") plus text lanes ("contains these words or things"). Each lane ranks the library independently, and the rankings are blended with Reciprocal Rank Fusion — a method that combines ranked lists by how high up each item appears in each list, rather than trying to compare a cosine distance against a text score on some invented common scale. A photo that's a strong visual match and also contains the right text rises to the top of the fused list. One that's only a visual match still ranks — just lower.

One honesty gate is worth naming. The model isn't chosen blindly. At selection time, the code refuses a model whose license forbids commercial use, or whose output dimension doesn't match the index. A wrong-dimension model would silently poison every fingerprint; a non-commercial model would be a license landmine waiting under the whole product. Both are rejected up front rather than discovered later.

flowchart TD
  A[You import a photo] --> B[Preprocess: resize 224 short side, center-crop, CLIP normalize]
  B --> C[On-device vision transformer, Neural Engine]
  C --> D[1024-number fingerprint, a point in CLIP space]
  D --> E[(On-device vector index, cosine distance)]
  F[You pick a photo: find more like this] --> G[Its fingerprint]
  G --> H[Nearest-neighbor search: closest fingerprints by angle]
  E --> H
  H --> I[Vision lane ranking: looks like this]
  J[Text lanes: words inside the image, concepts, place, faces] --> K[Text ranking]
  I --> L[Reciprocal Rank Fusion]
  K --> L
  L --> M[Blended results: looks-like fused with contains]

The honest limit — read this part twice

Here is the boundary, stated plainly, because overclaiming it would be a lie: this is image-to-image only.

The vision transformer we shipped is the vision half of CLIP. The text half — the encoder that turns the sentence "a dog on a beach" into a point in the same space — is not wired. Which means: show the system a photo and it will find visually similar photos. That works, and it's the capability worth having for dedup, for "more of this moment," for finding the near-duplicate you meant to delete.

But you cannot yet type "beach" and have the vision model go find beach photos. When you type words, they reach your photos only through the text lanes — the letters read out of the image, the detected concepts, the place, the caption. The pure-vision lane, asked a text question it has no text encoder to answer, returns nothing rather than pretending. It doesn't fabricate a match to look capable. Text-to-image — type a description, and the vision model itself finds the pictures that match it — is the natural next step, and it is honestly not shipped.

Four more limits, none of them buried:

• The model is dormant until you install it. The bytes are wired, but the model file arrives through the owner's signing ceremony. Until it's present, the system logs an honest warning and degrades to text search — and it tags each result as served by the vector lane or by the text lane, so you're never silently downgraded without knowing.
• The photos "full-text" lane is modest by design. On the photos surface it's an in-memory scan over the captions and the text pulled out of images — fine for a personal library, not a heavyweight search engine. We won't call it more than it is.
• Desktop and mobile fingerprints don't mix. The desktop model and the smaller image-only model used on phones both emit 1024 numbers, but into different, unrelated spaces. A fingerprint from one is meaningless to the other, so cross-device nearest-neighbor isn't safe until the index learns to keep each model's vectors in their own partition.
• Faces group only when you ask. There's no automatic face detection sweeping your library on import; grouping happens on an explicit action.

Why on-device is the whole point

Every claim above has a hidden clause: and none of it leaves your device. The model reads your photos where they already are. The fingerprints are stored locally. The search runs locally. There is no server that sees your pictures to index them, because the indexing is the model on your own Neural Engine.

That's not a performance footnote; it's the design. The usual way to get "search your photos by what's in them" is to upload the photos to someone who runs the model for you and keeps the fingerprints. You get the feature and they get your library. Doing the inference on-device is what lets the feature exist without that trade. It's the same commitment behind search that never phones home, and it's why the runtime matters as much as the model: a model you can only reach through someone else's cloud is a different product than the same model running on the metal in your pocket.

And the honesty is load-bearing, not decorative. The pure-vision lane returning nothing for a text query, the served-by-vector-versus-text tag, the refusal to mix two incompatible vector spaces — each is a place we could have faked capability and chose not to, in the same spirit as the rule that any test that passes over a gap is a lie. The system would rather tell you it found nothing than invent a match. Find a photo by what's actually in it — on your device, privately — and know exactly how it was found.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.