Figuring out who's who across thousands of photos

Face matching for one family is really two jobs sharing one toolkit. The first is the backfill: it runs once against a family’s entire photo history and sorts every face into the right child. That’s the story below. The second job — the live matcher that places each new upload on the spot — comes in Part 3.

You don’t need the mechanics yet — that’s the rest of this page. For now, just run it: one family’s library goes through the whole backfill below, and the pinned panel plays the algorithm while you scroll the story past it. Each idea it leans on — age buckets, the birthday age model, tracklets, the matcher, the cannot-link guardrails — gets its own interactive demo in Part 2.

Face recognition compares embeddings — a list of numbers per face — using cosine distance. Small distance = probably the same person. That works great for adults. It falls apart for kids in two specific ways:

Every face the vision model looks at gets tagged with one of eight age buckets. That tag is an LLM vision call step over in ingestion, not something the face embeddings give us. These are developmental stages, not even time spans — “neonate” is a month, “adult” is decades. Each card below shows the exact day range the algorithm uses.

Notice it: drag across all of childhood — newborn to teenager, every dramatic change a face goes through — and the true-scale needle barely leaves the left edge. One nudge into “adult” and it leaps across the whole bar. These are developmental stages, not even time spans, and they’re deliberately wide to absorb the model’s mistakes. The algorithm never trusts a single bucket as gospel — it reads the bucket relative to a birthday, with slack. That’s next.

Here is the primitive everything else is built on. Given a child’s known birthday and the date a photo was taken, we can compute the bucket we’d expect that child to be in. Then we ask: is the bucket the model actually observed close enough to that expectation? “Close enough” means within a tolerance of ±1 bucket — which forgives the model’s most common mistake (calling a toddler a preschooler) without accepting wild errors.

The problem section showed that embedding distance fails across a childhood. But it’s still trustworthy over a short window — a face barely drifts in a month. So before any birthday reasoning, we group faces where the embedding can be believed: build a k-nearest-neighbour similarity graph, then keep only the edges that are tight on both axes at once —

• close in looks — cosine distance less than a certain threshold, and
• close in time — the two photos taken within an amount of time with each other.

The connected components of that graph are the tracklets. The 30-day bound is per edge, not per tracklet: a chain of short hops (A↔B, B↔C…) lets one tracklet span months while every individual link stays inside a window where looks are reliable. A component needs a minimum number of faces to anchor a child — a lone face waits for cross-time linking later.

Node colour is each face’s true person; the dashed boxes are the tracklets the graph actually forms. A box that wraps two colours is a contaminated seed — exactly what the tight default cutoff is tuned to avoid. (Edge distances here are illustrative; the two cutoffs are the real constants, and k=10 never binds at this scale.)

We have time-local groups now — but a tracklet is still anonymous. This is where the birthday decides whose it is: scale the age model up from one face to a whole group. Two rules:

• pinned_child — for a single face, which enrolled child does its age uniquely point to? If the age is consistent with exactly one child, it “pins” that child. If zero children fit, or two fit (siblings too close in age that day), it pins nobody — we’d rather stay silent than guess.
• dominant_child — for a whole group, every face votes for the child it pins, and the plurality wins. Using a vote (not unanimity) means one mislabeled face can’t flip the answer. A tie for first place returns nobody.

This single resolver is the shared brain of the algorithm: the seeding step uses it to label groups, and the cannot-link step uses it to keep siblings apart — so the two can never disagree about whose face is whose. Note one asymmetry that Part 3 builds on: the fast live matcher skips this birthday seeding entirely — the one-time backfill owns grouping quality, and live just places new faces against the identities it built. But live isn’t birthday-blind: it borrows the same birthday age model as a gate — before routing a face to the baby, it checks the face’s age bucket is consistent with the baby’s birthday.

The tracklet and linking steps are eager to merge anything that looks alike. The cannot-link constraints are the veto. Before any two groups merge, the algorithm asks three independent questions — if any says “these can’t be the same person”, the merge is blocked. Every rule fails open: if it can’t tell, it stays quiet rather than wrongly blocking.

Here are all three, in order. Each shows the case it’s built for — switch scenarios within a rule to watch it fire or stay silent, and read every rule’s verdict plus the composite constraint they OR into.

The backfill won the grouping quality once. The live matcher keeps the library sorted forever — every new upload placed on the spot, in about a second, against the identities the backfill built. It doesn’t re-cluster anything. Scroll on: the panel picks up exactly where the backfill left off, then a Tuesday-afternoon upload arrives.

Live matching is deliberately simple — and that simplicity is the whole point of the results later. A new face greedy-matches its closest existing identity, and joins it only if it clears two gates at once: close enough in looks (under an age-aware cosine cap) and an age the birthday will vouch for. Either gate can say no — and when looks say yes but the birthday says no, the birthday wins. That single rule is what keeps an adult face out of the baby.

That’s the whole live matcher in one decision. No tracklets, no birthday voting, no re-clustering — just nearest-identity, an age-aware cap, and the birthday gate. The backfill earned the hard-won grouping; live only has to preserve it. The next section shows why keeping live this simple turned out to be the right call.

Let’s be honest up front: face matching for kids will never be 100%. Siblings really do look alike, a newborn really doesn’t look like the toddler they become, and some photos are just too dark or too sideways to place anyone. Perfection isn’t the goal — the goal is to keep dragging the error down, iteration after iteration, and to know exactly how much each change moved the needle.

Of all the numbers our test harness reports, the most tangible is the headline below: how many faces ended up filed under the wrong person? A face is miscategorized when it lands in an identity that mostly belongs to someone else — a sibling, the grandparent — or gets stranded away from where the child’s photos mostly live. We track it on both golden families: the easy single child (581 faces) and the hard two-sibling case with Mia, Leo and the grandparent (1,193 faces).

Walk it end to end and the shape of the story shows up: v1 → v5 is 127 fewer faces in the wrong pile on the hard family — about a third of the original errors gone — but the road there was anything but a straight line. Here’s every iteration laid out as a table, on both families:

Two things jump out. On the single-child family, just moving from per-photo matching to batch k-NN tracklets (v2) roughly halved the errors, and they’ve stayed low ever since. On the two-sibling family, v2 actually made things much worse — chasing recall, it happily fused the siblings into one blob — and it took the cannot-link constraints (v3–v4) to claw that back and the birthday anchor (v5) to finally beat the original baseline by a wide margin. Real progress, earned the hard way: the same change that’s a clear win on an easy family can be a regression on a hard one, which is exactly why we score every version on both.

And then we made the headline number worse — on purpose. Everything above chased one count down: fewer faces in the wrong pile. The most recent change breaks that streak deliberately. After clustering, the shipped backfill runs the correction pass from Part 1 — the age bucket reconciler — which ejects any face whose age bucket flatly contradicts the known age of the child it was filed under. Measured on the same two families (isolating the pass alone), it moves two numbers in opposite directions:

This round of work reached beyond the algorithm, too. The birthday anchor is only as good as the birthday behind it — so as part of v5 we updated onboarding to ask for each sibling’s birthday up front. That turns the anchor from something we could only hand-label on golden families into real data every family provides.

Step back and the whole algorithm is one idea applied over and over: the face alone is never enough, so anchor it to the birthday. Every face gets a developmental-stage bucket, and the birthday age model reads that bucket against a known birthday with ±1 slack. Faces that sit close in both time and appearance get grouped into tracklets — connected components of a k-nearest-neighbour similarity graph — and each tracklet is tagged with the child its age fits, seeding high-confidence anchors. From there, linking grows each identity across the years while a cannot-link rule, gated by that same birthday anchor, keeps siblings and strangers out. One final backfill pass then ejects any face whose age flatly contradicts the child it was filed under — trading a sliver of aggregate neatness for cleaner named identities, on purpose. And once the backfill has built those identities, live matching keeps it running forever: every new upload is placed on the spot by closest-cosine against the existing identities, with the age model deciding which faces are even allowed to be the baby. Face matching for kids will never be 100% — but every one of these layers is a cheap, independent axis that makes it harder to get wrong. What the named identities are for is the next article: building meaningful moments out of a family’s photos →