Pulling the rich data out of every photo
A family trusts us with their whole photo library, and every photo arrives as raw bytes. But a photo carries far more than its bytes: locked inside it is a time and place, the day’s weather, the faces in the frame, a quality score, and a read on which milestone it might show. Ingestion’s whole job is to pull every one of those signals out into a structured field. This page walks that first half of the pipeline — how we extract everything a single photo carries, so the curators can later recombine it into moments.
Everything one photo carries
Before any moment can exist, the pipeline has to pull everything out of each photo on its own: read its metadata, group it with its burst, screen it for safety, and fan out a small army of jobs that score it, place it, read its weather, and find the faces. Nothing downstream works until this does.
From raw bytes to a record
A photo arrives at the API as raw bytes plus a few headers — a client asset id, the filename, a device timestamp, maybe GPS. The bytes are the least interesting thing about it: everything we actually care about is latent — when and where it was taken, which other shots it belongs with, whether it’s even safe to keep. The first job is to pull that out and write it down synchronously, before the photo even counts as saved, in this order:
- EXIF extraction. Pull the photo's taken at timestamp (preferring the camera’s
DateTimeOriginal, falling back through digitized/modified dates to the device’sX-Taken-Atheader), GPS latitude/longitude, and pixel dimensions — interpreting the timestamp in the family’s timezone. - Burst assignment.Groups shots taken within a 30-minute window into a burst — the run of photos someone fires off at a single event — merging overlapping windows atomically.
- Fan out the analysis jobs. The record is saved, and the per-photo analysis pipeline is enqueued. That’s the next section.
A backstop keeps this honest: the same upload retried (same asset_id in one family) hits a unique constraint and comes back as the existing photo, so a flaky connection can’t spawn duplicate records.
The analysis fan-out
A saved photo isn’t ready yet — it’s a blob with a timestamp. Pulling out everything it carries means a fan-out of analysis jobs: five root jobs kick off the moment the photo is saved, and one more — age bucketing — waits its turn behind face recognition. Each records its result, and the photo only becomes analyzed once a completion gate sees every step terminal and dispatches the moment curators.
Look at what these readings actually are, though, and a pattern jumps out: most of them are the same move — hand the photo to a vision-capable LLM and ask it one precise question. Three of the six — how good is this photo, does it show a milestone, and how old is each face — are LLM vision calls. That’s the part that simply wasn’t practical a few years ago: instead of training a bespoke model per question, we describe the judgment we want in a prompt and get back a calibrated 0–1 score. Finding the faces is the one reading that doesn’t lean on the LLM — it runs through a dedicated face-recognition service — and location and weather are ordinary API lookups. But a photo’s meaning — its quality, its milestone relevance, the ages in it — is read by an LLM. That capability is a big part of why this pipeline was buildable at all.
Scroll through the graph with one example photo — a beach shot of the baby — and watch exactly what each step pulls out of it.
beach_day.jpg — raw bytes, nothing pulled out yet.
Field values are an illustrative example; the weights, age buckets and thresholds are the real ones.
Step 0 · The fan-out
One photo, six readings
The instant the record is saved, five jobs fire in parallel — scoring, location, weather, moment scoring, and face recognition — and a sixth, age bucketing, waits behind the faces. Here’s the graph. Scroll, and we’ll light up each step and show exactly what it writes down about this one photo.
Step 1 · Photo scoring
How good a photo is it?
A single LLM vision call rates four dimensions 0–1 and folds them into one shareability score. Technical quality carries the most weight (0.37) and has a hard floor: dip below 0.5 and it caps the whole composite. This beach frame is sharp and bright, so it scores well.
Step 2 · Location
Where was it taken?
The photo carried GPS, so we reverse-geocode it into a real location. (No GPS? The step skips, and the photo simply has no place — that’s fine.)
Step 3 · Weather
What was the day like?
We use the latitude and longitude coordinates plus when the photo was taken to fetch historical weather records and stors that day’s observation. A warm, clear afternoon at the coast.
Step 4 · Moment scoring
Does it show a milestone?
We make another LLM vision call, asking a different question than quality: does this photo include a significant milestone? It writes a relevance_score — the signal that later powers milestone moments. This one lights up “first time at the beach.”
Step 5 · Face recognition
Who's in the frame?
This step calls the dedicated face-recognition service — not the LLM — and detects each face in the photo. The same response also carries a perceptual hash of the whole image we use for deduplication between photos.
Step 6 · Age bucketing
Roughly how old is each face?
This one dependent step makes one more LLM vision call fpr each recognized face to tag it with an age_bucket from a fixed scale — neonate · infant · toddler · preschooler · child · adolescent · adult · elderly. It’s skippable; an unbucketed face just isn’t used downstream.
Step 7 · The gate
Every field filled
When all the steps are terminal and the photo is still eligible, the completion gate flips it to analyzed and dispatches the curators. The blob with a timestamp is now a dense, structured record — everything the photo carried, written down.
One step quietly does double duty. Face recognition’s response carries a perceptual hash of the image. It earns its keep later: when the curators collapse near-identical shots using clustering with Hamming distances.
The face chain has one more link in production — identity matching — and the gate waits on it too. But it isn’t extraction from this photo: it’s a family-wide pass that asks whose face each one is by clustering against everyone else, and for an imported library it doesn’t even run per-photo — it’s deferred and run as a single batch once the whole backfill finishes. That’s a story of its own: matching faces to the right child →
From a raw upload to every signal extracted
Ingestion is three moves. First it records what the raw upload already carries — pulling EXIF time and place, assigning the photo to a 30-minute burst, and opening an async safety screen. Then it fans out: six analysis steps (five in parallel, one dependent) score the photo, place it, read its weather, find its faces, and bucket their ages. Finally it gates — a photo counts as fully extracted only once every step has reached a terminal state, and clearing that gate is what dispatches the work downstream. From there the now-legible photo feeds two systems in parallel: matching each face to the right child → and building meaningful moments out of the family’s photos →.