How to count calories from a photo

Here's how the technology actually works, five things that make a photo scan more accurately, and what to do when it still gets your meal wrong.

How photo-based calorie counting actually works

Under the hood, it's a vision model doing three jobs in sequence. First, it identifies what's on the plate — "grilled chicken," "white rice," "steamed broccoli" — the way it might caption any photo. Second, it estimates portion size, using the size and shape of the food relative to the plate, or any reference object in frame, as a rough gauge of volume. Third, it maps each identified item to a nutrition database and multiplies by the estimated portion to get calories, protein, carbs, and fat.

Modern models are genuinely good at step one — naming food correctly. Step two, portion size, is the weaker link, because a single 2D photo is trying to guess a 3D quantity. That's the part most likely to be off, and it's also the part you have the most control over as the person taking the photo.

5 ways to take a photo that scans more accurately

A blurry, angled shot of a crowded table isn't giving the model much to work with. A few habits make a real difference:

• Shoot from straight above, not an angle. A top-down shot shows the plate's full outline, which gives the model a size reference. An angled shot hides part of the food behind itself and makes the portion harder to judge.
• Fit the whole plate in frame — don't crop it. If the model can't see the edges of the plate, it has no sense of scale. A visible plate edge does a similar job to the "banana for scale" trick photographers use.
• Separate mixed items where you can. A stir-fry that's all sauced together is a harder guess than the same ingredients arranged so the model can distinguish protein from rice from vegetables. You don't need to plate it like a food photographer — just don't stack everything into one indistinguishable pile.
• Use good, even lighting. Dim or backlit photos make it harder to tell food apart from shadow. Natural light near a window, or just decent kitchen lighting, beats a dark restaurant booth.
• Take one photo per distinct dish, not a whole spread. If you're eating three different things, three photos of individual dishes usually beat one wide shot trying to capture all of them at once.

None of this is complicated — it's the same instinct as taking a photo you'd actually want to look at again, which happens to be the kind of photo a vision model reads best too.

A 2024 study in the Journal of the Academy of Nutrition and Dietetics tested three consumer photo-recognition apps against a known reference value and found accuracy dropped hard once a photo moved away from a clean, standard shot — one app's estimate jumped to 378 kcal for a dish with a 133 kcal reference once the plate was photographed at an angle or cluttered, while the best performer in the test stayed close to accurate (142 kcal) across conditions.¹ The photo itself isn't a minor detail — it's a meaningful chunk of the accuracy.

What a photo alone can't tell the AI

Even a perfect photo has a ceiling, and it's worth being honest about where it is. A camera can see the outside of your food; it can't see what's inside it or how it was made.

• Cooking method and added fat. Grilled chicken and pan-fried chicken in a tablespoon of butter can look nearly identical in a photo and differ by 150+ calories.
• Sauces and dressings mixed in. A salad's calorie count can double depending on how much dressing is under the greens, and a photo mostly shows the greens.
• What's hidden inside. A burrito, a dumpling, a casserole — the model can see the wrapper or the surface, not the filling.
• Mixed and non-Western dishes. These tend to be where estimates drift furthest, because a dish with everything combined (a curry, a noodle soup, a stir-fry) gives the model less visual separation to work with than a plate with distinct, isolated items.

A University of Sydney study published in Nutrients found this pattern clearly: AI photo estimation overestimated a beef pho by roughly 49%, and underestimated a bubble tea by up to 76% — both dishes where the calories are largely hidden in broth, sauce, or a drink rather than visible on the surface.² That's not a flaw specific to one app — it's a limit of estimating a hidden quantity from what a camera can actually see.

When the estimate is still off, fix it in one sentence

A good photo narrows the gap; it doesn't close it entirely. So the part that matters just as much as the photo is what happens after — when the number looks wrong.

I built Bocco's correction flow before I spent much time chasing a fancier vision model, because every study above says the first guess will sometimes be off, for every app, ours included. The fix isn't re-typing the whole meal from a database. You just say what's actually different — "this had two tablespoons of olive oil," "it was a larger bowl than that," "I used oat milk, not skim" — and the numbers recalculate properly instead of getting a rough relabel. A photo gets you 80–90% of the way there fast; the correction is what gets the rest.

That's the honest version of "count calories from a photo": a genuinely useful shortcut, with a known, specific ceiling, and a fast way to close the gap when you hit it — not a claim that a camera can replace a kitchen scale.