Technical Barriers in Smartphone Skin Analysis, and How We Address Them

In the last post I wrote that measuring often is what reveals the trend. But that comes with a condition attached: you have to shoot under similar conditions each time. If today's photo is straight-on by a window and tomorrow's is at an angle in a dim room, you can't tell whether the difference is your skin or your setup.

This is, in fact, the question we field most often in outside meetings: "Users will shoot however they like; can you trust the result?" It's a fair question, and it's where we spend most of our time.

Angle, distance, and lighting change the result

Let people shoot freely and the photos come out all over the place.

• Angle: tilt your chin up or down and nasolabial folds and sagging read differently
• Distance: too close and the face distorts, too far and the detail smears
• Lighting: pigmentation and redness look nothing alike under window light versus fluorescent

The problem is that these differences can be larger than the actual change in your skin. When that happens, the line those points draw isn't a trend in your skin — it's a trend in your shooting conditions.

And every phone is different

There's another layer. Everyone uses a different phone.

Sensor resolution and lens characteristics vary by model, and each manufacturer processes color its own way. Photograph the same face from the same spot and the output still differs slightly from device to device. This is a fundamentally different premise from a clinic, where you control a single machine.

That's the price of choosing the smartphone. You gain the fact that anyone can measure with what's already in their hand — and in exchange, you take on all that variety.

So we're looking for the sweet spot

The simplest fix is to control capture strictly: fixed distance, fixed angle, fixed lighting. Accuracy goes up and nobody shoots daily anymore. Frequency is the smartphone's greatest advantage, and that approach throws it away.

So we went the other direction. We observe how people actually shoot, and use that data to find where two things meet: the least friction we can ask of someone, and the accuracy we need to keep.

It isn't a problem with one fixed answer. It's closer to continuous adjustment, guided by what the data shows. The method we arrived at along the way has been filed as a patent.

You can't reduce variance you haven't measured

Differences across devices and environments are handled through training — but not by simply piling on more data.

We run the model over images captured under a wide range of conditions, then analyze how much the outputs scatter as those conditions change. How far does a score move when the same skin is shot on a different handset? Which axes are especially sensitive when the lighting shifts? We look at these separately, because among the six axes, pigmentation and redness react more to lighting while wrinkles and sagging react more to angle — so each calls for a different response.

Once we've established the size of the variance and where it comes from, we reinforce the data for the weak conditions, retrain, and measure again the same way. You can't reduce variance you can't see.

Building a model tuned to one handset under one lighting condition is relatively easy; the conditions are fixed. The hard part is getting similar values no matter which phone you use or where you shoot. And for a service where everyone measures with their own phone, the hard part is the only option.

In an earlier post I wrote that analyzing skin through a phone's front camera isn't technically easy. What's in this post is the substance of that difficulty. We chose this direction anyway, for one reason: we think the owner of your skin data should be you, not a clinic's server.