The problem: shape from a single image

A photograph is a flat projection of a 3D world. The moment the camera captured it, one whole dimension — how far each surface was from the lens — was collapsed away. Every point along a given ray from the lens lands on the same pixel, so the absolute distance to that point is simply not recorded. Recovering 3D shape from a single image therefore means recovering that lost dimension, the depth of every pixel.

This is a famously ill-posed problem: because perspective projection discards scale, infinitely many different 3D scenes can produce the exact same picture. A small object up close and a large one far away can be pixel-for-pixel identical. No purely geometric procedure can undo that — the information is gone.

Yet humans judge depth from a single eye-closed glance effortlessly, using monocular cues: perspective convergence, the relative size of familiar objects, shading and shadow, texture that grows finer with distance, and the way nearer things occlude farther ones. None of these give exact distances, but together they give a confident relative sense of shape. The central idea behind modern single-image 3D is that these cues can be learned from data.

Idea 1 — Learning depth from one image (a pretrained model)

We don't hand-write any of those cues. Instead the heavy lifting is done by a pretrained foundation model — in our case Depth Anything V2, a vision transformer trained by its authors on millions of images so it generalises to scenes it has never seen. We use it as-is; the science we apply is everything that happens to its output. Shown a new photo, it returns a depth map: a single-channel image where each pixel encodes how near or far that point is.

A crucial subtlety shapes everything downstream. Because one image fundamentally cannot fix absolute distance, these models are trained to be affine-invariant — their output is correct only up to an unknown scale and an unknown shift. In plain terms: they nail the relative structure (this is nearer than that; this region curves away; these proportions), but the overall size and the zero-point of distance are undetermined. Concretely, Depth Anything predicts inverse depth (disparity-like, so nearer surfaces get larger values) normalised between the scene's nearest and farthest points — a representation that also behaves better for very distant regions than raw distance does.

The core trade-off. One image gives you rich, plausible relative geometry for free — no calibration, no second camera, no depth sensor. What it cannot give you is true scale or the parts of the object the camera never saw. Every limitation later in this article traces directly back to those two facts.

Idea 2 — Turning a depth map into 3D points

A depth map is still a 2D image; the 3D shape is implicit. To make it explicit we reverse the camera's projection — a step called back-projection. A camera maps a 3D point onto the image by perspective: points twice as far away appear half as large, and everything converges toward the optical centre. Knowing a pixel's position and its depth, we can run that mapping backwards to place the point in space.

This needs the camera's intrinsics — essentially its field of view, which sets how strongly perspective spreads things out. A webcam doesn't report this, so we assume a typical value (a field of view around 60°). Each pixel is then sent out from the optical centre along its line of sight, to the distance the depth map specifies: 

focal length  f = (image width / 2) / tan(field-of-view / 2)

X = (pixel_x - centre_x) · depth / f
Y = (pixel_y - centre_y) · depth / f
Z =  depth

Do this for every pixel and the flat depth map blossoms into a point cloud — a constellation of coloured 3D points (each point simply keeps the colour of the pixel it came from), tracing the surface that faced the camera.

From relative values to a believable surface

Because the predicted depth is relative and disparity-like, how you convert it back to distance matters. Mapping the values to distance linearly warps the geometry — flat walls come out gently bowed, like a fish-eye. The physically correct move is to interpolate in inverse-depth (disparity) space, mirroring how real cameras encode depth. This single choice is the difference between a subtly inflated blob and a relief that reads as the right shape.

Idea 3 — Cleaning the cloud

A raw back-projected cloud has two characteristic artefacts, and both have simple, principled fixes.

Separating subject from background. The depth map covers the whole scene, but we only want the figure. Since the subject is nearer than its surroundings, depth itself is the separator: discard everything beyond a chosen distance. It isn't true object segmentation — a wall right behind you won't split off — but for a subject standing clear of the background it isolates the figure cheaply and predictably.

Removing "flying pixels." At the silhouette of an object, depth jumps abruptly from foreground to background. Pixels straddling that cliff get back-projected into long, stretched triangles — phantom "curtains" connecting the object to whatever is behind it. The fix is an edge-aware test: if a point's depth differs sharply from its neighbours, it sits on a discontinuity and is dropped. This is a classic technique in depth-image meshing, and it's what gives the figure clean edges instead of webbing.

One more detail concerns colour fidelity: screen colours are stored in a perceptual (sRGB) encoding, while light in a 3D renderer is computed in linear units. Colours are converted from one space to the other so the figure looks the way the camera saw it, rather than washed out or too dark.

Idea 4 — From points to a solid surface

A point cloud is just dots in space — it has no surface, no "skin." To make a watertight object we use Poisson surface reconstruction (Kazhdan, Bolitho & Hoppe, 2006), a well-established method that needs each point to carry an estimated outward-facing direction, its normal.

The idea is elegant. Imagine an indicator function that is 1 everywhere inside the object and 0 outside; the surface is exactly the boundary where it jumps. That function is flat almost everywhere, so its gradient is zero except right at the surface, where it points along the normal. So the oriented points can be read as samples of that gradient field — and recovering the function from its gradient is precisely a Poisson equation, the same class of equation that governs heat flow and electrostatics. Solving it globally, over all points at once, yields one coherent, hole-free surface that is naturally robust to noise and uneven sampling.

Idea 5 — Painting the surface

Reconstruction gives geometry but not colour: the new surface has fresh vertices that don't line up one-to-one with the original coloured points. So colour is transferred by proximity — each point on the new surface takes the colour of the nearest original captured point. Done naively this is a slow all-pairs search, so in practice space is divided into a grid that lets each lookup check only nearby points. The captured colour ends up faithfully wrapped onto the solid figure.

The limits — and they are fundamental

These aren't rough edges to be polished away; they follow directly from using a single image.

  • Relative, not metric. Without a second viewpoint or a known reference, absolute scale is unknowable. The shape is proportionally plausible but not measurable in real-world units.
  • Only what the camera saw (2.5D). One image captures one side. The back and any occluded surfaces simply weren't observed; a closed solid has to invent them. The result is a relief or figure of the visible surface, not a true 360° model.
  • Guessed camera, guessed perspective. Assuming the field of view instead of measuring it means wide-angle or cropped lenses bend the proportions in predictable ways.
  • Depth-based separation isn't segmentation. A background at a similar distance to the subject can't be cleanly removed by a distance threshold alone.
  • Why many views don't trivially help. It's tempting to think you could just spin around and merge several single-image depth maps into a full model. The catch is that monocular depth is estimated independently and at relative scale for each frame, so the same surface comes back a slightly different shape and size from every angle. There is no common scale to align them to, so the merge has nothing consistent to converge on. Getting a true 360° model from a plain camera requires a fundamentally different principle — triangulating geometry from the camera's motion across frames (structure-from-motion / multi-view stereo), where the shared scene itself enforces consistency. A better single-image depth network does not solve this.
1 image
No depth sensor, no calibration
Relative
Proportional, not metric, scale
2.5D
The visible surface only
Learned cues
Perspective, shading, occlusion

So — can you scan with just a webcam, in a browser?

Back to the question we started with. The honest answer has two halves:

Yes: you can produce a genuine, coloured 3D surface of whatever faces the camera, from a single frame, with no special hardware and no calibration — and, because a modern pretrained depth model runs right inside the browser, with nothing ever uploaded. As a way to turn a moment into a tangible 3D figure you can spin, mesh, and export, it works, and it's the right tool for that job.

But not a metrology-grade 360° scan. What you get is a 2.5D relief at relative scale — the visible side, proportionally believable but not measurable in real units, with the unseen back inferred rather than observed. That isn't a flaw in our method or the model; it's the ceiling of what a single image can contain. A true all-around, true-to-scale model needs a different kind of evidence — multiple genuinely-related views that let geometry be triangulated from the camera's motion (structure-from-motion / multi-view stereo), where the shared scene enforces a single consistent shape. That's a heavier system for a different goal.

The bigger picture

Single-image 3D is a beautiful example of trading a hard physical measurement for a learned prior. We can't measure the missing dimension, so we let a pretrained network predict it from the same cues the human visual system evolved to use — then borrow centuries-old camera geometry and surface mathematics to turn that prediction into a tangible, coloured object. One photograph, it turns out, is already enough to conjure a surprisingly convincing 3D figure — as long as you're honest about what that figure is.

Sources & further reading