From pixel to real world

How do you where in the world a pixel maps to

Let's start with an example

We are viewing a parking lot, which has 8 spots, 2.5 meters wide, and 5 meters deep. We want to detect when a car arrives, and which spot it has taken. We are viewing through a camera, which is 6 meters up a lamp post, ca. 10 meters behind the parking row.

We detect a car parked at location , but how do we figure out which spot it has parked in? describes where in the image the car is detected, but NOT where in the real world it resides.

Issue is, that our camera flattens our world to this image. Meaning our depth is completely lost. If you were to take an image of someone standing 5 & 50 meters away from the camera, in the same direction, the pixel could be the exact same location on the image, but the actual xz, or real world location would be completely different. So some prerequisites to create this real world mapping, is understanding how the camera lens works, where the camera is, and the fact that the car is on the ground.

Prerequisites:

Two ways to describe the same spot

So we need to agree on our different coordinate systems.

From the camera's point of view, the car might be a few meters ahead, and to the right or left. Whereas in the parking lot's coordinate system, it might simply be described as "in spot 7".

For the parking lot, we pick our coordinate system, called the "world":

Since points up, is the ground. And any position on the ground can be described as . To move between coordinate systems, we use matrices, and label them according to what they do. camera2world takes a point from the camera's point of view, and gives you the respective coordinate in world perspective.

Going the other way is just the inverse: .

If you name every matrix like this, mistakes become easy to spot: the name on the left of each 2 has to match the name on the right of the next one. For example, .

From pixel to direction

Our detector gives us the car's position in the image as a pixel. Often we divide it by the image size, so it lands between 0 and 1. That's called uv, and it means we don't have to care about the resolution: is the top-left corner and is the bottom-right.

To turn a pixel into something useful, we need to know how the camera's lens and sensor work. That's described by the intrinsic matrix :

Multiplying our pixel by the inverse of turns it into a direction from the camera:

Say our camera films in 4K (), with , and the optical center in the middle at . The car is detected at pixel :

So the car is in the direction : for every 1 m forward, it's 0.4 m to the right and 0.2 m down. We still don't know how far away it is. We will figure that out once we know where the camera is.

The lens bends things

Real lenses aren't perfect. Wide lenses especially bend straight lines into curves, more and more towards the edges of the image. This is called distortion, and it means the direction we got from is slightly off.

Distortion is described by a handful of numbers (usually called for the radial bending, and for the lens sitting slightly crooked). Like , they're measured once per camera, typically by filming a checkerboard.

A catch is that this formula only goes one way. It tells us where a perfect direction ends up in the distorted image, but not the other way around. To undo it, we guess, apply distortion to the guess, see how far off we are, correct and repeat. After a few rounds the guess stops moving.

For our example we'll assume the distortion near the car is small, so we keep the direction .

Where is the camera?

Our direction is still described from the camera's point of view. To use it in the parking lot we need camera2world, a matrix that holds the camera's rotation (how it's tilted) and its position:

Our camera sits in the middle of the row, 6 m up and 12.5 m behind the origin, and it's tilted about 16° down:

The first three columns are the camera's right, down and forward directions written in world coordinates, and the last column is the camera's position: .

To transform our direction we add a 4th number, . A point gets , so it's rotated and moved. A direction gets , so it's only rotated, because moving a direction makes no sense:

So is the same direction, just described with the parking lot's axes instead of the camera's. The camera is already tilting a bit down, and we compensate for that in the world's perspective.

So in the parking lot, the car lies along a line starting at the camera and pointing slightly right, down and forward.

camera2world comes from clicking the 4 corners of the parking row in the image. We know where those corners are in the world, , so we can solve for the rotation and position that make them line up. This is called PnP (Perspective-n-Point), and OpenCV does it with cv2.solvePnP.

Hitting the ground

Now we use the last prerequisite: the car is on the ground. We start at the camera and walk along the direction until the height is 0:

We drop (it's 0) and get : real meters in the parking lot. is between and , so the car is inside the row. Which spot?

The car is in spot 7.