Point Cloud Segmentation: How Self-Driving Cars Sort a 3D World

A single rotation of a Velodyne LiDAR sensor -- the spinning cylinder often mounted on top of self-driving test vehicles -- can generate over 100,000 individual 3D points in a fraction of a second, and the car does this roughly 10 times per second while driving. That's over a million data points every second, and every single one of them is meaningless until the car's software can answer one question: what IS this point part of?

- What a point cloud is and how LiDAR sensors generate one - What segmentation means and why raw point clouds are useless without it - The difference between semantic segmentation and instance segmentation - How a real algorithm (PointNet) processes point clouds without needing a grid like an image

LiDAR (Light Detection and Ranging) works by firing laser pulses in all directions and measuring how long each pulse takes to bounce back. Since light travels at a known, constant speed, the sensor can calculate the exact distance to whatever the laser hit. Do this millions of times per second across many angles, and you get a 'point cloud' -- a 3D map made of individual dots, each with an (x, y, z) position in space. Unlike a camera image, a point cloud has no color and no grid structure -- it's just a scattered swarm of coordinates.

A raw point cloud tells you THAT something is 2 meters ahead and 1.5 meters tall, but not WHAT it is. Segmentation is the process of labeling every point (or grouping points) into categories: road, pedestrian, cyclist, parked car, curb, traffic sign. Without segmentation, a self-driving car's planning system has no way to decide whether the object ahead is a plastic bag blowing across the road (ignore it) or a child stepping off a curb (stop immediately).

Semantic segmentation labels every point with a category, like 'car' or 'road,' but doesn't distinguish between two different cars -- both get labeled 'car' with no separation between them. Instance segmentation goes further, identifying 'car #1' and 'car #2' as distinct objects, which matters enormously for tracking: the car's planning software needs to know that the vehicle 3 meters ahead is the SAME vehicle it saw one second ago, not a new one, so it can predict where that vehicle is heading.

Before 2017, many algorithms tried to force point clouds into a grid (like pixels in an image) so existing image-processing techniques could be reused -- but this wasted memory and lost precision. Researchers at Stanford introduced PointNet, a neural network architecture that processes the raw, unordered list of 3D points directly. PointNet uses a clever trick: since the order points are listed in shouldn't change the answer (a point cloud is a set, not a sequence), the network uses a symmetric function (max pooling) so that shuffling the points doesn't change the output at all.