Perception Under Uncertainty

Perfect perception is a fiction. Every sensor introduces noise. Every detection model sometimes misses objects and sometimes invents them. Every depth measurement has error. Every classification has a probability of being wrong. A robot that can only act when its perception is certain will act very rarely — the real world never provides certainty. The difference between a robot that works and one that fails is not perfect sensors; it is the ability to reason correctly and act safely despite imperfect information.

Sources of Perceptual Uncertainty

Perceptual uncertainty arises from multiple independent sources, each requiring different mitigation strategies. Sensor noise is the irreducible randomness in any physical measurement. Even a high-quality lidar has range error on the order of 1-2 centimeters. An accelerometer has white noise that accumulates with integration. Thermal effects, shot noise from photon counting, and electrical interference all contribute. Sensor noise is typically modeled as additive Gaussian noise with a known variance — the foundation of Kalman filtering. Ambiguity occurs when a single observation is consistent with multiple world states. A closed door and a wall may produce identical sensor returns. A person at the edge of a camera's field of view produces a partial silhouette that could be a person, a post, or a shadow. Ambiguity cannot always be resolved from a single frame; it requires integrating observations over time or from multiple viewpoints. Occlusion is the partial or complete hiding of objects by other objects or structures. A robot cannot perceive what it cannot see. A person walking behind a parked car is invisible to a front-facing camera until they emerge. Safe robot behavior in occluded environments requires modeling what might be in unseen regions, not just what is in seen ones. Distribution shift, explored in the context of machine learning generalization, is acutely present in perception: a detector trained on sunny outdoor scenes fails in rain, at night, or in fog. A SLAM system trained in one building type may struggle in a building with different geometry. Perception systems must either be trained on sufficiently diverse data to generalize, or must explicitly monitor for distribution shift and adapt. Model error is the gap between how the perception system models the world and how the world actually is. An object detector that has never seen a specific model of wheelchair will not recognize it. A lidar-based ground segmentation algorithm that assumes flat ground will fail on stairs.

Robust perception requires going beyond best-guess outputs to representing and propagating uncertainty explicitly. A detector that outputs only 'pedestrian at (x, y)' is less useful than one that outputs 'pedestrian at (x, y) with position uncertainty (sigma_x, sigma_y) and identity confidence 0.87.' The downstream planner can use these uncertainty estimates to make better decisions: slow down more when pedestrian uncertainty is high, take a wider berth when positional uncertainty is large. False positive rate and false negative rate quantify two modes of detection failure. A false positive (FP) — detecting an object that is not there — wastes resources and causes unnecessary avoidance maneuvers. A false negative (FN) — missing an object that is there — can lead to collision with an undetected obstacle. These rates trade off with the detection threshold: raising the confidence threshold reduces FPs at the cost of more FNs. The correct operating point depends on the relative cost of each error type. For an autonomous vehicle, a false negative on a pedestrian detection is catastrophic; a false positive causes an unnecessary brake. The operating point should be set very low on the FN side. For a warehouse robot detecting inventory items, the cost of a missed item is rescheduling a pick; the cost of a false detection is a wasted arm motion. The operating point is different. Perception system design always involves this asymmetric cost analysis. Out-of-distribution (OOD) detection is the problem of recognizing when an input falls outside the model's training distribution. A classifier that confidently labels a novel object with a familiar category name is dangerous. Modern approaches to OOD detection include monitoring the distribution of internal neural network activations, using ensemble methods that disagree on OOD inputs, and training explicit OOD detectors. When an OOD input is detected, the system should flag it rather than confidently misclassify it.