How Self-Driving Cars Perceive the Road: Sensors, Processing, Limits

A self-driving car does not see the road the way you do. It does not have a unified visual field or intuition about where to look. Instead, it stitches together data from multiple sensors—cameras, radar, and light-detection systems—feeds that data through neural networks running on specialized processors, and makes decisions in milliseconds. Understanding how this works explains both what autonomous vehicles can do today and why they still struggle with edge cases that human drivers handle effortlessly.

The challenge is not building sensors. The challenge is building a system that fuses conflicting sensor readings, prioritizes relevant information, predicts what other road users will do, and makes safe decisions under uncertainty. Every major autonomous vehicle company has chosen a different balance between sensor types and processing philosophies, and those choices shape what roads each system can safely navigate.

The Multi-Sensor Approach

Most production autonomous vehicles rely on redundancy by design. Waymo, which operates paid robotaxi services in Phoenix, San Francisco, Los Angeles, and Austin, uses a suite of cameras positioned around the vehicle, custom-built LiDAR sensors, and radar—each providing complementary information. Cruise, General Motors’ autonomous division, employs a similar architecture with 29 cameras, 5 LiDAR units, and 6 radar sensors on its Origin vehicle. Tesla’s approach is notably different: it uses eight cameras, one radar, and twelve ultrasonic sensors, with the strategy of processing primarily through cameras and neural networks rather than relying on expensive LiDAR.

Why Redundancy Matters

No single sensor type is perfect. Cameras excel at identifying the type and color of objects—a red traffic light versus a green one—but struggle in heavy rain or fog. Radar can see through weather and detect moving objects, but cannot distinguish between a person and a cardboard box at close range. LiDAR creates a precise three-dimensional map of the environment even in darkness, but can be fooled by reflective surfaces or snow. When one sensor degrades, the others continue to provide information, allowing the system to maintain situational awareness rather than failing completely.

Processing Pipeline and Latency

Data from all sensors arrives asynchronously. A camera might capture an image at 30 frames per second, radar at 20 Hz, and LiDAR at 10–20 Hz depending on the system. The autonomous vehicle’s onboard computer must synchronize these streams, run object detection on images and point clouds, track moving objects across frames, predict their future positions, and plan a safe path—all within 100–200 milliseconds. Waymo uses custom silicon alongside Tesla’s Full Self-Driving Computer and NVIDIA’s Drive Atlan, processors designed specifically for this workload.

Cameras and Deep Learning

Read also: How Data Breaches Happen: From Phishing to Credential Stuffing · Inside Smartphone Chip Design and Manufacturing · From Pre-Seed to Series D: How Startup Funding Really Works

Modern autonomous vehicles use convolutional neural networks trained on millions of labeled images to detect pedestrians, cyclists, vehicles, traffic signs, lane markings, and road hazards. Each camera in a multi-camera system covers a specific field of view—some looking straight ahead, others covering the sides and rear. The vehicle’s perception system stitches these viewpoints together into a 360-degree understanding of its surroundings.

What Neural Networks Learn

A camera by itself is just a sensor. The neural network transforms raw pixel data into meaningful categories: this is a stop sign, this is a pedestrian wearing red clothing, this vehicle is turning left. Training these networks requires feeding them hundreds of thousands of hours of labeled driving footage and iterating until the model’s predictions match ground truth. Tesla has published that it trains on data collected from its entire fleet, while Waymo maintains controlled datasets and simulation environments for testing edge cases that might occur only once per million miles of driving.

The Limitation of Visible Light

Cameras cannot see in pitch darkness without infrared sensors, and they are degraded by heavy rain, fog, and snow. A camera cannot reliably read a stop sign covered in snow or detect a pedestrian wearing dark clothing at night. This is why redundancy with active sensors—radar and LiDAR, which generate their own signal rather than relying on reflected light—is critical for safety.

LiDAR: Seeing in 3D

LiDAR (Light Detection and Ranging) fires laser pulses in many directions and measures how long the reflection takes to return. By doing this thousands of times per second, it creates a dense point cloud representing the exact three-dimensional positions of every object around the vehicle. A LiDAR point cloud is not a photo—it is a set of coordinates, but that data is so detailed that nearby objects register as hundreds of thousands of points.

Cost and Miniaturization

LiDAR was once prohibitively expensive. Waymo’s custom sensors, integrated into the vehicle’s roofline, cost tens of thousands of dollars when the company first deployed them. Solid-state LiDAR—sensors without moving parts—have dropped in price from approximately $70,000 a few years ago to under $300–$500 in mass production, making the technology viable for consumer vehicles. This price drop is why LiDAR adoption is accelerating: several Chinese autonomous vehicle companies and legacy automakers are now integrating solid-state LiDAR into Level 2 and Level 3 driver-assistance features aimed at the mass market.

Weather Limitations

LiDAR struggles with heavy rain, dense fog, and snow, particularly wet snow that scatters laser light. Unlike radar, LiDAR cannot see through precipitation as well. Additionally, some LiDAR systems can be confused by reflective surfaces like wet road markings or metal barriers. These limitations mean that LiDAR alone is not sufficient—it must be fused with radar and camera data.

Radar for Motion and Weather

Radar emits radio waves and detects reflections from moving and stationary objects. It provides two critical pieces of information that cameras and LiDAR sometimes miss: precise velocity (how fast something is moving toward or away from the vehicle) and performance in adverse weather. Radar works reliably in heavy rain, fog, snow, and darkness because radio waves pass through precipitation and do not require line of sight.

The tradeoff is resolution. Radar cannot read a license plate or tell you the exact shape of an object. It excels at detecting that a car ahead is braking suddenly or that a motorcycle is approaching from the side. A typical autonomous vehicle uses multiple radar units—front, rear, and side-facing—to cover the entire perimeter and detect threats from all directions.

Sensor Fusion: The Brain’s Real Work

Individual sensors are pieces of the puzzle. Sensor fusion is the algorithm that combines them. When the camera detects a pedestrian and the LiDAR point cloud shows the same spatial location, the system’s confidence in that detection increases dramatically. When the camera is obscured by glare but the radar confirms something is moving in that direction, the vehicle trusts the radar. Modern autonomous vehicles run this fusion in real time, assigning confidence scores to each detection and making decisions based on the highest-confidence model of the world.

Prediction and Planning

Perception is not just about detecting what is around the vehicle—it is about predicting what will happen next. The system tracks every moving object over time, learns its trajectory, and predicts where it will be in the next few seconds. A cyclist leaning to the right and drifting toward the vehicle’s lane triggers different risk calculations than one riding straight. This predictive layer runs on top of the raw sensor fusion and feeds into path-planning algorithms that choose safe, legal, and efficient routes.

Real-World Challenges Remain

Despite advances, several failure modes persist. Autonomous vehicles struggle with unusual or never-before-seen scenarios: an overturned truck blocking an entire lane, a person standing still in the middle of the road without obvious reason, construction equipment that does not match any trained patterns. Rain-soaked roads that wash away lane markings confuse vision-based lane detection. Severe snow covers road edges, making it impossible to know where the road ends and a ditch begins. Extreme glare from low sun angles can overwhelm cameras. These edge cases do not happen every day, but they happen often enough that companies like Waymo limit their robotaxi service to specific geographies and weather conditions—Phoenix is sunny and warm most of the year, which helps.

In Brazil, additional challenges apply: many roads lack clear markings, potholes and sudden obstacles are common, and weather varies significantly by region. Autonomous vehicle companies have been slow to deploy in countries with less standardized road infrastructure, partly because their perception systems are trained on data from well-mapped regions with consistent road standards.

How Different Companies Approach the Problem

Waymo’s Sensor-Rich Model

Waymo’s robotaxis in the United States use a full sensor suite: custom LiDAR, cameras, and radar. The company has spent over a decade building detailed 3D maps of the roads where its vehicles operate, and every sensor reading is validated against these maps. This redundancy makes Waymo’s system extremely reliable in mapped urban environments but requires expensive up-front mapping work before a vehicle can operate safely in a new city.

Tesla’s Vision-First Approach

Tesla’s Full Self-Driving beta relies primarily on eight cameras and neural networks for perception, with radar and ultrasonic sensors as secondary inputs. This approach reduces hardware cost—Tesla does not use LiDAR—and enables rapid iteration through over-the-air software updates. The tradeoff is that vision-only systems can fail in ways that sensor-redundant vehicles might handle: a patch of dirt on a camera lens, extreme glare, or unusual weather can degrade performance. Tesla trains its networks on data from millions of vehicles in the real world, a data advantage that partially compensates for the simpler sensor set.

The Road Ahead

The cost of autonomous perception hardware continues to fall. Solid-state LiDAR is moving from premium to mainstream. Neural networks are becoming more efficient, requiring less compute to run at high frame rates. The next phase of improvement will likely focus on handling rare, dangerous scenarios better: more robust behavior in severe weather, more graceful degradation when sensors fail, and better prediction of human unpredictability.

For Brazilian roads specifically, autonomous vehicle adoption will depend on whether companies can train their perception systems on local road conditions—poor markings, irregular surfaces, informal traffic patterns—and whether the cost of the perception stack falls far enough to justify the vehicle’s price. That timeline remains years away.