Magic Windows: How Your Phone Adds Pictures to the Real World

In the summer of 2016, a game called Pokemon GO got downloaded more than 500 million times in just two months. Players walked around real parks, sidewalks, and living rooms, holding up their phones to 'see' cartoon creatures standing right there on the grass or the carpet — creatures that weren't really there at all. That trick is called augmented reality, or AR, and once you understand the three things it needs to work, you'll start noticing it everywhere.

- What augmented reality means, and how it's different from virtual reality - The three ingredients a phone needs to make AR work - Real apps that use AR today, and what job AR does in each one - How to sketch your own idea for an AR app

Augmented reality means adding digital images on top of the real world you can actually see, usually through your phone or tablet's camera. This is different from virtual reality (VR), which replaces the real world completely with a fake one you view through a headset. In AR, you still see your actual living room, backyard, or classroom — the app just adds something extra on top, like a cartoon character standing on your real floor or a piece of furniture placed in your real hallway.

An AR app needs three things working together at the same time. First, it needs the camera feed — the live video your phone's camera is already capturing. Second, it needs tracking — sensors inside the phone (like the gyroscope and accelerometer) that constantly measure exactly how you're tilting, moving, and turning the phone, so the app knows where 'down' and 'forward' are. Third, it needs a 3D object to place — a digital model of whatever it's adding, like a Pokemon or a virtual couch. The app blends all three, dozens of times per second, so the digital object appears to stay locked in one spot in the real world even as you walk around it.

AR shows up in more places than games. The IKEA Place app lets you point your phone at your actual living room and see how a real couch would look, at real size, before you buy it. Snapchat and Instagram filters track the exact position of your face to add glasses, animal ears, or makeup that moves naturally as you turn your head. Some navigation apps overlay glowing arrows directly onto the live camera view of the street in front of you, showing exactly which way to turn. In every case, the app is combining a live camera feed with digital objects that are tracked to stay in the right place.

If you were sketching a plan for your own AR app, you would need to answer three questions, matching the three ingredients above: What will the camera be pointed at (a room, a book page, a face)? What digital object do you want to add (a creature, furniture, a fact bubble)? And how should that object behave as the camera moves (stay still on the floor, follow a face, pop up when it spots a specific picture)? Real AR developers sketch out answers to these exact questions before they write a single line of code.