Spatial Computing

When Apple announced the Vision Pro headset in June 2023 at a starting price of $3,499, they did not call it a VR headset or an AR device. They called it a 'spatial computer.' The framing was deliberate: spatial computing is not a display technology — it is a computing paradigm in which digital information is positioned and manipulated in three-dimensional physical space. A surgeon overlaying MRI data onto a patient's skull, an architect walking through a building that exists only as a 3D model, a student pulling apart a holographic human heart — all of these are spatial computing, regardless of whether a headset, a phone, or a wall-sized display delivers the experience.

By the end of this lesson you will be able to: • Define spatial computing and distinguish it from traditional 2D screen-based computing • Explain the four core enabling technologies: SLAM, depth sensing, spatial anchors, and hand tracking • Describe concrete real-world applications in medicine, construction, education, and retail • Identify the major development frameworks (ARKit, ARCore, WebXR) and when to choose each

Traditional computing is flat. You interact with pixels arranged on a 2D plane using a mouse, keyboard, or touchscreen. The device has no model of the physical world around it — it does not know where it is in a room, how far away you are, or what surfaces exist nearby. Spatial computing adds a third dimension by making the device context-aware: it continuously tracks where it is in physical space, builds a model of the surfaces and objects nearby, and can place persistent digital content at specific real-world coordinates. The spectrum runs from fully transparent to fully synthetic: Augmented Reality (AR) — the real world is primary; digital elements are overlaid. Your phone camera sees your kitchen table and places a virtual coffee cup on it. Mixed Reality (MR) — digital objects interact physically with real surfaces. A holographic ball rolls down an actual physical ramp; your hand reaches into a virtual drawer. Microsoft HoloLens 2 operates here. Virtual Reality (VR) — the real world is entirely replaced. Meta Quest 3 and PlayStation VR2 create fully synthetic environments. Spatial awareness of the physical room is still needed for safety (the passthrough cameras on Quest 3 prevent you from walking into walls).

Spatial computing depends on four interacting hardware and software systems: 1. SLAM (Simultaneous Localization and Mapping) — the algorithm that allows a device to determine its own position and orientation in a physical space while simultaneously building a 3D map of that space, in real time, without GPS. It fuses data from cameras and an IMU (inertial measurement unit). The Apple Vision Pro runs SLAM at 90 Hz across 12 cameras to maintain sub-millimeter positional accuracy. SLAM was first formally described by Hugh Durrant-Whyte and John Leonard in 1986 and now runs in Roomba vacuums, Tesla Autopilot, and Mars rovers as well. 2. Depth sensing — hardware that measures the exact 3D distance to every point in the scene. Methods: structured light (project a pattern onto the scene, measure how it deforms — used in iPhone Face ID), time-of-flight (pulse a laser and time the return echo), and stereo cameras (triangulate depth from two offset images). LiDAR scanners on iPad Pro and iPhone 15 Pro use time-of-flight at centimeter accuracy up to about 5 m. 3. Spatial anchors — persistent digital markers tied to a real-world location or surface. An anchor records that 'the virtual whiteboard lives on this specific wall at these world coordinates.' When you close the app and return 6 hours later, the anchor is still there. Microsoft Azure Spatial Anchors and Apple's ARWorldMap both support cross-session and multi-user anchor persistence. 4. Hand tracking — cameras detect the 3D position of all 21 joints in each hand without gloves or physical controllers. Meta Quest 3 and Apple Vision Pro both use machine-learning models trained on tens of millions of hand pose images to infer finger position and gesture in real time at 90 fps.

Medicine — AccuVein projects a live map of the patient's subsurface veins directly onto their skin using near-infrared imaging and AR overlay, reducing first-attempt IV insertion failures by 3.5 times. Medivis and Surgical Theater allow neurosurgeons to plan tumor resections by walking through a 3D holographic reconstruction of the patient's MRI before the first incision. The HoloLens 2 is FDA-cleared for use inside the operating room. Construction — Trimble XR10 (a hardhat-integrated HoloLens 2 variant) overlays BIM (Building Information Model) blueprints at 1:1 scale on a building under construction, flagging deviations from spec in real time. AECOM and Skanska report 40-50% reductions in rework costs on pilot projects. Education — Case Western Reserve University's HoloAnatomy app replaced cadaver labs with a holographic human body students can disassemble layer by layer, from skin to bone to neurovascular bundle. Over 100 medical schools worldwide licensed it by 2024. Retail — IKEA Place uses ARKit to drop photorealistic 3D furniture models at accurate scale into the customer's actual room. Wayfair reports a 3.4 times reduction in return rates for products previewed in AR before purchase, because customers can verify fit and color before buying.

Choosing how to build a spatial app depends on your target platform: ARKit (Apple) — the most mature and highest-fidelity AR framework for iOS and visionOS. Offers advanced scene understanding, person segmentation, LiDAR support on Pro hardware, and tight integration with RealityKit for 3D rendering. iOS and macOS only — no Android support. ARCore (Google) — Google's AR framework for Android. Strong persistent anchor support and the Geospatial API allows AR objects to be pinned to GPS coordinates at city scale (used for outdoor AR experiences and navigation overlays). Works on 400+ certified Android devices. WebXR — an open W3C web standard that lets AR and VR experiences run inside a browser tab. Supported in Chrome (best), Firefox, and Safari (partial). No app install required — distribute an AR experience as a URL. Performance is lower than native, but the zero-friction distribution model is a major advantage for consumer applications. Unity and Unreal Engine — both provide AR Foundation plugins that target ARKit, ARCore, HoloLens, and Meta Quest from a single shared codebase. Most commercial spatial applications use Unity or Unreal because writing to one codebase and deploying to multiple platforms is far more efficient than maintaining separate native codebases.