Syllabus

Module 3: Augmented Reality

Introduction to Augmented Reality: Definition and Scope, Methods of
Augmentation, Spatial Display Models, Visual Display, Stationary Tracking Systems,
Mobile Sensors. Introduction and Installation of Unity 3D, using 3D View, controlling
lamps, lights.

www.augmentedrealitybook.org Displays 1
 Introduction to
Augmented Reality
Augmented Reality – Principles and Practice

http://www.augmentedrealitybook.org
What is AR?
Azuma‘s definition:

• Combines real and virtual

• Interactive in real time
• Registered in 3D

www.augmentedrealitybook.org Introduction
AR Feedback Loop
AR uses a feedback loop between human user and computer system. The user observes
the AR display and controls the viewpoint. The system tracks the user’s viewpoint, registers
the pose in the real world with the virtual content, and presents situated visualization.

Registration
Situated
Virtual content of virtual
visualization
content

SPATIAL MODEL

Real world model User input

Pose
and camera
tracking
movement

www.augmentedrealitybook.org Introduction
 BACKGROUND OF THE STUDY
• Augmented Reality (AR) is a live view
of a physical real-world environment
whose elements are merged with (or
Augmented by) virtual computer-
generated imagery.

• With the help AR technology,

information about the surrounding real
world of the user becomes interactive
and digitally usable.
Figure 1: Augmented Reality
 BACKGROUND OF THE STUDY

• AR should not be confused for

Virtual Reality (VR).

• VR is a completely simulated
computer generated environment
where everything is under the
control of a system.

Figure 2: Virtual Reality

 History
• Historical Development of
Augmented Reality
• In 1968, a Harvard professor and computer
scientist by the name of Ivan Sutherland
invented what he called The Sword of
Damocles. He invented this first sort of
augmented reality device with his student, Bob
Sproull.

Figure 3: Sword of Damocles

 Historical Development (cont…)

• One of the next big developments in augmented reality was in 1974 by Myron
Krueger. The project was called, Videoplace, which combined a projection system
and video cameras that produced shadows on the screen.

• In 1990, a Boeing researcher named Tom Caudell coined the term “Augmented
Reality”.
 Historical Development (cont…)

• In 1992, Louis Rosenburg from

the USAF Armstrong’s Research
Lab created the first real
operational augmented reality
system, Virtual Fixtures.

• Figure 4: Virtual Fixtures

 Historical Development (cont…)
• In 1994 Julie Martin produced the first theater production to use AR, it
was called “Dancing in Cyberspace”.

• In 1998, Sportsvision uses the 1st and Ten-line computer system.

• In 1999, NASA used a hybrid synthetic vision system that integrated

augmented reality in their X-38 spacecraft.
 Historical Development (cont…)
• In 2000 when Hirokazu Kato from the Nara Institute of Science and
Technology in Japan created and released software called ARToolKit.

• In 2003, the NFL used the popular Skycam, which was used for aerial views
of the field to insert the virtual first down marker.

• In 2009, Esquire magazine, in collaboration with Robert Downey Jr., used

augmented reality in their print media.
 Historical Development (cont…)

• In 2013, Volkswagen used an

AR application called
MARTA as their car
manuals.

• Figure 5:
 Historical Development (cont…)

• In 2014, the Google

Glass is revealed and is
made available for
consumers.
• Figure 6: Google Glass
 Historical Development (cont…)

• In 2016, Microsoft
introduces the next iteration
of wearable augmented
reality, the HoloLens.

• Figure 6: Microsoft HoloLens

 Historical Development (cont…)

• In 2018 Magic Leap an AR

technological company came
out with their AR glasses

• Figure 7: Magic Leap

 REVIEW OF AUGMENTED REALITY
• Creating Augmented Reality

• To create AR, you first need to capture some actual reality with Sensors and
Cameras that gather information on the users' actual surroundings.

• Processing: Realistic augmented reality also requires enough processing

power to analyze inputs like acceleration, position, tilt, and depth in real-time to
create immersive interactions.
 REVIEW OF AUGMENTED REALITY
• Creating Augmented Reality (cont..)

• Projection: After capturing real-world information, the augmented

reality device then uses projection to layer digital renderings onto the
scene.
 APPLICATIONS OF AR

• Architecture: AR can aid in

visualizing building projects.
Computer-generated images of a
structure can be superimposed
into a real-life local view of a
property before the physical
• Figure 8: AR in Architecture
building is constructed there.
 APPLICATIONS OF AR

• Education: In educational settings,

AR has been used to complement a
standard curriculum. Text, graphics,
video, and audio may be
superimposed into a student's real-
time environment.

Figure 9: AR in Education
 APPLICATIONS OF AR

• Medical: AR can be very helpful in

the medical field. It could be used
to provide crucial information to a
doctor or surgeon with having them
take their eyes off the patient.

• Figure 10: AR in Medical

 APPLICATIONS OF AR

• Military: In combat, AR can serve

as a networked communication
system that renders useful
battlefield data onto a soldier's
goggles in real time.

• Figure 11: AR in Military

 APPLICATIONS OF AR

• Gaming: AR gaming uses the

existing environment and creates a
playing field within it. AR games
are typically played on devices like
smartphones, tablets and portable
gaming systems.

• Figure 12: AR in Gaming

 APPLICATIONS OF AR

• Workplace: Big machines are

difficult to maintain because of
their multiple layers or structures.
AR permits users to look through
the machine as if it’s an x-ray,
pointing them to the problem right
away. • Figure 13: AR in Workplace
Augmented Reality Building Acceptance
The Planar is a touchscreen display on wheels discrepancy analysis directly on the factory floor

Image: Ralph Schönfelder

www.augmentedrealitybook.org Introduction
 Underground Infrastructure Inspection
Tablet computer with differential GPS Geo-registered view of a virtual excavation revealing a gas pipe

Image: Gerhard Schall

www.augmentedrealitybook.org Introduction
Flight Control for Aerial Reconstruction

While the drone has flown far away and

is barely visible, its position can be
visualized using a spherical AR overlay
Image: Stefanie Zollmann
www.augmentedrealitybook.org Introduction
Explaining How Things Work
Ghost visualization revealing the interior of a coffee machine to guide end-user maintenance

Image: Peter Mohr

www.augmentedrealitybook.org Introduction
Maintainence Instructions

Automatically generated disassembly sequence of a valve

Image: Peter Mohr

www.augmentedrealitybook.org Introduction
Office of the Future
The Office of the Future project at UNC used projector-camera systems for immersive telepresence

Image: Henry Fuchs, UNC Chapel Hill

www.augmentedrealitybook.org Introduction
 Image Based Telepresence

A car repair scenario assisted by

a remote expert via AR tele-
presence on a tablet computer

Image: Steffen Gauglitz Remote expert draws hints directly on 3D model of

car incrementally transmitted from the repair site
www.augmentedrealitybook.org Introduction
 Original

Medical Diagnosis
X-ray

Live CamC

90° CamC view

Dry phantom
With shrapnel

The CamC is a mobile C-arm, which allows a

Camera view
physician to seamlessly blend between a X-ray overlay

conventional camera view and X-ray images

Bone

Metal inside tissue

Hand and tool

Image: Nassir Navab

www.augmentedrealitybook.org Introduction
 AR Browser

AR browsers such as Yelp

Monocle superimpose points
of interest on a live video feed

www.augmentedrealitybook.org Introduction
 Translation

Google Translate superimposes spontaneous

translations of text, recognized in real time,
over the camera image

www.augmentedrealitybook.org Introduction
Navigation
Peak.AR showing mountain tops Wikitude Drive superimposes a
perspective view of the road ahead

Image: Wikitude GmbH

www.augmentedrealitybook.org Introduction
Parking Assistant
The parking assistant is a commercially available AR feature in many contemporary cars

Image: Brigitte Ludwig

www.augmentedrealitybook.org Introduction
Sport Broadcast Visualization
Augmented TV broadcast of a soccer game

Image: Teleclub and Vizrt, Switzerland (LiberoVision AG)

www.augmentedrealitybook.org Introduction
Augmented Magazines
The lifestyle magazine Red Bulletin was the first print publication to feature dynamic content using AR

Image: Daniel Wagner

www.augmentedrealitybook.org Introduction
Marketing
Marketing presentation of a Waeco air-conditioning service unit

Image: magiclensapp.com
www.augmentedrealitybook.org Introduction
Virtual Try-On

Pictofit extracts garment images

from shopping sites and renders
them to match customer image

Image: Stefan Hauswiesner, ReactiveReality

www.augmentedrealitybook.org Introduction
Eye of Judgement
Sony‘s Eye of Judgement was a
Mixed Reality tabletop game for
the PlayStation 3

www.augmentedrealitybook.org Introduction
Vuforia SmartTerrain

Vuforia SmartTerrain scans the environment and turns it into a game landscape

Images: © 2013 Qualcomm Connected Experiences, Inc. Used with permission

www.augmentedrealitybook.org Introduction
 Immersive Games
Using a TV-plus-projector setup, the IllumiRoom extends the game world beyond the screen boundaries

Image: Microsoft Research

www.augmentedrealitybook.org Introduction
 APPLICATIONS OF AR
• Other areas of AR application includes:

Music

Retail and Advertising

Broadcast and Live Events

Social Media…
Research Themes
 AR / VR / Display hardware

 3DUI / Context-aware systems

Agenda
• Displays • Tracking
• Visual perception • Calibration and registration
• Display characteristics • Tracking system characteristics
• Spatial display model • Stationary and Mobile Tracking
• See-through displays • Optical tracking, Computer Vision
• Hand-held Displays • Sensor fusion
• Projected Displays

www.augmentedrealitybook.org Introduction
 AR Displays
Augmented Reality – Principles and Practice

http://www.augmentedrealitybook.org
Multi-modal Augmented Reality Displays
• Visual displays (see)
• Audio displays (hear)
• Haptic displays (sense)
• Olfactory displays (smell)
• Gustatory displays (taste)

www.augmentedrealitybook.org Displays 47
Audio Augmented Reality

Applications:
• Museum Audio Guides (since the 1950s)
• Assistive Audio Guidance (Loomis et al. 1993)
• Workplace Communication (Mynatt et al. 1998)

Considerations:
• Earphones vs. near-ear speaker arrays (“hear-through” audio)
• Spatial Audio (personalized HRTF vs. one-size-fits-all)

www.augmentedrealitybook.org Displays 48
Visuo-Haptic RegistrationVideo: Cosco et al., ISMAR 2009

The stylus of a PHANToM haptic device is

highlighted by visual AR

Image: Ulrich Eck and Christian Sandor

www.augmentedrealitybook.org Displays 49
Olfactory AR Display
MetaCookie: An olfactory display is combined with visual
augmentation of a plain cookie to provide the illusion of a
flavored cookie (chocolate, in the inset).

Image: Takuji Narumi

www.augmentedrealitybook.org Displays 50
 Method of
Augmentation
Three types:
1. Optical see-through
2. Video see-through
3. Spatial projection (spatial AR, projection-based
AR, or spatial projection.

www.augmentedrealitybook.org Displays 51
Visual Displays
• See-through displays
• Optical see-through
• AR headsets
• Video see-through
• HMDs
• Hand-held AR

• Projection-based Displays:
• Head-Mounted Projective Display
• Spatial Augmented Reality

www.augmentedrealitybook.org Displays 52
 Requirements and
Characteristics
• Method of Augmentation
• Ocularity and Stereoscopy
• Focus
• Occlusion
• Brightness and Contrast
• Resolution and Refresh Rate
• Field of View
• Viewpoint Offset
• Latency
• Distortions and Aberrations
• Ergonomics
• Social Acceptance

www.augmentedrealitybook.org Displays 53
Optical See-Through Displays

Fig: An optical see-through display uses an optical element to combine a user’s view
of the real world with computer-generated images

www.augmentedrealitybook.org Displays 54
Video See-Through Displays

Fig: A video see-through display captures the real world with a video camera and electronically
modifies the resulting image using a graphics processor to deliver a combined real + virtual image to
www.augmentedrealitybook.org the user
Displays 55
Spatial Augmented Reality
Fig: Spatial projection casts
images directly onto real-world
objects

No combiner unit is required

www.augmentedrealitybook.org Displays 56
Brief History of Spatial Augmented Reality
• Concept (before ’90) ~ development (’00) ~ widespread (’10)

1969 1980 1993 1998 1999

Disney’s Naimark’s DigitalDesk Office of the Future Shader Lamps

Haunted Mansion Displacement

2005 2010 2013 2014

MicroSoft IllumiRoom Once Upon a Time

SmartProjector Coded aperture at Tokyo Disney Land
Example of Spatial AR
• RoomAlive (2014)
 Non-See-Through Displays

Oculus Rift is a binocular HMD intended for immersive computer games. Oculus was acquired by Facebook in
2014 for $2 billion, raising the interest in HMD technology worldwide

The Samsung Gear VR is an example of a

head-mounted display that uses a
smartphone (here: Galaxy S6) as the
main I/O and computational engine
www.augmentedrealitybook.org Displays 59
Optical See-Through Headsets

Microsoft HoloLens
ODG R-7 Smart Glasses

Meta-2
DAQRI Smart Helmet
www.augmentedrealitybook.org Displays 60
Displays

See-Through Display Taxonomy

See-Through
Displays

Monocular Binocular

Video Optical Video Optical

See-Through See-Through See-Through See-Through

Single Dual Monoscopic Stereoscopic

Camera Camera Overlays Overlays

Example
Products Monoscopic Stereoscopic Monoscopic Stereoscopic
E.g.: smartphone- or E.g.: Microvision Overlays Overlays Overlays Overlays
tablet-based Nomad,
hand-held AR DigiLens DL40, E.g.: Vuzix iWear VR920 E.g.: Microsoft HoloLens,
Also: Google Glass in TacEye ST, E.g.: Trivisio with Possible, but no E.g.: Canon COASTAR, Epson Moverio BT-200,
VST mode Vuzix M2000AR ARVision iWear CamAR clear advantage Vuzix Wrap 1200DXAR E.g.: Lumos DK-40 Vuzix STAR 1200XLD

www.augmentedrealitybook.org 61
Three Factors of AR
１）Real + Virtual  Display technology
２）Real-time interaction  UI technology
３）3D registration  Tracking technology

Real + Virtual
Image composition

Wearable Movies /
computing TV ads
AR /
MR
2D GUI 3DCG software
VR

Real-time Interaction 3D Content

R. T. Azuma, “A Survey of Augmented Reality,” in MIT Press Presence: Teleoperators and Virtual
Environments Vol.6, No.4 (August 1997), pp.355-385.
AR Display Classification
Projectors
Virtual
retinal displays
Brother
Eyepiece-based
HMDs
Handheld Stationary displays
displays with half mirrors

1. Head mount
Oculus + Moverio Once Upon a Time
Ovrvision
displays

2. Handheld display systems

VeinViewer

IKEA Catalog Target

real
3. Stationary display systems object
4. Projection
-based systems
(Spatial AR) 63
MRsionCase
Inspired by the original figure by O. Bimber
AR Display Classification
Type Pros Cons
HMD • Available anywhere
• VRD • Cumbersome wearing
• Also suitable for
• Optical STH • Social acceptance
outdoor / multi-
• Video STH • Poor registration
user
Handheld • On-screen overlay,
• Simple and handy not in the real world
• Video STH • Very high • No stereovision
penetration • Needs to hold the
device
Stationary • Optical STH
• Video STH • Precise registration • Physically bulky
• Volumetric relatively easy • Image unreachable
displays
SAR • High congruence • Affected by material /
• Image btw real and virtual lighting
projection • Suitable for large • Not suitable for mid-
audience air images
64
 Stereo Video See-Through Display

COASTAR was the first

commercial parallax-free
video see-through HMD

Canon MR Laboratory, 2002

Vuzix Wrap 1200DXAR (2014)

Image: Hiroyuki Yamamoto

www.augmentedrealitybook.org Displays 65
Video See-Through with Half-Silvered Mirror

Example of VST HMD using cameras

above the eyes with mirror optics.

Design by Andrei State, 2005.

Image: Andrei State, UNC Chapel Hill

www.augmentedrealitybook.org Displays 66
Oculus Rift with Stereo Video See-Through
The AR-Rift, a modified Oculus Rift with two video cameras

Image: William Steptoe

www.augmentedrealitybook.org Displays 67
 Focus
(Accommodation)
• Depth of Field (lens systems: only a
certain range in focus)
• Eyes can accommodate at varying
distances (muscles shaping lens)
• Eyes can verge (rotation of eyballs) at
different locations/distances

• Conventional optical systems have

fixed focal depth
 Accommodation-Vergence Conflict

www.augmentedrealitybook.org Displays 68
Light-Field Displays
Pinlight display prototype

Panel with a dense array of point light sources

View through
the display

www.augmentedrealitybook.org Displays 69
Image: Andrew Maimone, UNC Chapel Hill
Focus (Accommodation)

www.augmentedrealitybook.org Displays 70
 Optical See-Through with Real Occlusion

Virtual image

Real image

LCD panel
Half-silvered
mirror

The ELMO HMD uses an additional LCD panel

between display and optical combiner for pixel-wise Image: Kiyoshi Kiyokawa
blocking of occluded real-world objects

www.augmentedrealitybook.org Displays 71
Image Quality Comparison

Image quality in optical see-

through displays is higher for the
real world, but generally
inconsistent. The (normally
occluded) tips of the pincers are
rendered as augmentations. This
illustrative mockup shows
exaggerated resolution artifacts for
the augmentation part on the left
and the entire image on the right

www.augmentedrealitybook.org Displays 72
Field of View
Comparison
• AR systems typically have a limited field
of view, resulting in an “overlay FOV” area,
in which augmentations are visible, and a
“peripheral FOV” area, in which they are
not

www.augmentedrealitybook
.org Displays 73
Field of View Simulation

www.augmentedrealitybook.org Displays 74
Registration Comparison

Insufficient eye-to-display
calibration can lead to
distracting offsets. In video
see-through displays, pixel-
accurate registration is
easier to achieve

www.augmentedrealitybook.org Displays 75
 Image Displacement
Pose Sensors (optional) A camera pointing diagonally downward from
Image behind the display captures an AR interaction
Sensors space centered on the user’s hands

Image
displacement
Monitors

In general, an offset between the user’s viewing direction

and the camera’s optical axis is not desirable Image: Morten Fjeld

www.augmentedrealitybook.org Displays 76
Brightness Comparison

Optical see-through displays

depend on the transparency of
the optical combiner, while
video see-through displays can
change brightness and contrast
arbitrarily, as long as the display
itself can deliver sufficient
contrast. On the right, the
contrast limit is reached, and
some real-world detail is lost.

www.augmentedrealitybook.org Displays 77
Failure Comparison

If the display fails, video see-

through will not allow the
user to see anything.

This can be dangerous in

critical situations such as
surgery or piloting an aircraft.

www.augmentedrealitybook.org Displays 78
Display Mounting

Helmet-mounted Clip-on Visor Display

Rockwell Collins SimEye Google Glass Epson Moverio

www.augmentedrealitybook.org Displays 79
Display Space Taxonomy
AR displays can be categorized according to the distance from eye to display

Head-mounted Hand-held Stationary Projected

display display display display

Head space Body space World space

www.augmentedrealitybook.org Displays 80
Optical See-Through Examples

MicroVision Nomad Retinal Scanning Display Sony Glasstron LDI-D100B,

Image: MicroVision retrofit on a custom mount as part of
the Columbia MARS system
Image: Columbia University
www.augmentedrealitybook.org Displays 81
See-Through Display with Optical Prism

The Light-Guide optical element technology by Lumus

Image: Jens Grubert propagates an image through a special optical prism
www.augmentedrealitybook.org Displays 82
Spatial Model of AR Displays
AR displays can be characterized according to the spatial relationship of up to five components:
the user’s eye, the display, the camera, an object to be augmented, and the world.

Each coordinate transformation can be fixed and calibrated, tracked dynamically, or left unconstrained.

www.augmentedrealitybook.org Displays 83
Optical See-Though

T
C

T
T

Without eye tracking With eye tracking

C – Constant, Calibrated Transformation

T – Tracked Dynamic Transformation

www.augmentedrealitybook.org Displays 84
Video See-Though

C T T T
C C

Without eye tracking With eye tracking

C – Constant, Calibrated Transformation

T – Tracked Dynamic Transformation

www.augmentedrealitybook.org Displays 85
Head-Mounted Projective Display
incident light

diffusion Retro-reflective materials send incident rays

reflection retro-reflection back to the illuminating source, so they work
well with head-mounted projector displays
Lambertian reflector
Mirror reflector Retro-reflector
(e.g. unfinished wood)

Spatial relationship
C Spatial relationship C schematics for HMPDs with
schematics for HMPDs T head tracking - virtual
C without head tracking C objects are stable in space,
while the viewer is moving

www.augmentedrealitybook.org Displays 86
Head-Mounted Projective Displays

www.augmentedrealitybook.org Displays 87
Hand-Held Display
T
C

A handheld AR display can be realized from an

unmodified smartphone or tablet computer

Image: Daniel Wagner

www.augmentedrealitybook.org Displays 88
 User-Perspective Hand-Held Display
Handheld display with device perspective Handheld display with user perspective

T T
C

Image: Domagoj Baričević

www.augmentedrealitybook.org Displays 89
User-Perspective Hand-Held Display

www.augmentedrealitybook.org Displays 90
Desktop AR

A desktop AR display can be built using the eyeball-in-

T hand metaphor, in which the camera is tracked and its
recordings are fed to the display. In the application
depicted here ([Lee and Höllerer 2007]), we are tracking
the camera relative to an object (user’s hand), which is
recognized as a marker and subsequently augmented.

Often, the camera is stationary, covering a working

T
volume, in which augmentations can occur. Again, we
C are tracking the camera relative to a moving object
(checkerboard pattern)

www.augmentedrealitybook.org Displays 91
Video-See Through Magic Mirror The user (=box) must be tracked
with respect to the camera.
T
C
T

C
Display always shows the user,
independent of viewing angle.

T
C

C
Display behaves like real mirror.
Image: Matthias Straka and Stefan Hauswiesner
www.augmentedrealitybook.org Displays 92
Optical See-Through Magic Mirror
Andy Wilson of Microsoft Research showing the HoloFlector

Image: Microsoft Research

www.augmentedrealitybook.org Displays 93
Virtual Showcase T
C

The Virtual Showcase is a stationary optical see-through C

display intended for exhibitions, museums, and showrooms

Image: Oliver Bimber

www.augmentedrealitybook.org Displays 94
Transparent Display
Samsung Transparent Smart Window display, showcased at CES 2012

www.augmentedrealitybook.org Displays 95
Immaterial Display
Dual-sided
interactive
FogScreen

Two FogScreens in an
L-shaped configuration
produce a depth-fused
3D rendering for a
People can augment each other and tracked observer
interact through the FogSscreen

www.augmentedrealitybook.org Displays 96
Spatial Augmented Reality
Spatial AR can be used to turn generic objects into textured models

C
View-independent
spatial AR

Image: Michael Marner

www.augmentedrealitybook.org Displays 97
View-Dependent Spatial Augmented Reality
View-dependent spatial AR requires tracking the user,
but can present free-space 3D objects

Projector

Head tracker

Virtual
object

Image: Oliver Bimber

www.augmentedrealitybook.org Displays 98
 Spatial Augmented Reality with Projector Array
Multiple projectors can be Projectors
combined to minimize pixels
projected out of focus

The geometry of the projection surface

needs to be known (here: a display
calibrated to the world)

Projection Surface

www.augmentedrealitybook.org Displays 99
 Occluder Shadows
Projector-
based
illumination

The occlusion shadows technique uses controlled

illumination to blank out those portions of the real
world where opaque graphics should be visible Occluded
part masked
out from
illumination

Virtual Real

Image: Oliver Bimber

www.augmentedrealitybook.org Displays 100
Dynamic Shader Lamps Dynamic shader lamps
T
deliver spatial AR on
Painting with light on real surfaces tracked objects

Animatronic character with animated facial projection

Image: Michael Marner Image: Greg Welch, UNC Chapel Hill

www.augmentedrealitybook.org Displays 101

Steerable Projector
Everywhere Projector Display

T
A steerable, tracked projector
can display images anywhere

Image: Claudio Pinhanez, IBM Research

www.augmentedrealitybook.org Displays 102

 Tracking
Augmented Reality – Principles and Practice

http://www.augmentedrealitybook.org
Agenda
• Tracking, Calibration, and Registration
• Coordinate Systems
• Characteristics
• Stationary Tracking Systems
• Mobile Sensors
• Optical Tracking
• Sensor Fusion

www.augmentedrealitybook.org Tracking 104

Tracking, Calibration, and Registration
• Registration = alignment of spatial properties
• Calibration = offline adjustment of
measurements Calibration
• Spatial calibration yields static registration
• Offline: once in lifetime or once at startup
• Alternative: autocalibration
• Tracking = dynamic sensing and measuring Tracking Registration
of spatial properties Dynamic
registration

• Tracking yields dynamic registration

• Tracking in AR/VR always means “in 3D”!

www.augmentedrealitybook.org Tracking 105

Coordinate Systems
Eye
Local object coordinates
coordinates

Perspective transformation
• Calibrate offline
• For both camera and display

Model transformation View transformation

• Track for moving objects, • Track for moving objects,
if there are static objects as well if there are no static objects
• Track for moving observer

Global world
coordinates

www.augmentedrealitybook.org Tracking 106

Sensor arrangement
• Multiple sensors in rigid geometric configuration
• E.g., stereo camera rig
• Sparse or dense sensors
• E.g., digital camera is dense 2D array of intensity sensor with know angles
• Advanced technical issues
• Sensor synchronization
• Sensor fusion

www.augmentedrealitybook.org Tracking 107

Sensor Group Arrangement
Outside-in Inside-out
• Stationary mounted sensors • Mobile sensor(s)
• Good position, poor orientation • Good orientation, poor position

www.augmentedrealitybook.org Tracking 108

Stationary Tracking Systems
• Mechanical Tracking
• Electromagnetic Tracking
• Ultrasonic Tracking

www.augmentedrealitybook.org Tracking 109

Mechanical Tracking
• Track end-effector of articulated arm
• Joints with 2 or 3 DOF
CyberGrask Fakespace BOOM
• Rotary encoders or potentiometers
• High precision
• Fast
• Freedom of operation limited

www.augmentedrealitybook.org Tracking 110

Electromagnetic Tracking
• Stationary source produces three orthogonal magnetic fields
• Current induced in sensor coils
• Measurement of strength and phase of signal
• Signal strength falls off quadratically with distance
• Working range: half-sphere with 1-3m radius
• Prolems with electromagnetic interference

www.augmentedrealitybook.org Tracking Razer Hydra 111

Ultrasonic Tracking
• Measures time of flight of sound pulse
• Trilateration of 3 measurements
• Requires synchronized time (cables) or more than 3 measurements
• Low update rate (10-50Hz) due to slow speed of sound
• Possible fusion with fast inertial sensors (e.g., InterSense IS-600)
• Requires open line of sight
• Suffers from noise or change of temperature
• Wide-area configuration, e.g., AT&T BAT system
• Microphones mounted in ceiling

www.augmentedrealitybook.org Tracking
Image: Joseph 112
Newman
Mobile Sensors
• Global Positioning System
• Wireless Networks
• Magnetometer
• Gyroscope
• Linear Accelerometer
• Odometer

www.augmentedrealitybook.org Tracking 113

Global Positioning System
• Planet-scale outside-in radiowave time-of-flight
• Requires clock synchronization
• Must receive signals from at least 4 satellites
satellites

GPS receiver

www.augmentedrealitybook.org Tracking 114

Differential GPS
• Compensate for atmospheric distortion
• Receive correction signal from base station via network
• Real-Time Kinematics (RTK) Differential GPS also uses signal phase
satellites

GPS receiver base station

correction
signal
www.augmentedrealitybook.org Tracking 115
Wireless Networks
• Measure signal strength from WiFi, Bluetooth, mobile phone towers
• Potential trilateration/triangulation
• Mostly only good for coarse location (e.g., based on WiFi SSID)
• Fingerprinting: carefully map the signal reception in a given area
• Recent use: Bluetooth iBeacon in department stores
• Assisted GPS: accelerate GPS initialization using WiFi or GSM id
• Skyhood, Google, Broadcomm etc.

www.augmentedrealitybook.org Tracking 116

 Magnetometer
• Electronic compass
• Measure direction of Earth
magnetic field in 3D
• Principle: magnetoresistance
(Hall effect)
• Often very distorted watch
measurements

www.augmentedrealitybook.org Tracking 117

Gyroscopes
Radial
movement

• Determines rotational velocity Coriolis

movement

• Electronic gyro Image: Hideyuki Tamura

• Measures Coriolis force of small vibrating object

• Micro-electromechanical system (MEMS)
• High update rate (1KHz)
• Only relative measurements
• Must integrate once to determine orientation  drift
• Laser gyro (fiber-optic gyro)
• Measures angular acceleration based on light interference
• Large, expensive, used in aviation rotation

www.augmentedrealitybook.org Tracking 118

Linear accelerometer
• MEMS device spring spring

• Displacement of small mass

mass
• Measures
• Change of electric capacity, or
• Piezoresistive effect of bending
• Subtract gravity (the difficult part!)
• Integrate twice numerically to get position
mass
• Drift problems
• Combine lin.acc., gyro + compass into inertial measurement unit (IMU)
www.augmentedrealitybook.org Tracking 119
Odometer
• Mechanical or opto-electrical wheel encoder
• E.g., traditional ball mouse

www.augmentedrealitybook.org Tracking 120