INTRODUCTION TO XR

COMP 4010 Lecture One

Mark Billinghurst
July 27th 2021
mark.billinghurst@unisa.edu.au
Who am I ..
• Mark Billinghurst
• Director of the Empathic Computing Lab
• Univ. South Australia, Univ. of Auckland
• Conducting research in:
• Collaborative AR, AR/VR Interface Design, HCI
• Empathic Computing
• Previous worked at:
• Google, Amazon, Nokia, British Telecom
• MIT, Univ. of Washington, NAIST, Univ. of Canterbury
• PhD from University of Washington (2002)
UniSA IVE - https://unisa.edu.au/research/IVE/
Class Logistics
• Weekly lecture (2 hrs)
• Thursday 9 am – 11am [Adelaide time]
• Room: Online

• Weekly Lab (1 hr) [UniSA Students]

• Thursday 11-12

• Assessment [UniSA Students]

• 3 projects @ 20%, 30%, 40%
• Class participation @ 10%

• What you will need

• iOS or Android phone/tablet
• Access to laptop/PC for development
Equipment Available
• VR Laboratory
• 6 VR capable PCs
• High end graphics cards
• 3 Oculus Rift HMDs
• Wide Field of View display
• 1 HTC Vive HMD
• Room scale tracking
• IVE Equipment
• 6 HoloLens2 AR displays
• See through AR display
• 10 Oculus Quest HMD
• Self contained tracking
Snap Lens Studio

Available from https://lensstudio.snapchat.com/

Unity3D

Available from www.unity3d.com

Lecture Schedule – 13 lectures
Assessment
• Assignment 1: Mobile AR (20%)
• Develop a Snap Lens
• AR Tracking and Interaction

• Assignment 2: Interactive VR (30%)

• Create an interactive VR scene
• Oculus Quest, HTC Vive, Rift

• Assignment 3: HMD Experience (40%)

• Develop either AR/VR HMD experience
• Oculus Quest, Microsoft Hololens2

• Class Participation (10%)

• Watch at least one short YouTube AR/VR video each week
• Be prepared to present video each week
• Submit at least one comment on Slack channel each week
• ARVRClass Slack channel – will send invites
WHAT IS AR/VR/MR/XR?
1967 – IBM 1401 – half of the computers in the world, $10,000/month to run
2013 Google Glass
 The Incredible Disappearing Computer

1960-70’s 1970-80’s 1980-90’s 1990-2000’s 2010 -

Room Desk Lap Hand Head

Graphical User Interfaces

• Separation between real and digital worlds

• WIMP (Windows, Icons, Menus, Pointer) metaphor
Making Interfaces Invisible

(c) Internet of Things

Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments.
Internet of Things (IoT)..

(c) Internet of Things

• Embed computing and sensing in real world

• Smart objects, sensors, etc..
Virtual Reality (VR)

• Users immersed in Computer Generated environment

• HMD, gloves, 3D graphics, body tracking
The First VR Experience …

https://www.youtube.com/watch?v=pAC5SeNH8jw
Virtual Reality Definition
• Defining Characteristics
• Sense of Immersion
• User feels immersed in computer generated space

• Interactive in real-time
• The virtual content can be interacted with

• Independence
• User can have independent view and reaction to environment
 David Zeltzer’s AIP Cube
VR

Autonomy – User can to react to events

and stimuli.
Interaction – User can interact with
objects and environment. Autonomy

Presence – User feels immersed through Presence

sensory input and output channels
Interaction

Zeltzer, D. (1992). Autonomy, interaction, and presence. Presence: Teleoperators & Virtual Environments, 1(1),127-132.
VR Demo
Types of VR

2
3
Augmented Reality (AR)

• Virtual Images blended with the real world

• See-through HMD, handheld display, viewpoint tracking, etc..
Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real and Virtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

https://www.youtube.com/watch?v=cCL3bFuC4IA
Strong vs. Weak AR
• Weak AR
• Imprecise tracking
• No knowledge of environment
• Limited interactivity
• Handheld AR

• Strong AR
• Very accurate tracking
• Seamless integration into real world
• Natural interaction
• Head mounted AR
Augmented Reality Applications
AR vs VR
From Reality to Virtual Reality

Internet of Things Augmented Reality Virtual Reality

Real World Virtual World

Milgram’s Mixed Reality (MR) Continuum
Mixed Reality

Internet of Things Augmented Reality Virtual Reality

Real World Virtual World

"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
 Milgram’s Reality-Virtuality continuum
"...anywhere between the extrema of the virtuality continuum."

Mixed Reality

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Reality - Virtuality (RV) Continuum

P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
Augmented Virtuality

• VR with view of the real world

 Extended Reality (XR)
Extended Reality

Mixed Reality

Internet of Things Augmented Reality Virtual Reality

Real World Virtual World

Ubiquitous Computing Continuum

• Weiser Continuum
Milgram – Weiser
Continuum
 The Metaverse
• Neal Stephenson’s “SnowCrash”
• The Metaverse is the convergence of:
• 1) virtually enhanced physical reality
• 2) physically persistent virtual space
• Metaverse Roadmap
• http://metaverseroadmap.org/
Metaverse Dimensions
• Augmentation technologies that layer information onto our
perception of the physical environment.
• Simulation refers to technologies that model reality
• Intimate technologies are focused inwardly, on the identity
and actions of the individual or object;
• External technologies are focused outwardly, towards the
world at large;
Metaverse Components
• Four Key Components
• Virtual Worlds
• Augmented Reality
• Mirror Worlds
• Lifelogging
Mirror Worlds
• Mirror worlds are informationally-enhanced virtual
models of the physical world.
• Google Earth, MS Street View, Google Maps
LifeLogging
• Technologies record and report the intimate states and
life histories of objects and users
• Nike+, FitBit, Apple watch
HISTORY OF AR/VR
 History Timeline

https://immersivelifeblog.files.wordpress.com/2015/04/vr_history.jpg
When anything new comes along, everyone, like a child
discovering the world thinks that they’ve invented it, but you
scratch a little and you find a caveman scratching on a wall
is creating virtual reality in a sense.

Morton Helig (Hammit 1993)

Early History (30,000 BC - )

The history of VR is rooted in human’s first

attempts to reproduce the world around them
1800’s – Capturing Reality
• Panoramas (1790s)
• Immersive paintings
• Photography (1820-30s)
• Oldest surviving photo (Niépce, 1826)
• Stereo imagery (1830s)
• Wheatstone (1832)
• Brewster (1851)

• Movies (1870s)
• Muybridge (1878)
• Roundhay Garden Scene (1888)
Stereo Viewers

Wheatstone (1832)
Brewster (1860)
Viewmaster (1939)
3D Cinema Golden Era (1950-60s)

• Polarized 3D projection or anaglyph (red/blue)

Pepper’s Ghost (1862)

• Projecting onto glass to make ghost image appear on stage

• Dates back to Giambattista della Porta (1584)
The Master Key (1901) – AR Glass
AR display showing if people are good or evil

"It consists of this pair of spectacles. While you

wear them every one you meet will be marked
upon the forehead with a letter indicating his or
her character. The good will bear the letter 'G,'
the evil the letter 'E.' … Thus you may
determine by a single look the true natures of
all those you encounter.”
L. Frank Baum
1900s – Interactive Experiences
• Early Simulators (<1960s)
• Flight simulation
• Sensorama (1955)
• Early HMDs (1960s)
• Philco, Ivan Sutherland
• Military + University Research (1970-80s)
• US Airforce, NASA, MIT, UNC

• First Commercial Wave (1980-90s)

• VPL, Virtual i-O, Division, Virtuality
• VR Arcades, Virtual Boy
Link Trainer (1929 – 1950s)

• Flight Simulator Training

• Full six degree of freedom rotation
• Force feedback and motion control
• Simulated instruments
• Modeling common flight conditions

• Over 500,000 pilots trained

Link Trainer Video (1966)

• https://www.youtube.com/watch?v=MEKkVg9NqGM
Early HMD Patents
Early HUD (1958)

F16 – Head Up Display

Showing flight
information over the
real world
Sensorama (1955)
• Created by Morton Heilig
• Experience Theater
• Multi-sensory
• Visuals
• Sound
• Wind
• Vibration
• Smell
• No financial support
• Commercial failure
First HMD? (1961)

• Philco Headsight – Remote Camera Viewing

• Showing live camera view in monocular HMD
Ivan Sutherland (1960s)

Ivan Sutherland’s Head-Mounted Display (1968)

6
1
Sutherland Display
Super Cockpit (1965-80’s)
• US Airforce Research Program
• Wright Patterson Air Force Base
• Tom Furness III
• Multisensory
• Visual, auditory, tactile
• Head, eye, speech, and hand input

• Addressing pilot information overload

• Flight controls and tasks too complicated

• Research only
• big system, not safe for ejecting
LEEP Optics (1979)

• Large Expanse, Extra Perspective optics

• Developed by Eric Howlett

• Lens design for extremely wide field of view

• High resolution in centre, lower resolution in periphery
• 90o direct FOV, 140o corneal FOV
• Used as basis for most VR HMDs
LEEP Optics Design
The Data Glove (1981-82)
• Precursor, Sayre Glove
• Univ. of Illinois, 1977
• Thomas Zimmerman (1982)
• Fiber optic bend sensors
• Detecting finger bending
• Commercialized by VPL
• Mattel PowerGlove (1989)
VPL DataGlove Demo

• https://www.youtube.com/watch?v=fs3AhNr5o6o
NASA VIEW/VIVED (1981-86)
• Early HMD (McGreevy Humphries)
• LCD “Watchman” displays
• VIEW (Scott Fisher)
• Polhemus tracker
• LEEP-based HMD
• 3D audio (Convolvotron)
• DataGlove gesture input
• Simple graphics
VPL Research (1985 – 1999)

• First Commercial VR Company

• Jaron Lanier, Jean-Jacques Grimaud
• Provided complete systems
• Displays, software, gloves, etc
• DataGlove, EyePhone, AudioSphere
The University of North Carolina
at Chapel Hill (1980s-1990s)

Head-Mounted Displays
Tracking, Haptics, Applications 7
1
University of Washington (1989 - )

• Human Interface Technology Laboratory (HIT Lab)

• Founded by Tom Furness III
• Many AR/VR Innovations
• Virtual Retinal Display
• ARToolKit AR Tracking library
• GreenSpace shared VR experience
• VR and pain care
• VR and Education
First Industrial Use of AR (1990’s)

• 1992: Tom Caudell at Boeing coined the term “AR.”

• Wire harness assembly application begun
• Lead by Tom Caudell, and David Mizell
CAVE (1992)

• Projection VR system
• 3-6 wall stereo projection, viewpoint tracking
• Developed at EVL, University of Illinois Chicago

• Commercialized by Mechdyne Corporation(1996)

C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon and J. C. Hart. "The CAVE: Audio Visual
Experience Automatic Virtual Environment", Communications of the ACM, vol. 35(6), 1992, pp. 64–72.
CAVE Demo Video

• https://www.youtube.com/watch?v=aKL0urEdtPU
Desktop VR - 1995
• Expensive - $150,000+
• 2 million polys/sec
• VGA HMD – 30 Hz
• Magnetic tracking
Mobile/Wearable Systems (1995)

• 1995 Navicam (Rekimoto)

• Handheld AR

• 1997 Touring Machine (Feiner)

• Backpack AR, GPS, see-through display

• 1998 Tinmith (Thomas, UniSA)

• Outdoor gaming, CAD
Virtual Reality was HOT! .. In 1995..
Rise of Commercial VR Companies
• W Industries/Virtuality (1985 - 97)
• Location based entertainment
• Virtuality VR Arcades

• Division (1989 – 1998)

• Turn key VR systems
• Visual programming tools

• Virtual i-O (1993 -1997)

• Inexpensive gamer HMDs

• Sense8 (1990 - 1998)

• WorldToolKit, WorldUp
• VR authoring tools
Dactyl Nightmare

• https://www.youtube.com/watch?v=L60wgPuuDpE
Overview of VR in the 1990’s
Development of AR Tools

• 1996 CyberCode (Rekimoto)

• First matrix code tracking
• 1999 ARToolKit (Kato & Billinghurst)
• Open source tracking library
Tracking Demos

ARToolKit

Cybercode

https://www.youtube.com/watch?v=TqGAqAFlGg0
• April 2007 Computer World
• VR Voted 7th on list of 21 biggest technology flops
• MS Bob #1
Lessons Learned

• Don’t believe the hype

• Not everything is better in VR
• Many factors determine technology acceptance
• Human Centered Design/Design for users
• Need to move from Demo to Production
• Profitable niche markets first
• Follow the money
 Mobile Phone AR (2005)

• Mobile Phones
• camera
• processor
• display

• AR on Mobile Phones
• Simple graphics
• Optimized computer vision
• Collaborative Interaction
AR Advertising (HIT Lab NZ 2007)

• Txt message to download AR application (200K)

• See virtual content popping out of real paper advert
• Tested May 2007 by Saatchi and Saatchi
Wellington Zoo Demo

https://www.youtube.com/watch?v=edTjuXcce_c
Eye of judgement (2007)

• Sony Playstation3 game

• First AR console game
• Over 300,000 copies sold
• Used Eye camera + tracking cards
2007 - AR Reaches Mainstream

• MIT Technology Review

• March 2007
• One of the 10 most exciting
technologies
• Economist
• Dec 6th 2007
• Reality, only better
Google Searches for AR

• Cross over in 2009, with more interest in AR than VR

2008 - Browser Based AR

• Flash + camera + 3D graphics

• ARToolKit ported to Flash
• High impact
• High marketing value

• Large potential install base

• 1.6 Billion web users

• Ease of development
• Lots of developers, mature tools

• Low cost of entry

• Browser, web camera
Demo: GE Smart Grid

• https://www.youtube.com/watch?v=vJO_AZkCL9U
2008: Location Aware Phones

Motorola Droid Nokia Navigator

Outdoor Information Overlay
• Mobile phone based
• Tag real world locations
• GPS + Compass input
• Overlay graphics on live video

• Applications
• Travel guide, Advertising, etc.
• Wikitude, Metaio, Layar, etc..
• iOS/Android, Public API released
Layar Demo (2008)

• https://www.youtube.com/watch?v=b64_16K2e08
 AR in Magazines (2009- )

• Esquire Magazine
• Dec 2009 issue
• 12 pages AR content

• Many Others
• Wired
• Colors
• Red Bull
• etc.
Esquire Demo
Google Glass (2011 - )
Google Glass Demo
VR Second Wave (2010 - )
• Palmer Luckey
• HMD hacker
• Mixed Reality Lab (MxR) intern

• Oculus Rift (2011 - )

• 2012 - $2.4 million kickstarter
• 2014 - $2B acquisition FaceBook
• $350 USD, 110o FOV
The Oculus Kickstarter Video

• https://www.youtube.com/watch?v=aNSYscbxFAw
Desktop VR in 2016
• Graphics Desktop
• $1,500 USD
• >4 Billion poly/sec
• $600 HMD
• 1080x1200, 90Hz
• Optical tracking
• Room scale
2016 - Rise of Consumer HMDs

Oculus Rift
HTC/Valve Vive

Sony Morpheus
HTC Vive

• Room scale tracking

• Gesture input devices
Example Vive App – Tilt Brush

• https://www.youtube.com/watch?v=ijukZmYFX-0
VR2GO (2013)

• MxR Lab
• 3D print VR viewer for mobiles
• Open source hardware + software
• http://projects.ict.usc.edu/mxr/diy/vr2go/

• Early Mobile VR viewer

Google Cardboard

+ =

• Released 2014 (Google 20% project)

• >5 million shipped/given away
• Easy to use developer tools
Multiple Mobile VR Viewers Available
Epson Moverio BT-300

▪ Stereo see-through display ($700)

▪ 1280 RGB x 720 pixels, 23 degree FOV, 30Hz, 69g
▪ Android Powered, separate controller
▪ VGA camera, GPS, gyro, accelerometer
Smart Glasses Available
Social Mobile Camera AR Apps (2015 - )

• SnapChat - Lenses, World Lenses

• Cinco de Mayo lens > 225 million views
• Facebook - Camera Effects
• Google – Word Lens/Translate
Hololens (2016)

• Integrated system – Windows

• Stereo see-through display
• Depth sensing tracking
• Voice and gesture interaction
• Note: Hololens2 coming September 2019
ARKit/ARcore (2017)

• Visual Inertial Odometry (VIO) systems

• Mobile phone pose tracked by
• Camera (Visual), Accelerometer & Gyroscope (Intertial)

• Features
• Plane detection, lighting detection, hardware optimisation

• Links
• https://developer.apple.com/arkit/
• https://developers.google.com/ar/
ARKIT3 Demo

• https://www.youtube.com/watch?v=SgAnnwl2VB8
MagicLeap ML-1 (2018)

• Bi-Focal Display – two focus planes

• Horizontal FoV of 40o, vertical FoV of 30o, diagonal value of 50o
• 1280×960 resolution, Eye-tracking
• Separate display and computer
• Nvidia "Parker" Tegra X2 CPU, 8GB RAM, 128 GB storeage
• 6 DOF handheld controller, magnetic tracking
History Summary
• 1960’s – 80’s: Early Experimentation
• 1980’s – 90’s: Basic Research
• Tracking, displays
• 1995 – 2005: Tools/Applications
• Interaction, usability, theory
• 2005 - : Commercial Applications
• Mobile, Games, Medical, Industry
QUESTIONS
www.empathiccomputing.org

mark.billinghurst@auckland.ac.nz

@marknb00