Augmented and Mixed Reality

Applications at VTT

Charles Woodward

April 16, 2010

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Augmented Reality

one of the technologies to

most likely to alter industries, fields of research,

and the way we live.

- MITs annual review 10 Emerging Technologies 2007

- Gartners Top Ten Disruptive Technologies 2008-2012

- First consumer applications on camera phones 2009

- Commercial breakthrough predicted 2010

1
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

INTRODUCTION

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

VTT Technical Research Centre of Finland

Largest contract research
instute in Northern Europe Basic ES
S ITI
Producing research, research
IV ER
development, testing and
information services
UN
Serving public sector and Applied VTT
companies as well as research
international organizations IN
DU
ST
Figures RY
Development
2700 employees work
245 M turnover
Funding1/3 government,
1/3 industry, 1/3 jointly funded

2
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Virtual and Augmented Reality at VTT

General VR/AR expertise around VTT
14 research teams in VR/AR roadmap work
Applications from games to digital media,
construction to production, ships, cars, military,
Technology incl. core VR/AR solutions, sensors,
GSM/GPS positioning, multimodality, 3D audio,
motion tracking, ubi & ambient intelligence,
Special strengths: Multi-disciplinary research,
strong industry relations and experience

Augmented Reality team

Research personnel 9+ people
AR core technology & applications

AR at VTT situation today

AR driven projects > 2 M annually
+ VTT coordinated 10 M EU project ManuVAR
Overall, AR is now bigger than VR

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Introduction / AR
Augmented Reality (AR) Related technologies
Mixes virtual objects with view of real world Augmented Virtuality
C.f. Virtual Reality (completely virtual worlds) Mediated Reality
Properties: real-time, interactive, 3D Diminished Reality
Tracking + rendering + display -> Mixed Reality (MR)

Reality Augmented Reality Augmented Virtuality Virtual Reality

Mixed Reality

3
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Example Application
ARCatalogue
Viewing with data glasses / web camera
Stereo 3D products appear from catalogue
Based on marker detection from camera view

Applications, e.g.
B2B marketing (by Lappset Group )
3D viewing of engineering drawings (by Pyry Inc.)

Also, camera phones

Consumer solutions

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Dibidogs
3D animated chirdrens TV series
By Futurecode (Finland) and Blue Arc (China)
Launched April 2nd, 2010, MTV3 / SubTV Junior

Augmented magazine advertisement

TV-magazines Katso and TVSeiska, March 2010
3D animations appear augmented from magazine
Interaction with augmented Dibidog

4
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Dibidogs, Next Steps

Mobile phone implementation

Introduced at Sanghai World Expo, May 1st, 2010
Nokia Maemo (N900) and Symbian S60 platforms

Augmented childrens storybook

Kustannus Paasilinna, August 2010
Various new interactions and narratives
Markerless tracking, photorealistic rendering etc.

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ALVAR Subroutine Library

A Library for Virtual and Augmented Reality
AR engine developed by VTT
Free trial download at www.vtt.fi/multimedia/alvar.html
Users e.g. Columbia Univ. (Goblin XNA), MIT (6th Sense)

Detecting and tracking 2D markers

Accurate marker pose estimation, with erro concealing
Marker fields can be defined manually, or autocalibrated
Hiding of markers as sepcial feature

Utilities
Camera calibration; Correcting lense undistortions; Optical
flow tracking; Several basic filters and Kalman library;
Finding exterior orientation of point-sets

Coming up
Markerless tracking; Rendering module;
Mobile phone implementation

5
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ENTERTAINMENT AND GAMES

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Entertainment Application - CamBall

Augmented networked table tennis
Played with real rackets, color marker
Video image of opponent shown at other end
of table
Virtual ball, table and net

Features
Ball speed, racket size can be adjusted
Solo game against a wall, or against "oneself"
Game viewing by Internet audience
Recording of game

Benefits
Socially and physically activating
Natural human-computer user interaction
Fits in small interiors, at home or office
Suits also for kids and elderly people
No special hardware required

6
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Entertainment - SymBall

Virtual table tennis game for camera phones
Camera driven game control by feature/motion detection
Data connection between players via Bluetooth or GPRS
First networked camera driven game in the world

Implementation
Symbian OS/Series 60 (& J2ME)
Features: sounds, ball speed, etc.
3D stereo game using r/b goggles!

Future plan: Augmented version

Using turnable camera, face detection
Show real opponent on phone display

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Experimental Work TelePet

Augmented Networked Games for Dogs
To entertain dogs over Internet during masters absence
Game control based on gesture detection

Example game
Master throws virtual ball, dog has to catch
Rewards, e.g. biscuit box operated over internet
Initial animal tests - positive results

Internet

7
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Vappunaamari.com
Virtual masks for May Day party
Face detection/augmenting on camera phones
Provided as public service, > 1500 downloads
Features: send photo to friends, user created masks,
compressing augmented video,

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

INTERIOR DESIGN

8
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Interior Design Alternative Solutions

Basic solution
Virtual furniture object tied to each
marker
Interior designed by moving markers
on the floor
Display on PC screen or data glasses

Benefits
Furniture is light to move around
Easy to try out different alternatives
No modelling of real environment
required

Things to improve
Web camera isnt very mobile
Data glasses / PC .. not too
convenient

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmenting in Digital Camera Images

Application ARInteriors
Single marker placed on floor, defining
scale and perspective
Digital camera images of the room are
uploaded to computer
Virtual furniture is selected and placed
with menu/mouse
Features: lighting, shadows, hiding of
marker etc.

Benefits
Hardware (camera, PC) found in
every home
Better image quality, resolution,
accuracy etc.
Photorealistic 3D rendering

9
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmenting on TabletPC
Application ARMobile
Tablet PC (Sony Vaio), camera
integrated
Optional data glasses
User interface to fit small screen,
stylus input
No configuration files required
Arbitrary marker orientations
Marker fields calculated automatically
Moving, scaling and rotating objects
Adjusting of lights, with soft shadows
Erasing of markers
Storing of video clip & project
Kalman filter & other stabilisation
Feature based tracking when markers
disappear

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmenting on Mobile Phones

Find appealing
Application ARPhone furniture in ad

Placing virtual furniture on phones camera image

Models are loaded using matrix code in product catalogue or advert Download virtual
furniture via
User places the furniture in camera view by 3D rotation and zoom matrix code
Furniture stays in place when moving the camera phone
Based on feature tracking - no markers required
Instantly view
Also, augmenting of real furniture furniture at home
on mobile phone

Verify by high-end
PC solution (if
required)

Get to shop,
purchase product

10
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ARStudio
Unified solution for general AR applications
Formats: real time / stored video <-> still images
Single/multimarker tracking -> markerless methods
Matching augmented content with video quality
Photorealism, shadows, materials, lighting,

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

VividAR
AR interior design solution by VividWorks
Based on VTTs technology
Launched February 2010
Consumer web solution later this spring

11
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ARCHITECTURE AND CONSTRUCTION

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Reality in Architecture

Case Study: VTTs Digitalo offices
Digitalo = DigiBuilding
Construction started Dec 2003
Building ready Oct 2005

Various AR/VR methods used

Radiosity visualization
3D Landscapes
CAVE Virtual Reality
Augmented Scale Model
Augmented Web Camera
Mobile Augmented Reality

Projects involved
Commissions by Senate Properties
Project ARARAT 2004-2006

12
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Scale Model

Digitalo augmented on conference table
Viewing through video glasses (stereo)
Move around model, communicate with people

Features
Virtual menu for user interaction
Virtual reality tour utilising feature based tracking
Predefined walk-through routes, with hyperlinks
Realism: textures, reflections, shadows etc.
Add, move and remove parts (AR-catalogue)

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Web Camera

Digitalo augmented in live web camera view
Presentation for future users of building
Follow-up and comparison with original plans
View with or without virtual building
Real-time interaction (zoom, pan, etc.)
Viewing inside to the building, etc.

Ongoing development (project AR4BC)

Integrating 4D IFC models to live web cameras

13
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Augmented Reality

Outdoors visualization of Digitalo
Pioneering work by VTT 2003

Handheld display device

PDA + WLAN + camera
Lightweight, user friendly, inexpensive
Allows frozen view, multiple users, etc.

Client/server solution
Server laptop PC for 3D computation
Streaming video between blient and server
Speed 2 frames / sec

Outdoors tracking
Using large 80 x 80 cm markers
Mobility by moving markers around

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Unprepared Environments
Markerless implementation AROnSite
Feature based tracking
Interactive positioning of building to real scene
Visualization in any unprepared environment

Applications, e.g.
Helsinki Music Hall visualization 2006

14
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Google Earth on Earth

Google Earth + AROnSite
Building placed in Google Earth
Semi-automatic positioning using
GPS

Current applications, e.g.

Paper mill visualizations by Pyry
Excavator visualization by Metso
Skanska new Helsinkk offices

Future work
Automatic positioning (direction)
Photorealism: e.g. lighting,
shadows, masking
Etc., being implemented in project
AR4BC (see next slide)

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project AR4BC
Augmented Reality for Building and Construction
vertical representation of Finnish & international B&C sector
duration 2008 2010
budget 420 k

Physical model authors

Structural
Mechanical
Architectural
Electrical

Process model authors

Schedule
Classification
Logistics
Safety
Information users
View
Review
Comments
Approvals
Status
Feedback

Building Information Models (BIM)

Reality Augmented Reality Augmented Virtuality Virtual Reality

Mixed Reality

15
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Goals
Independent of Google Earth data formats (Collada)

Support for general/proprietary maps (cf. inaccuracy of GE maps)

Support for Building Information Models (e.g. IFC)

Support for time component, 4D (e.g. MS Project)

Support for several building models at a time

Photorealistic visualization (CG tuned for AR)

User interface (wxWidgets & new features)

Visual feedback to design system

Augmented Reality + Augmented Virtuality = Mixed Reality

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

INDUSTRIAL APPLICATIONS

16
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Power Plants
Remote Maintenance Tasks
ITER fusion reactor mock-up at Tampere
Maintenance robot augmented / animated
inside test platform
Application e.g. for discrepancy checking
(after accident or other changes)

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project NOSE
Nomadic Use of a Plant Model
2007-2009, budget > 700 k
Motto: Learning by Walking

Partners: Pyry, Metso, Wrtsil, Fortum,

Intergraph, Nokia,

17
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Simantics Mobile Studio

NOSE Implementation
Simantics Mobile
Mobile AR user interface
Studio for content creation
Mobile database & Internet

Simantics Mobile
Locationing technology
Pseudosatellites (GPS inside Pyry Virtual Mill
factory hall), WLAN, gait etc.
Model & template based tracking
Sensor fusion

Applications, e.g. VTT Knowpap

Mobile training
Mobile operation
Mobile maintenance

Simantics Mobile Metso DNA

Database OPC Server

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project AugAsse
Increasing Efficiency in Assembly Work by
Augmented Reality
Duration 2006-2008, budget 800+ k
Partners Valtra, Metso, Wrtstil. Kalmar. Sandvik,

Implementation phases
3D Puzzle, multimodal interaction
Camera phone implementation, client/server
Industrial case: hydraulic block by Valtra Plc
Incl. data pipeline CAD -> AR, user studies etc.

18
 VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Data Pipeline
Data pipeline from 3D CAD to augmented assembly
CATIA -> STEP -> Reduced B-rep -> Design for Assembly -> Animations -> Augmenting

Work Phase:23

Return tube

Tool:
27 mm wrench

s
ter s e
me u re tur
ara fig uc
n p ilit y st r
uctio bla b mb
ly
od m se
Pr asse As
d
an

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Phone Implementation

Client/server
easy to implement, general solution
tracking and other heavy computation can be done on server
no need to store large model data on mobile device
using subimages, no need for any 3D support on mobile device

Still images
in most cases, the assembly station is static
it is sufficient that the augmented content is live
it is more ergonomic to view the image without having to point at target
robust, provides good image size (accuracy) for tracking
enables general implementation, e.g. on Java ME

19
 VTT TECHNICAL RESEARCH CENTRE OF FINLAND

AugAsse User Study

Group Statistics

Std. Error
Instruction type N Mean Std. Deviation Mean
Assembling duration paper instructions 30 0:36:34 0:07:42 0:01:24
(h:mm:ss) augmented-reality
29 0:31:45 0:04:31 0:00:50
instructions
Observed frequency paper instructions 26 1,96 1,455 ,285
of using a wrong tool augmented-reality
25 ,32 ,627 ,125
instructions
Observed frequency paper instructions 26 3,54 1,964 ,385
of false positioning augmented-reality
25 1,48 1,873 ,375
instructions

Assembling is on average quicker with animated

instructions and results in less mistakes than with paper
instructions.
The usage of wrong tool and false positioning are on
average less frequent with augmented-reality instructions
than with paper ones.
A target for improvement is the HMD hardware. Operation
with the helmet and wires was often experienced
uncomfortable.

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project ManuVAR
Manual Work Support Throughout System
Lifecycle by VR/AR
EU FP7 NMP Beyond lean manufacturing
18 partners, 70+ participants, VTT as coordinator
Duration 2009-2012, budget 9.7M

The general objective is to develop

(1) technology platform and
(2) methodological framework

for supporting all aspects of manual work throughout the

system lifecycle with the ultimate goal to
1. increase productivity and quality and reduce cost
of manual work in the whole lifecycle
2. facilitate adaptation to product customization and
changes
3. support efficient knowledge and skill
management through the lifecycle

20
 VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ManuVAR Applications

Thales Alenia Space NEM Solutions TNO

Satellite assembly Remote training Manufacturing design

Industrial plant
maintenance
Tecnatom
Maintenance of
heavy machinery

Metso Minerals

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

RECENT PROJECTS: MIXED REALITY

21
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Mixed Reality

Co-creation project MMR with Nokia

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mixed Reality Video Conference

Teleconference btw real people and SL avatars

Avatars appear augmented in real room

Peoples gestures are replicated by avatars

Collaboration with virtual objects

Partners VTT, TKK, Nokia, IBM

22
VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Multitouch Interaction
Multitouch table developed by VTT

LCD display with cameras in the corners

Any flat surface can be used for multitouch interaction

Application example: connecting floor plan with CAVE,

navigation in 3D architectural model

At Sanghai World Expo 2010: multitouch in the air,

to control giant projector walls, tens of people

Collaborative work,
cf. Minority Report

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Thank You!

Dr. Charles Woodward

VTT Technical Research Centre of Finland
Vuorimiehentie 3, Espoo, Finland
Tel: +358 40 500 1514

charles.woodward@vtt.fi

http://www.vtt.fi/multimedia

23