Realism in Virtual Reality

Elements for Realism in Virtual Reality

 Visual
 Auditory
 Tactile
 Smell
 Taste
 Visual
How to make things look ”real” in the virtual world?
How to Make Things Look “Real”
in the Virtual World?
 Fundamental question in Computer Graphics
How to Make Things Look “Real”
in the Virtual World?
 Fundamental question in Computer Graphics
 Modeling
(a) (b)

Which one is more realistic?

How to Make Things Look “Real”
in the Virtual World?
 Fundamental question in Computer Graphics
Motion retargeting
 Modeling
 Animation

Source: https://vimeo.com/72803884
How to Make Things Look “Real”
in the Virtual World?
 Fundamental question in Computer Graphics
 Modeling
 Animation
 Rendering Direct illumination Global (indirect) illumination

Images by Henrik Wann Jensen

How to Make Things Look “Immersively Real”?
 Human visual system: binocular vision
 Disparity:

 Informally: difference between 2 images

 Allows us to gain a sense of depth
 Stereopsis:

 Ability to perceive depth from disparity

 Render a pair of images!
How to Make Things Look “Immersively Real”?
 Human visual system: binocular vision
 Disparity:

 Informally: difference between 2 images

 Allows us to gain a sense of depth
 Stereopsis:

 Ability to perceive depth from disparity

 Render a pair of images!

Source: https://www.vision3d.com/stereo.html
And…
 Frame rate!

Source: https://youtu.be/pRWKVYbZtTM
Modeling
 Creating 3D content in the virtual scene
 Realistic: appearance and functional

How?
Representing Surfaces: Triangle Meshes
 Triangle represented by 3 vertices
 Surface analogue of polylines, this is what
GPUs draw
 Pro: simple, can be rendered directly
 Cons: not smooth, needs many triangles to
approximate smooth surfaces (tessellation)
Add Details to the Surface
 Common techniques
 Subdivision surface
 Displacement mapping
Subdivision Surfaces
 Start with polygonal mesh
 Subdivide into larger number of polygons,
smooth result after each subdivision
Lots of ways to do this.
 The limit surface is smooth!
Cool: Displacement Mapping
 Not all surfaces are smooth...
 “Paint” displacements on a smooth surface

For example, in the direction of normal

 Tessellate smooth patch into fine grid,
then add displacement D(u,v) to vertices
 Heavily used in movies, more and more in games

Control mesh Tessellated smooth patch

Cool: Displacement Mapping

Image from https://commons.wikimedia.org/wiki/File:Displacement_Mapping.jpg

Displacement Mapping Example

Low-res mesh Smooth base surface Displaced surface

Image from https://www.zbrushcentral.com/t/rendering-zbrush-displacement-maps-in-external-renderers-may-9th-2nd-file-added/204576

Realism
 Appearance v.s. Functional

or
Scene Modeling
 Assume individual object is well modeled,
how to put them together?
Scene Modeling
 Assume individual object is well modeled,
how to put them together?
 Use data as reference

…
Data-Driven Optimization For Scene Modeling

Make it Home: Automatic Optimization of Furniture Arrangement.

Lap-Fai Yu, Sai-Kit Yeung, Chi-Keung Tang, Demetri Terzopoulos, Tony F. Chan, Stanley J. Osher
ACM Transactions on Graphics (Proceeding of SIGGRAPH 2011).
Rendering
 Render 3D content in the virtual scene
realistically
 Speed v.s. Quality

George Borshukov. 2003. Measured BRDF in film production: realistic cloth

appearance for "The Matrix Reloaded". In ACM SIGGRAPH 2003 Sketches &
Applications (SIGGRAPH '03)
What is Rendering?
 “Rendering” refers to the entire
process that produces color values for pixels,
given a 3D representation of the scene
What is Rendering?
 “Rendering” refers to the entire
process that produces color values for pixels,
given a 3D representation of the scene

For each pixel, answer this

question:
What color should it be?
-What object does it see?
-Object properties, lighting, etc.
Rendering Algorithms
 Two major algorithms
 Ray casting (tracing): slow, photorealistic.
Good for static architecture showcases.
 Rasterization: fast, less realistic. Good for
gaming and interactive applications
 Both are useful for VR applications.

Source: http://www.digitalmasters3d.com/
Rendering Algorithms
How to Make Realistic Rendering
 Input for realistic rendering
 Geometry (from modeling), Lighting and Materials
 Material appearance

 Intensity and shape of highlights

 Glossiness
 Color
 Spatial variation, i.e., texture
How to Make Realistic Rendering
 Use different “maps”
 Most common:

 Texture mapping
 Normal mapping
Effect of Textures
Normal Map
Normal Map Example Paolo Cignoni

Original Mesh
4M triangles
 Normal Map Example Paolo Cignoni

Simplified mesh,
500 triangles

Simplified mesh +
normal mapping
Normal Map Example

Normal Map

Diffuse texture kd

Final render
 Rendering for VR
 360 panorama (omni-directional
rendering)
 Both eyes see the “same image”
 Rendering for VR
 360 panorama (omni-directional
rendering)
 Both eyes see the “same image”
 Rendering for VR
 360 panorama (omni-directional
rendering)
 Both eyes see the “same image”

 3D 360 panorama (omni-directional

stereoscopic rendering)
 Left and right eyes see slightly
different image for the “3D effect”
Rendering for VR
Frame Rate?
 How fast you can render ONE frame?
 Image resolution, i.e., # of pixels
HTC Vive Game: SUPERHOT VR
 Scene complexity,
e.g., how many objects?
 Object complexity,
e.g., how detail is each object?
What material is it (transparent/reflective
objects take more time)?
 VR applications require high frame rate to
avoid dizziness, minimum 60 fps, typical 120 fps
Auditory
 Generate 3D/Spatial Audio
Auditory

https://youtu.be/IYdx9cnHN8I
Auditory

https://youtu.be/W9MHHZiIjxU
Tactile
 Haptic suit

https://www.youtube.com/watch?v=O5peqC1tZnM
Tactile
 Artificial skin

https://www.youtube.com/watch?v=Cv2ha_Fmliw
Smell
 Create smell according to the content

https://feelreal.com/
Taste
 More difficult than smell

Source: http://cutecenter.nus.edu.sg/projects/virtual-cocktail.html
Taste
 More difficult than smell

Source: https://www.youtube.com/watch?v=Ii1HHNZmYI4
Taste
 More difficult than smell

Digital Lollipop Study on Control Method of Virtual Food Texture by

https://dl.acm.org/citation.cfm?id=2996462 Electrical Muscle Stimulation
https://dl.acm.org/citation.cfm?id=2984768
VR Applications:
Simulation And Training – Military

https://youtu.be/V34gCw4fyLs
Simulation and Training – NASA

https://youtu.be/WOPmOJSfAXA
Visualization And Entertainment
Visualization and Entertainment

https://www.youtube.com/watch?v=2IASj7qvCjg
Clinical Virtual Reality

https://www.youtube.com/watch?v=lzbBT6ZQm5s
Summary
 5 Senses
 Visual
 Auditory
 Tactile
 Smell
 Taste
 Example applications