Commonwealth of Australia

Copyright Act 1968

Notice for paragraph 135ZXA (a) of the Copyright Act 1968

Warning

This material has been reproduced and communicated to you by or on behalf of

Federation University Australia under Part VB of the Copyright Act 1968 (the Act).

The material in this communication may be subject to copyright under the Act.
Any further reproduction or communication of this material by you may be the subject of
copyright protection under the Act.

Do not remove this notice.

Computer Graphics and
Interactive Systems
In this lecture
2D Graphics
Raster, Vector
3D Graphics
Ray tracing, rasterization
GPUs
Virtual Reality
Ethics - Digital characters and Game Addiction

3
Abstraction

C.P.U.
Central Internal and external interfaces Network
PCI, PCI Express, USB Ethernet, WiFi,
Processing Unit
Does the "thinking" Bluetooth, 4G

R.A.M. G.P.U Storage

Random Access Graphics Hard Disks, Flash
Memory Processing Unit drives
"Volatile" storage - goes
Connects to a monitor Non-volatile
away when the power dies

4
Abstraction

C.P.U.
Central Internal and external interfaces Network
PCI Express Ethernet, WiFi,
Processing Unit
Does the "thinking" Bluetooth, 4G

R.A.M. G.P.U Storage

Random Access Graphics Hard Disks, Flash
Memory Processing Unit drives
"Volatile" storage - goes
Connects to a monitor Non-volatile
away when the power dies

5
Abstraction

C.P.U.
Central Internal and external interfaces
Processing Unit PCI Express
Does the "thinking"

R.A.M. G.P.U
Random Access Graphics
Memory Processing Unit
"Volatile" storage - goes
Connects to a monitor
away when the power dies

6
Computer graphics is the generation of
graphics by a computer
This includes:

- Generating text and images on screen

user interface, games

- Generating graphics for other media

such as print or film
We won't spend much time on graphics recorded
and stored in computers
such as digital photography or video

7
Two-dimensional Graphics

8
Raster graphics
By far, the most common type of display is a
raster display
from the latin for "rake"
A raster display is made up of a big grid of
pixels, each of which has a colour
Short for picture elements
Ideally, each pixel is too small to notice on its
own

9
 Pixels!

The Mona Lisa

Leonardo da Vinci, 1503 10
 Each pixel has a
single colour
How do we represent
colours?

The Mona Lisa

Leonardo da Vinci, 1503 11
Colour
frequency in Hz
Visible Light

104 106 108 1010 1012 1014 1016 1018 1020

(longer waves) (shorter waves)

am radio microwave ultraviolet gamma rays

fm radio, tv infrared x-rays

Light is an electromagnetic phenomenon, like radio and x-rays.

When we talk about light, we usually mean the visible spectrum.
Our eyes can detect waves in this narrow band of frequencies.
12
Perception
Displays are designed to be perceived by
humans - so display technology is almost
always tied to human physiology and
psychology
the physical systems that detect colour, and the
systems in the brain which interpret colour and
images
There's no point making displays and
representing colours which humans can't see
13
Light is focused
by the lens
through the
pupil
and onto the
retina, which
detects light.
the optic nerve
sends messages
from the retina to
the brain

14
Detecting colour in the eye
The retina is packed with cells called rods and
cones, which respond to different frequencies
of light
rods are only used for seeing at low light levels - regular
colour vision is handled by cones
Cones come in three types: Short, Medium
and Long
for the relative wavelength of light they respond to

15
Detecting colour in the eye
This is an approximation of the
distribution of cones on the retina

Most of the cones are Long and

Medium, which respond most
strongly to Red and Green
wavelengths, respectively
About 7% are Short, which
respond most strongly to blue
light
Adapted from http://www.cis.rit.edu/people/faculty/montag/vandplite/pages/chap_9/ch9p1.html

16
Each type of cell
responds to different
ranges of light in the
visible spectrum

These curves are

normalized, in reality
the cells in the eye is
more sensitive to
green (but your brain
compensates) http://cnx.org/contents/FPtK1zmh@8.80:s3XqfSLV@8/Sensory-Perception

17
Light that appears
yellow stimulates
both the red-
sensitive Long
cones, and the
green-sensitive
Medium cones

These curves are slightly

different for everybody

http://cnx.org/contents/FPtK1zmh@8.80:s3XqfSLV@8/Sensory-Perception

18
Yellow light
stimulates both the
Long and Medium
cones

Yellow
Short

Medium
Long

http://cnx.org/contents/FPtK1zmh@8.80:s3XqfSLV@8/Sensory-Perception

19
If we mix light of two
different wavelengths
we can get the same
ratio

Yellow
Short

Medium
Long

http://cnx.org/contents/FPtK1zmh@8.80:s3XqfSLV@8/Sensory-Perception

20
Colour blindness
In Australia, about 8% of men (and < 1% of
women) have some form of colour blindness
The most common form, deuteranomaly, is due to
the M cones (most sensitive to green)
overlapping significantly with the L cones
this is commonly called red-green colour blindness,
because red and green look very similar

As a designer, it's important to ensure you don't

rely on colour to communicate information
21
RGB colour
Colour displays mix three colours to trick our
eyes into thinking we're seeing lots of colours:

Red Green Blue

These are the so-called primary colours for light
also called additive colours

By changing the relative levels of red, green and

blue, we can simulate the colours that humans
can see
22
 On an LCD, Plasma or
OLED screen, each pixel
appears to have a single
colour, but actually has
three subpixels, one for
each primary colour

The Mona Lisa

Leonardo da Vinci, 1503 23
Representing colour
The most common way to represent colours in
a computer system is using 8-bit numbers for
each Red, Green and Blue value
0 = Off
255 = Full brightness

24
Some common colours,
Black
expressed as RGB 0, 0, 0

Red Green Blue

255, 0, 0 0, 255, 0 0, 0, 255

Yellow Cyan Magenta

255, 255, 0 0, 255, 255 255, 0, 255

Orange White Hot Pink

255, 128, 0 255, 255, 255 255, 105, 180
25
Images
Now that we know how to represent a colour
using 24 bits (8-bit RGB values), an image
made of pixels can be represented as a big
array of 24-bit values
plus some extras, like the width and height of the
image
Images are usually stored like this in RAM or on
the GPU for display
but often compressed on disk

26
Resolution
The resolution of an image can mean two things:
• the number of pixels in an image or screen
for example, a projector might have a resolution of 1280 x 720
pixels - 1280 pixels wide and 720 high
A camera might take photos with a resolution of
4000 x 3000, or in marketing speak 12 megapixels
• Less commonly, the density of pixels in real-world
units
For example, an image might have 40 pixels/cm
This is important for printing images - insufficient resolution makes
prints blurry

27
Resolution names
Various standards have given labels to
particular resolutions. The HD labelling is from
the television industry
VGA (Video Graphics Array) 640 x 480
SVGA (Super VGA) 800 x 600
HD 1,280 x 720
Full HD 1,920 x 1,080
QHD 2,560 x 1,440
"4K" UHD 3,840 x 2,160

28
Image size
If
  we want to work with uncompressed images, how
big is an image in bytes?
Disregarding extra book-keeping, its just

At 3 bytes per pixel, a UHD image requires

That's quite a lot of bytes

29
Limitations of raster images
Raster images are:
• large, even compressed
• difficult to resize
for example to show a low-resolution image on a
high-resolution screen

30
Vector graphics
An alternative way to represent graphics is
using vector graphics
Consider this triangle
One way to represent this
is as a raster image

31
Triangle as a raster image
 
This is a 30 x 30 pixel image,
which looks fine!
Its size is
But what happens when we zoom
in?

It's either blurry or pixelated!

32
 NO
Can you zoom and
enhance that?
Law & Order : SVU
Triangle as a vector shape
This triangle is just three points, the lines between them,
and a colour - why not represent like that?
Store a position for each point
x and y, could store in 2 bytes each
(4 bytes per position)
Store a colour
3 bytes, RGB
Perhaps store what type of shape (if required)
1 byte
That's 16 bytes to represent the same shape

34
Vectors
Not only is the vector
representation smaller, it
scales perfectly to larger
sizes

35
What's the catch?
Vector images are great for shapes, but are not
good at representing photographs
It can be done, but usually the vector version takes up
more space for complicated photographs
Vectors are also more work for the CPU or
GPU
it has to convert the shapes into raster images
(rasterize them) to present them on a display

36
Fonts
Modern fonts are almost always specified as vector
graphics, with some hints about how to rasterize them
at smaller sizes

AA AA AA A
37
Three-dimensional Graphics

38
3D graphics
The idea behind 3D graphics is to generate
images that appear to have three dimensions
they are still 2D images, but the maths to generate
them is 3D

In the film and television industry, 3D graphics

are commonly called CGI
Computer Generated Images

39
 This is
Starfox (1993)
for the Super
Nintendo

Note the illusion

of depth which
makes it appear
three
dimensional
40
Rendering
There are a few approaches to creating 3D
graphics, but the two most popular are:
• Ray tracing
Used in almost every modern film
• Rasterization
Used in almost every modern game

41
Ray tracing
Ray tracing is relatively simple in concept
Shoot rays out from lights, and figure out how they
arrive at a virtual camera
In practice, most of the rays from a light won't hit the
camera, so we do it backwards
Shoot rays out from the camera through each pixel,
and work out how light might have arrived at that
pixel
For certain effects, we might do a bit of both

42
Ray tracing

43
Ray tracing
Ray tracing is used in film and television
• Simulates the reality of light transport, and so can
produce very realistic images
Can trade off reality vs computer time
• Handles reflection, refraction and other physical
effects
• Can be sloooow
An image might require hundreds to thousands of rays per
pixel, depending on the complexity of the scene

44
Piper (2016), a Pixar animated short film
Every single grain of sand is included in the ray trace – and each contains about 5000 polygons
45
Rasterization
Ray tracing is too slow for real-time applications
such as games, architectural visualization, editors
Most real-time 3D graphics applications use
rasterization
Modern rasterization is about drawing
triangles, very, very quickly
Rasterizers used to also draw lines, points and
quadrilaterals, which are now drawn using triangles

46
Representing objects with triangles
Each vertex (corner)
of each triangle is a
three-dimensional
point - it has an
x, y, and z coordinate

47
Rasterization
Whereas ray tracing
sends virtual light rays
through the scene,
rasterization is about
finding which pixels
are inside a triangle

48
Rasterization
The simplest is to
determine which pixels
are part of the triangle
by sampling at the
pixel centre
If the centre is inside
the triangle, then
colour it in

49
Rasterization
Which gives these pixels
.. doesn't look much like a
triangle though
Where did our nice straight lines go?
And wasn't that line on the right on
an angle?
This phenomenon is called
aliasing, and is sometimes
informally called jaggies or
stairstepping

50
Antialiasing
Aliasing is what happens when you try to
represent a signal, but don't (or can't) take
enough samples to capture all the high-
frequency details
you get new and weird shapes
Antialiasing algorithms try to remove aliasing
in various ways, the most common is to take
more samples and average them

51
Rasterization
Let's try four sample
locations per pixel, and
average them out

That means we have

five possible amounts
of coverage
None, 1, 2, 3, all covered

52
Rasterization
Which gives these
pixels

still doesn't look much

like a triangle, but
getting closer

53
Rasterization
With more samples,
we can get pretty close
to a "triangle“
Usually, each pixel in
the image is mapped
to a single pixel on the
display, so this would
look more like…

54
Making it pretty
Besides actually working out which pixels a triangle
is inside of, a rasterizer also performs:
• Depth testing
don't draw distant objects over nearer objects
• Texture mapping
like gluing wallpaper to each triangle
• Vertex and pixel shading
running small programs for every vertex and every
rendered pixel

55
Whoa, this is getting complicated, maybe it's
too much for a CPU…

56
An NVIDIA GTX 1080 GPU
Obscenely fast at rasterizing
lots and lots of triangles

57
Graphics Processing Unit
A modern GPU is able to rasterize triangles
significantly faster than a general-purpose CPU
Complicated slightly because some CPUs have an
embedded GPU

Rasterizing triangles is a parallel task - the same

work can be done on multiple pixels simultaneously
so it can be sped up using multiple processors
The NVIDIA 1080 on the previous slide has 2560
"shader units", which are like little CPUs

58
 This is
Assassin’s
Creed
Odyssey
(2018)
for PC and consoles
This is rendered in
real-time, at 60 Hz

59
Hardware raytracing
Like rendering triangles, ray-tracing is
embarrassingly parallel
The latest innovation in hardware accelerated
graphics is Real-time Raytracing
See https://www.youtube.com/watch?v=lMSuGoYcT3s
for an example
First-generation RTR products are currently
available, but pretty expensive
From about A$600 to A$2500

60
Virtual Reality
One technology which is having a renaissance is virtual
reality, or VR
VR is the convergence of several technological advances:
• Fast CPUs and GPUs
thanks to the steady advance of Moore's law
• High resolution smartphone displays
Sony have a 4K phone, and Samsung have 4K screens ready to go
• Computer vision algorithms and techniques
for detecting orientation of position

61
Virtual Reality
VR renders a slightly different view of a 3D
scene to each eye
For people with normal vision, this gives very
strong depth cues
Not all of them! We also get depth cues from the focus
length of our lens, and tiny changes as our eyes jitter
around. VR can't do these, yet.

To be effective, it must also accurately track the

location of your head
62
If it's so good, why did it fail in the 90s?
A few very good reasons:
• VR needs high frame rate - at least 90 frames per
second
90's VR worked at a cool 12-15 fps
• VR needs low latency - less than 10ms between
movement and seeing the change
90's VR was 100s of milliseconds - instant motion sickness
• VR needs high resolution, lightweight screens
Even in modern VR it's difficult to read text. In 90's VR pixels
were much larger

63
 90's VR
The Forte VFX1
headset had two
screens with a
resolution of
263x230 pixels
It had very limited
field of view
like "looking through
toilet paper tubes"

64
How did we get here?
Arguably the biggest leap in computer graphics
was in 1963, with the creation of Ivan
Sutherland's Sketchpad
This is simultaneously the best and most
demoralizing PhD research ever – it’s hard to
innovate this much in a single research project
anymore!

65
Sketchpad was
the first graphical
user interface
(GUI)
It used a vector
(not raster)
display, and a
light pen to
interact with the
objects on screen
66
In the late 1970's,
Xerox created the
Alto, and its
commercialized
version, the Star, the
first GUIs to use the
mouse, icon, and
menu approach that
is still familiar today
Apple and Microsoft
built on their good ideas 67
What next?
Other display technologies are also emerging
Augmented reality aims to overlay content over
the real world - augmenting or adding to reality
Microsoft's HoloLens is an in-development project
which does this
Mixed reality aims to blend virtual objects into
real environments
Much harder! A company called Magic Leap are
rumoured to have a solution to this

68
What else is next?
Stimulating other senses
• VR lets us stimulate our visual system, our auditory
system and interacts with our prioperception
Prioperception is knowing where your body is - VR can replicate
parts of your body in the virtual world
• The vestibular system can be stimulated using
electrodes
responsible for balance and equilibrium, to give the sensation of
movement
• Touch, temperature, and smell are open problems

69
Ethics
Two ethical problems this week!

70
In the 2017 film, Blade
Runner 2049, the character
Rachel returned from the
original Blade Runner
Here she is in the 1982
original, played by Sean
Young

71
And here she is again, in
2017, played by a digital
recreation of 1982 Sean
Young
She was paid for the use of
her likeness, but a different
actor recorded the facial
performance

https://vimeo.com/249369342 72
In the 2016 Star Wars
film, Rogue One, the
special effects team
created a virtual clone
of Peter Cushing, who
died in 1994
He played Grand Moff Tarkin

Here he is in 1977's
original Star Wars
73
And here's the computer generated likeness
74
Digital characters
Is it ok to digitally resurrect dead people to act in
new films?
• Peter Cushing's estate approved of the
recreation - but Peter didn't
$$$
• At what point should a person's likeness enter
the public domain?
i.e. no copyright. There is a legal battle over whether
Marilyn Monroe's image is public domain

75
Ethics problem 2
We regulate many addictive chemicals and
activities through law, social programs and
education
alcohol, cigarettes, opioids, and gambling
Computer gaming addiction is a genuine
phenomenon, particularly for online multiplayer
games like MMORPGs

76
Computer game addiction
As VR approaches the mainstream, the risks of
addiction may increase

Should we legally regulate addictive

computer games in the same way as alcohol
and tobacco?

77
Next lecture
Ubiquitous, Personal and Assistive Computing

78
Questions?