Selection and

Manipulation in
Virtual Environment

GROUP 3
Sanyam Agarwal IIT2019129
Abhinav IIT2019098
Sangam Barnwal IIT2019094
Ritik Parmar IIT2019155
What is Selection
● Object selection is an important task in virtual environments and one of the most frequent
activities for most common user interactions. Tasks like moving or manipulating objects are
often preceded by a selection task.
● The amount of information available around selection methods within virtual environments
is rather extensive and describes in-depth the use of different interaction metaphors such
as voice, gaze, laser, and virtual hands. Surprisingly not much is available when looking at
MULTI-object selection methods.
● Creative tools are a great example as the action of selecting, moving and manipulating
multiple objects simultaneously are core activities the user performs to give shape to his
ideas and creative vision.

2
What is Manipulation
● In everyday language, manipulation usually refers to any act of handling physical objects
with one or two hands.
● For the practical purpose of designing and evaluating 3D manipulation techniques, we
narrow the definition of the manipulation task to spatial rigid object manipulation—that is,
manipulations that preserve the shape of objects.
● There are still many variations of manipulation tasks characterized by a multitude of
variables, such as application goals, object sizes, object shapes, the distance from objects
to the user, characteristics of the physical environment
● Designing and evaluating interaction techniques for every conceivable combination of these
variables is not feasible; instead, interaction techniques are usually developed to be used in
a representative subset of manipulation tasks. There are two basic approaches to choosing
this task subset: using a canonical set of manipulation tasks or using application-specific
manipulation tasks.
3
Canonical Manipulation Tasks
Each canonical task defines a task space that includes multiple variations of the same task
defined by task parameters— variables that influence user performance while accomplishing this
task.
Each of these parameters defines a design dimension, for which interaction techniques may or
may not provide support.

Tasks and their parameters 4

Application-Specific Manipulation Tasks
● The canonical tasks approach simplifies manipulation tasks to their most essential
properties. Because of this simplification, however, it may fail to capture some
manipulation task aspects that are application-specific.
● Examples of such application-specific manipulation activities include positioning of a
medical probe relative to virtual 3D models of internal organs in a VR medical training
application, moving the control stick of the virtual airplane in a flight simulator, and
exploring the intricacies of an object's surface such as a mountain range.
● in these examples, generalization of the manipulation task does not make sense—it is the
minute details of the manipulation that are important to capture and replicate.

5
Manipulation Techniques and Input
Devices
Just like input devices, visual display devices and their characteristics can significantly affect the
design of 3D manipulation techniques.
Properties of input devices
1. Control Dimensions and Integrated Control in 3D Manipulation
Two characteristics of input devices that are key in manipulation tasks are, first, the
number of control dimensions, and second, the integration of the control dimensions.
1. Force versus Position Control
Another key property of input devices that significantly affects the design of interaction
techniques is whether the device measures position or motion of the user’s hand, as
motion trackers and mice do (isomorphic control), or whether it measures the force applied
by the user, as joysticks do (elastic or isometric control
6
Manipulation Techniques and Input
Devices(cont)
3. Device Placement and Form Factor in 3D Manipulation
In 3D UIs, these objectives are also highly relevant; furthermore, the shape of the input
device strongly influences the choice of 3D manipulation interaction techniques.
Two popular device configurations used in 3D UIs are

Two strategies for handling the input device: attached to he hand (power grip) and
rolled in the fingers (precision grip). 7
Classifications for 3D Manipulation
Many 3D manipulation techniques relate to one another, and many share common properties.
Classifying them according to common features is useful in understanding the relations between
different groups of techniques and can help us to grasp a larger picture of the technique design
space.
there is more than one way to group techniques together, some of them are :-
Isomorphism in Manipulation Techniques
● The design of 3D manipulation interfaces has been strongly influenced by two opposing
views.
● The isomorphic view suggests a strict, geometrical, one-to-one correspondence between
hand motions in the physical and virtual worlds on the grounds that it is the most natural
and therefore is better for users.
● The isomorphic view suggests a strict, geometrical, one-to-one correspondence between
hand motions in the physical and virtual worlds on the grounds that it is the most natural
8
and therefore is better for users.
Classifications for 3D Manipulation
Classification by Task Decomposition
● 3D manipulation techniques as being constructed out of several “building blocks”
(technique components), where each block allows accomplishing a single suboperation.
● The advantage of this approach is that we can structure the interaction technique design
space so that new techniques can be constructed out of basic elements.
Classification by Task Decomposition
● Most current 3D UI manipulation techniques are based on a few basic interaction
metaphors or a combination of metaphors. Each of these metaphors forms the
fundamental mental model of a technique—a perceptual manifestation of what users can
do by using the techniques

9
 Grasping Metaphors

● We have identified two distinct approaches to providing the ability to grasp virtual
objects—hand-based techniques and finger-based techniques.
● Hand-based grasping techniques take a simplified approach to implementing grasping
interactions by representing the user’s virtual hand as a single-point effector (i.e., all
interaction calculations are based on a single transform that usually corresponds to the 6-DOF
position of the user’s physical hand).
● Finger-based grasping techniques take the more realistic, and much more complex, approach
of modeling the user’s virtual hand as a hierarchy of finger segments that are normally directly
controlled by the positions of the user’s physical fingers.

10
Hand-Based Grasping Techniques
▫ Hand-based techniques are the most common approach to implementing grasping metaphors
because most 3D UI systems do not support the ability to track the user’s physical fingers.
▫ However, many 3D UI systems employ at least one tracker to capture the position and orientation of
the user’s dominant hand. Hand-based grasping techniques use this 6-DOF information to provide a
virtual representation of the user’s physical hand within the 3D UI.
▫ A 3D cursor is used to visualize the current locus of user input; for example, the cursor can be a 3D
model of a human hand.
▫ These are two hand-based grasping techniques in this section: Simple virtual hand and Go-Go

11
Finger-Based Grasping Techniques
▫ Finger-based grasping techniques have received attention due to depth-based optical sensors and
physics-based computer vision techniques.
▫ By using such technologies or traditional motion capture systems, finger-based grasping techniques
allow the user to interact with and manipulate objects with more precision.
▫ A challenge of all these techniques, however, is that without haptic feedback, it may be difficult for
users to determine how to move their fingers to achieve the desired manipulation.
▫ The major difference among various finger-based grasping techniques is how they physically simulate
the fingers and interactions. Following are the finger-based grasping techniques: rigid-body fingers,
soft-body fingers, and god fingers

12
Pointing Metaphors
▫ The motivation behind the pointing technique is to allow the user to easily select and
manipulate objects located beyond the area of reach by simply pointing at them.
▫ If the vector defined by the direction of pointing intersects a virtual object, the user can
select it by issuing a trigger event that confirms the selection. Examples of triggers are
buttons and voice commands.
▫ Pointing, is generally a very poor positioning technique: object manipulation can be
efficiently accomplished only in radial movements around the user (perpendicular to the
pointing direction) when the task does not require changing the distance between the user
and objects.

13
Vector-Based Pointing Techniques
▫ Vector-based pointing techniques require only a vector in order to calculate what object the
user intends to select and manipulate. This makes vector-based pointing rather easy to
implement.
▫ The pointing vector technique is estimated from the direction of the virtual ray that is
attached to the user’s virtual hand and the 3D position of the virtual hand.
▫ As a result, these techniques are commonly used for pointing in 3D UIs.
▫ These are the following vector-based pointing techniques: ray-casting, fishing reel and
image-plane pointing.

14
Volume-Based Pointing Techniques

▫ Volume-based pointing techniques require the definition of a vector and a volume in order to
determine what the user intends to select and manipulate.
▫ The volume is defined in some relation to the vector, such as using a vector to define the axis of
a cone.
▫ However, in some cases, the vector is used to intersect an object, which then defines the
position of a volume given that intersection point.
▫ Following are the volume-based pointing examples: flashlight aperture selection sphere-casting

15
Surface Metaphors
▫ In the past decade, there has been a proliferation of multi-touch surfaces due to the
popularity of smartphones and tablets.
▫ As a result, the general population has become experienced with using touch gestures to
directly interact with virtual objects, usually in a 2D context.
▫ However, multi-touch surfaces can also be used to interact with 3D UIs. Hence, now we
distinguish between surface-based 2D interaction techniques and surface-based 3D
interaction techniques.

16
 Surface-Based 2D Interaction Techniques
Numerous interaction techniques have been investigated and developed for interacting with 2D
contexts on multi-touch displays and surfaces.
Techniques of surface-based 2D interaction techniques
1. Dragging
Dragging involves directly selecting and translating an object by touching it with one or more
fingers and then sliding them across the surface. The most common approach is to use a single
finger for this interaction.
1. Rotating
A number of surface-based 2D interaction techniques have been investigated for rotating virtual
objects.The most commonly used approach is an independent rotation that occurs about the
center of the object. When the object is touched within a specified area, usually its corners, the
user can drag his or her fingers across the surface to rotate the touched area about the center of
the virtual object.
17
Indirect Metaphors
In this article we discuss a number of interaction techniques that allow the user to manipulate
virtual objects without directly interacting with them. These techniques are referred to as indirect
metaphors. We have categorized indirect metaphors into three distinct approaches—control
spaces, proxies, and widgets. Widget techniques place within the virtual environment widgets
that the user can directly manipulate.

18
Indirect Control-Space Techniques
Many indirect control-space methods have been researched. A large number of these methods
offer a control area that is physically apart from the display area and virtual world. Some of the
solutions integrate a control space with the display area, but offset in order to offer distant
viewing avoid occlusions and interactions. This section, in particular, focuses on talk about the
following control-space methods:
▫ Indirect contact
▫ Interactive virtual surface
▫ Precision level cursor
▫ Digital pad

19
Indirect Proxy
Proxy techniques are interaction techniques that address many of the issues with remote object
manipulation (McMahan et al. 2014). By giving the user local proxies, or representations of
remote objects, proxy techniques allow the user to use a more natural grasping method to
perform direct manipulations. Any manipulations of these local proxies are in turn applied to the
remote objects that they represent, circumventing issues such as needing to travel to reach an
object or not being able to select an occluded object. We discuss two indirect proxy techniques in
this section:
▫ World in miniature
▫ Voodoo Dolls

20
World-in-Miniature : World-in-miniature (WIM) provides the user with a miniature handheld
model of the virtual environment, which is an exact copy of the VE at a smaller scale. WIM also
requires the careful use of back-face culling techniques to ensure that only the inside of the walls
of the room model should be rendered. Although WIM works relatively well for small and
medium-sized environments, using it in a very large environment would require an extreme
scalefactor, resulting in very small object copies in the WIM.

Voodo Dolls : Voodoo Dolls is a two-handed interaction technique that allows the user to
manipulate virtual objects indirectly using temporary, miniature,handheld copies of objects called
dolls. The doll that the user holds in her nondominant hand represents a stationary frame of
reference, and the corresponding virtual object does not move when moved. Voodoo Dollscan
scale the copies to a convenient size to interact with, overcoming one of the limitations of the
WIM technique, which lacks a mechanism for setting the scale of the world-in-miniature.

21
Indirect Widget Techniques
Another common indirect interaction metaphor is the use of widgets. Widgets are objects placed
within the virtual environment that the user can directly manipulate to indirectly affect other
virtual objects or contexts. Here we mention those widgets that afford indirect interactions,
including the following:
▫ 3D widgets
▫ virtual sphere
▫ Arcball

22
Bimanual Metaphors
We have already discussed a number of asymmetric bimanual techniques, such as balloon
selection, world-in-miniature, and Voodoo Dolls. However, we discuss two other asymmetric
interactions:
▫ Spindle + Wheel : Cho and Wartell (2015) extended the original Spindle technique to include
an additional feature for rotating the pitch of the selected virtual object.
▫ Flexible pointer : Ray-casting uses a virtual hand position and the direction of the virtual ray
to define the pointing direction. In cases where both hands are tracked, the direction can be
specified using a two-handed technique.

23
 Integrated Hybrid Techniques
In the simplest case of technique integration, the interface can simply switch from the selection
technique to the manipulation technique after the user selects an object, and switch back to the
selection mode after the user releases the manipulated object. We discuss the following ones:
▫ HOMER : It allows a user to easily reposition an object within the area between the virtual object
and himself, no matter how far away the object is at the moment of selection. As with the Go-Go
technique, the virtual hand (and. object) is placed along a line defined by the user's body and the
user's real hand, at the scaled distance.
▫ Scaled-World Grab : The scaled-world grab technique is based on principles similar to HOMER.
The scaling coefficient is calculated so that an object is brought within the user's area of reach
and,therefore, can be manipulated using the simple virtual hand technique.

24
Other Aspects of 3D Manipulation
● There are still other aspects of 3D manipulation that 3D UI designers should be concerned with.
These include non isomorphic 3D rotation mappings, multiple-object selection, and the
progressive refinement of selections.
● Non isomorphic 3D Rotation : The 3D interaction techniques discussed above deal only with
object selection and translation. In some applications, it might also be useful to design and
implement other mappings between the device and the object.
● Absolute and Relative 3D Rotation Mappings : Some interesting and nontrivial properties of 3D
rotation mappings must be taken into account when designing 3D rotation techniques. These
properties follow from the fundamental difference between the absolute and relative
amplifications of 3D device rotations.

25
Other Aspects of 3D Manipulation
● Absolute amplification : On each cycle of the simulation loop, we scale the absolute orientation of
the 3D device. Alternatively, we can amplify only relative changes in the device's orientation. We
calculate how much the device orientation has changed from the orientation that we measured in
the previous cycle. We amplify this difference between the current and previous orientation.
● Usability Properties of 3D Rotation Mappings : Unlike translations, relative and absolute mappings
of 3D rotations produce completely different rotation paths from the same 6-DOF device rotation.
Relative mappings always preserve the directional correspondence between rotations of the
device and virtual object. Absolute mappings have only limited applications in 3D interaction.

26
 Multiple-Object Selection
Some 3D UI tasks require multiple objects to be selected, such as grouping two objects together as
one. For example, it is faster to delete multiple objects from the virtual environment by selecting
multiple objects and invoking a single delete command, as opposed to selecting and deleting each
individual object. In this section, we mention the following multiple-object selection techniques:
▫ Serial selection mode.
▫ Volume-based selection techniques.
▫ Defining selection volumes.
▫ Selection-volume widget

27
 Progressive Refinement
Progressive refinement for 3D UIs involves gradually reducing the set of selectable objects until only
one target remains. Because multiple steps are used for a single selection, each step can use a fast
selection technique that does not require precision. Kopper identified three dimensions for
designingprogressive refinement techniques. There are several examples of progressive refinement
techniques for 3D UIs, we will discuss the following approaches in
this section:
▫ SQUAD
▫ Expand
▫ Double Bubble

28
 Design Guidelines
Manipulation is a rich user activity, and some interaction technique sare better for certain conditions of
manipulation than for others. The choices and design of manipulation techniques should provide high
levels of usability and performance for the most common common tasks in an application. When a high
level of manipulation precision is required, it might be preferable to use devices that permit a precision
grip and can be manipulated by the fingers. Clutching is wasted motion: by using techniques that
amplify the user hand motion in rotation and positioning tasks, we can decrease the need for clutching
and increase comfort and user performance. Most manipulation techniques depart from real-world
interaction to a greater or lesser degree by allowing "magic" interactions with virtual objects. Some
very effective selection techniques provide very effective interaction by restricting DOFs.

29
 Case Studies
VR Gaming Case Study : We have designed a bimanual interaction concept for selection,based on the
handheld controllers we selected in the input device section. The nondominant-hand controller acts as
a flashlight players can use to light up broad regions of the room around them, while the dominant
hand holds a tool that can be used for selection of a specific object. Since only objects illuminated by
the flashlight beam are selectable, one of them can be highlighted at all times, and movements of the
player's dominant hand don't have to be very precise. A flashlight and a tool can be used to select
objects from the world or inventory. A button press shoots an a sticky "frog's tongue" out of the tool,
which flies out to the object, sticks to it, then brings it back to the tool.
Since the frog tongue is unlikely to be effective against big scary monsters, the player may have to
choose a different tool, like a dart gun or a disintegration ray. We leave the design of the specifics to
your imagination.
Key Concepts :
▫ Progressive refinement selection techniques can help users avoid fatigue by not requiring precise
interactions.
▫ Basic 3D selection and manipulation techniques can be customized to fit the theme or story of a
particular application.
30
 Case Studies
Mobile AR Case Study : Selecting of system control items, labels and camera viewpoints in HYDROSYS
was strongly connected to the limitations of the input and output devices. In AR, you want to keep
menu items small so as to limit perceptual overlap with the real-world content. However, this comes
with a trade-off, as small items will be more difficult to select. This illustrates how performance and
user comfort often affect each other,leading to interesting trade-offs.
Key Concepts :
▫ Size of selectable items: keep the size of your selectable objects or menu items as small as
possible, while reflecting the limitations of your input method and the visibility (legibility) of these
items. Basic 3D selection and manipulation techniques can be customized to fit the theme or
story of a particular application.
▫ Selection method: depending on the frequency of selection tasks, different input methods could
be preferable. Often, there is a direct relationship between input method, selection performance
and frequency, and user comfort.

31
 Conclusion
we have discussed how 3D selection and manipulation techniques affect how users interact with 3D
UIs. We have presented six categories of manipulation metaphors, including grasping, pointing, surface,
indirect, bimanual, and hybrid techniques. Within these categories, we have covered a broad range of
3D manipulation techniques. Many are applicable to immersive 3D UIs, while some are specifically
designed for desktop or surface-based interactions. We have also discussed several design aspects of
3D manipulation techniques, including non isomorphic 3D rotations, selection of multiple objects at
once, and progressive refinement. We presented several design guidelines for choosing and designing
3D manipulation techniques

32
Thank You

33