PAPER PRESENTATION

ON

• SENSE
BY

B. Sai Sushmitha

07251A0565

1
 CONTENTS :
• ABSTRACT

• INTRODUCTION

• HISTORY OF SIXTH SENSE TECHNOLOGY

• WORKING PRINCIPLE

• APPLICATIONS

• SOCIETAL ISSUES

• CONCLUSION

• REFERENCES

2
ABSTRACT:

'Sixth Sense' is a wearable gestural interface that augments the physical world around us with
digital information and lets us use natural hand gestures to interact with that information. It is
a sense that would give seamless and easy access to Meta information or information that
may exist somewhere. The Sixth Sense prototype is comprised of a pocket projector, a mirror
and a camera. The hardware components are coupled in a pendant like mobile wearable
device. Both the projector and the camera are connected to the mobile computing device in
the

user’s pocket. The projector projects visual information enabling surfaces, walls and physical
objects around us to be used as interfaces. ‘Sixth Sense’ frees information from its confines
by seamlessly integrating it with reality, and thus making the entire world your computer.
Integrating information to our everyday objects will not only help us to get rid of the gap
between the physical world and digital world, but it will also help us in some way to stay
human, to be more connected to our physical world. The goal is to bring a part of the physical

3
world to digital world to make computing devices more computing devices more intuitive.
The reverse process is also made possible. In this paper, we explained the sixth sense device,
its working and the various applications that demonstrate the usefulness, viability and
flexibility of the system also providing information about its availability and adoptability.

INTRODUCTION:

We've evolved over millions of years to sense the world around us. When we encounter
something, someone or some place, we use our five natural senses to perceive information
about it; that information helps us make decisions and chose the right actions to take. But
arguably the most useful information that can help us make the right decision is not naturally
perceivable with our five senses, namely the data, information and knowledge that mankind
has accumulated about everything and which is increasingly all available online. Although
the miniaturization of computing devices allows us to carry computers in our pockets,
keeping us continually connected to the digital world, there is no link between our digital
devices and our interactions with the physical world. Information is confined traditionally on
paper or digitally on a screen. Most of the people have some portable technology like smart
phones, laptops, e-book and iPods. But you need to carry all the stuff with you. We grew up
interacting with the physical objects around us. And there are enormous numbers of them that
we use every day. Unlike our most computing devices these objects are much more fun to
use. When you talk about objects one another thing that automatically comes to that thing is
gestures how we manipulate the objects. We use gestures not only to interact with each other
but also to interact with the physical world around us.

SIXTH SENSE
DEVICE

4
WHY THE NAME ‘SIXTH SENSE’:

From thousands of years mankind used the five senses to gain information. But now as this
device is completely portable and it can provide information about anything, it

is named as ‘SIXTH SENSE DEVICE’.

HISTORY:

The exploration of this sixth sense technology started with a question why can’t we
use a computer to interact with physical world.This literally started with amouse that comes
with a ball in it and TWO ROLLERS that actually guides the movement.So taking out rollers
from another mouse totally we have four rollers.Using these a system interface device was
formed.With this whatever you do in physical world will be reflected in the digital world.

With this initially a pen to draw 3-dimensions in digital world was made that can help
designers and architects not only think but they can actually draw so that it is more intuitive
to use that way. The goal is to bring part of physical world to digital world to make
computing devices more intuitive. Not only this, the reverse process is made possible.

5
 Initially the device started with a helmet consisting of the camera, projector and color
markers. Later it was converted into a form so that it is easy to use it by wearing around the
neck.

Sixth Sense bridges this gap, bringing intangible, digital information out into the tangible
world, and allowing us to interact with this information via natural hand gestures. ‘Sixth
Sense’ frees information from its confines by seamlessly integrating it with reality, and thus
making the entire world your computer.

WORKING PRINCIPLE:

The Sixth Sense prototype is comprised of a pocket projector, a mirror and a camera.
The hardware components are coupled in a pendant like mobile wearable device. Both the
projector and the camera are connected to the mobile computing device in the user’s pocket.

The projector projects visual information enabling surfaces, walls and physical
objects around us to be used as interfaces; while the camera recognizes and tracks user's hand
gestures and physical objects using computer-vision based techniques. The software program
processes the video stream data captured by the camera and tracks the locations of the

6
colored markers (visual tracking fiducially) at the tip of the user’s fingers using simple
computer-vision techniques. The movements and arrangements of these fiducially are
interpreted into gestures that act as interaction instructions for the projected application
interfaces. The maximum number of tracked fingers is only constrained by the number of
unique fiducially, thus Sixth Sense also supports multi-touch and multi-user interaction.

CPU & MEMORY WITH 3G

PROJECTOR CAMERA

TECHNOLOGY USED:

Gesture Recognition:
Gesture recognition is a topic in computer science and language technology with the goal of
interpreting human gestures via mathematical algorithms. Gestures can originate from any
bodily motion or state but commonly originate from the face or hand. Current focuses in the
field include emotion recognition from the face and hand gesture recognition. Many
approaches have been made using cameras and computer vision algorithms to interpret sign
language. Gestures can exist in isolation or involve external objects. Free of any object, we
wave, beckon, fend off, and to a greater or lesser degree (depending on training) make use of
more formal sign languages. With respect to objects, we have a broad range of gestures that
are almost universal, including pointing at objects, touching or moving objects, changing
object shape, activating objects such as controls, or handing objects to others.

Gesture recognition can be seen as a way for computers to begin to understand human body
language, thus building a richer bridge between machines and humans than primitive text
user interfaces or even GUIs (graphical user interfaces), which still limit the majority of input

7
to keyboard and mouse. Gesture recognition enables humans to interface with the machine
(HMI) and interact naturally without any mechanical devices.
Gestures can be used to communicate with a computer so we will be mostly concerned with
empty handed semiotic gestures. These can further be categorized according to their
functionality.

Symbolic gestures:

These are gestures that, within each culture, have come to a single meaning. An Emblem such
as the “OK” gesture is one such example, however American Sign Language gestures also
fall into this category.

Deictic gestures:

These are the types of gestures most generally seen in HCI and are the gestures of pointing,
or otherwise directing the listeners attention to specific event or objects in the environment.

Iconic gestures:

As the name suggests, these gestures are used to convey information about the size, shape or
orientation of the object of discourse. They are the gestures made when someone says “The
plane flew like this”, while moving their hand through the air like the flight path of the
aircraft.

Pantomimic gestures:

These are the gestures typically used in showing the use of movement of some invisible tool
or object in the speaker’s hand. When a speaker says “I turned the steering wheel hard to the
left”, while mimicking the action of turning a wheel with both hands, they are making a
pantomimic gesture.

Using the concept of gesture recognition, it is possible to point a finger at the computer
screen so that the cursor will move accordingly. This could potentially make conventional
input devices such as mouse, keyboards and even touch-screens redundant. Gesture

8
recognition can be conducted with techniques from computer vision and image processing.
The literature includes ongoing work in the computer vision field on capturing gestures or
more general human pose and movements by cameras connected to a computer.

Computer vision:

Computer vision is the science and technology of machines that see. As a scientific
discipline, computer vision is concerned with the theory behind artificial systems that extract
information from images. The image data can take many forms, such as video sequences,
views from multiple cameras, or multi-dimensional data from a medical scanner.
Computer vision, on the other hand, studies and describes the processes implemented in
software and hardware behind artificial vision systems. The software tracks the user’s
gestures using computer-vision based algorithms. Computer vision is, in some ways, the
inverse of computer graphics. While computer graphics produces image data from 3D
models, computer vision often produces 3D models from image data. There is also a trend
towards a combination of the two disciplines, e.g., as explored in augmented reality.
The fields most closely related to computer vision are image processing, image analysis and
machine vision. Image processing and image analysis tend to focus on 2D images, how to
transform one image to another. his characterization implies that image processing/analysis
neither require assumptions nor produce interpretations about the image content. Computer
vision tends to focus on the 3D scene projected onto one or several images, e.g., how to
reconstruct structure or other information about the 3D scene from one or several images.
Machine vision tends to focus on applications, mainly in manufacturing, e.g., vision based
autonomous robots and systems for vision based inspection or measurement.

The Recognition Algorithms

The computer vision system for tracking and recognizing the hand postures that control the
menus is based on a combination of multi-scale color feature detection, view based
hierarchical hand models and particle filtering. The hand postures or states are represented in
terms of hierarchies of multi-scale color image features at different scales, with qualitative
inter-relations in terms of scale, position and orientation. In each image, detection of
multiscale color features is performed. The hand postures are then simultaneously detected
and tracked using particle filtering, with an extension of layered sampling referred to as
hierarchical layered sampling. To improve the performance of the system, a prior on skin

9
color is included in the particle filtering.

Working of GBC Devices:

• GDML (GestureDefinitionMarkupLanguage).

• <gesture>

• <sequence>

• <acquisition>

• <update>

• <loss>

10
APPLICATIONS:

The sixth sense prototype implements several applications that demonstrate the usefulness,
viability and flexibility of the system .sixth sense technology has got a wide range of
applications in different fields. The following can explain the areas in which this amazing
technology can be applied.

->U CAN KNOW THE TIME:

You hold up your left hand, fingers pointing to the right. The system recognizes that
you want to make a call, and projects a dialing pad onto your fingers. You tap the virtual
keypad with your right hand to dial the call.

->MAKE A CALL:

If you want to make a call to anyone and you find yourself with no such calling
device, it is possible to make a call with sixth sense technology. It’s enough to stretch your
palm and then a virtual dialing pad will be projected. In this way you can call with no
external device.

11
->TAKE A SNAPSHOT:

If you find beautiful scenery and want to capture it, you hold your fingers out at arm’s
length forming square. The system snaps a photo of what’s enclosed by your fingers. Later,
you can sort, resize and fiddle with these photos by projecting them onto any wall and
dragging their images with your fingertips, à la Microsoft Surface.

That’s it you captured it and you can project it on any surface say a wall, and you can
even edit it and mail to a friend.

->GET INFORMATION ABOUT ANY THING INSTANTLY:

When you go to a book store to buy a new book, as soon as u hold a book in your
hand the sixth sense device immediately projects a rating of the book enabling you to choose
the best one and moreover it provides you with other additional information. Not only a book
any purchase you make this device can help you. You hold up your boarding pass; the words
DELAYED 20 MINUTES or ON TIME are beamed onto it.

12
->WATCH VIDEOS ON A NEWSPAPER:

While you’re reading a newspaper, depending on the news and related photos you can
immediately watch the related videos on the news papers itself and make it useful and
interesting rather than going for other sources.

->LIVE INFORMATION:

Your newspaper will show you the live of weather information rather than starting
your computer to know this.

->GPS:

13
Using the global positioning system, you can project maps anywhere which provide
information on your position depending on your moves and find a way by using zoom out
and zoom in options.

When this topic is explained to our former president DR.A.P.J .ABDUL KALAM ,he
suggested a new use saying that a person who is dumb can use speaker system to express his
thoughts using this technology.many other processes are going on to make this technology
reach masses.

->PLAYING GAMES, BROWSING WEB ON PAPER:

Attach the microphone available with the webcam to a paper. Now the sound of touch
is exactly getting a where a person is touching the paper.

->GETTING INFO ABOUT OTHER PEOPLE:

14
You run into a guy at a party. The system projects his name and Face book keywords onto his
T-shirt, so you can remember who he was and start a conversation.

->PAPER WORK: And why only computers, we can play with papers. We can
take a part of a document putting over another paper and you can even modify the paper and
print it. So the work flow is more intuitive. In a more dynamic way you can pinch the
information on the paper to your system.

In this way, many things are made possible with sixth sense technology.

SOCIETAL ISSUES:

15
 This technology in essence seems fantastic; however, there are a number of societal issues
that come with it. The two major ones are: the accuracy of the information and the privacy of
individuals.

The first major issue which this new technology arises is how accurate the
information displayed is. For example, if you wish to see the environmental statistics of
certain product, what guarantees that the information displayed will be, at least, relatively
accurate? If the company which produces the product were to make sure to not let certain
information be displayed the product might seem harmless to the environment when it really
isn't.

The second major issue that arises from "Sixth Sense" is the issue of personal privacy.
The project is being promised to be able to get relevant information of people you meet. For
example, if you meet someone at a party you would get some relevant facts about that person.
The problem here is how much "relevant facts" there are. Even if only the name and the
birthday of the person is displayed, this would facilitate identity theft immensely not only
because the information is readily available but also because it would be much harder to track
the culprit since this technology would be used by a number of people. The possible solutions
to this, however, are unfortunately not much different from the solutions to internet privacy
that is to not provide any personally identifiable information. Of course, not even this
guarantees personal privacy since other people can upload information about yourself without
your knowledge.

ADVANTAGES:
SixthSense is an user friendly interface which integrates digital information into the physical
world and its objects, making the entire world your computer.
SixthSense does not change human habits but causes computer and other machines to adapt
to human needs. It uses hand gestures to interact with digital information.
1. Supports multi-touch and multi-user interaction.
2. Data access directly from machine in real time.

3. It is an open source and cost effective and we can mind map the idea anywhere.
4. It is gesture-controlled wearable computing device that feeds our relevant information and
turns any surface into an interactive display.

5. It is portable and easy to carry as we can wear it in our neck.

16
6. The device could be used by anyone without even a basic knowledge of a keyboard or
mouse.

7. There is no need to carry a camera anymore. If we are going for a holiday, then from now
on wards it will be easy to capture photos by using mere fingers

Future Enhancements:

To get rid of color markers

To incorporate camera and projector inside mobile computing device.
Whenever we place pendant- style wearable device on table, it should allow us to use the
table as multi touch user interface.
Applying this technology in various interest like gaming, education systems etc. To have 3D
gesture tracking.
To make sixth sense work as fifth sense for disabled person.

Impact:

Sixth Sense Technology May Change How We Look at the World Forever. The true power of
Sixth Sense lies on its potential to connect the real world with the Internet, and overlaying the
information on the world itself. Clearly, this has the potential of becoming the ultimate
“transparent” user interface for accessing information about everything around us.

TEACHERS:

Teachers may find that Sixth Sense, because of its unique ability to work with many of
Gardner’s multiple intelligences, will be a catalyst for students who might otherwise remain
unengaged in learning.

Sixth Sense uses the visual, kinesthetic, intrapersonal and interpersonal among others.

If this technology does nothing else, the idea that students can simply use their hands to draw
images on walls and take pictures with their hands will be (at least initially) quite motivating

Research

17
Sixth Sense will give new meaning to “looking something up.”

Since we are essentially talking about the internet, students will be given the opportunity to
portably research the world.

They can all go on a virtual field trip and project the images on their own desks, for example.

Collaboration:
Students will be given an opportunity through this technology to feel more connected to
others.

Although there does seem to be the sense that individuals could walk through life wired and
lonely, I like to think the classroom would serve as the environment in which students and
teachers using Sixth Sense would want to share.

Not only could the sharing take place in the classroom, I suppose since the users are
connected online, sharing could take place around the world.

CONCLUSION:

Imagination is the only limit for technologies like this one. It augments the physical
world around us with digital information and lets us use natural hand gestures to interact with
that information. It is a sense that would give seamless and easy access to Meta information
or information. So, with this technology we can have our digital world with us wherever we
go. And we can start using any surface around us as an interface. It will change the way we
interact with people not only with physical world. Actually the hardware is not that hard to
manufacture. The open source software is going to be made available .It costs around
$350.So the technology can reach masses than staying to laboratories or environments.

REFERENCES:

18
• P. Maes, P. Mistry. Unveiling the "Sixth Sense," game-changing wearable tech. TED
2009. Long Beach, CA, USA 2009

• GIZMODO - Sixth Sense Technology May Change How We Look at the World
Forever

• SOFTPEDIA - Next-Gen 'Sixth Sense' Device Created at MIT

• PCWORLD - MIT Wearable Gadget Gives You Sixth Sense

• http://www.pranavmistry.com/projects/sixthsense/

• BBC - Sixth Sense blurs digital and the real.

19