SIXTH SENSE

NOVA COLLEGE OF ENGINEERING AND TECHNOLOGY FOR WOMEN INFORMATION TECHNOLOGY

PRESENTED BY: Reshma.G Roll no:07s21a12a5 Email id:reshma.gollapudi@gmail.com Bhargavi.G Roll no:07s21a1203 Email id:bhargavi.gondi@gmail.com

ABSTRACT
Augmented Reality enhances user perception by overlaying real world information with virtual computergenerated information. The aims of the 6 th Sense project are to improve real-time interaction between the real environment and the virtual world and to maximize the user experience in mobile Augmented Reality. To achieve these objectives a generic framework constituted of two main layers is proposed. The End-to-End Adaptation Layer adapts in realtime the parameters of the Augmented Reality system to provide the user with the best possible experience despite the varying operating conditions such as the transmission link and user head motion. The Generic Augmented Reality Layer encompasses solutions to the problem of overlaying adequate information in the real scene and manages multimodal interaction with the virtual environment. Basically, Sixth Sense is a mini-projector coupled with a camera and a cellphone which acts as the computer and your connection to the Cloud, all the information stored on the web. Sixth Sense can also obey hand gestures, like in the infamous Minority Report. However, instead of requiring you to be in front of a big screen like Tom Cruise, Sixth Sense can do its magicand a lot more everywhere, even while you are jumping hysteric over Oprah's sofa.
2

he camera recognizes objects around you instantly, with the micro-projector overlaying the information on any surface, including the object itself or your hand. Then, you can access or manipulate the information using your fingers. Need to make a call? Extend your hand on front of the projector and numbers will appear for you to click. Need to know the time? Draw a circle on your wrist and a watch will appear. Want to take a photo? Just make a square with your fingers, highlighting what you want to frame, and the system will make the photowhich you can later organize with the others using your own hands over the air. But those are just novelty applications. The true power of Sixth Sense lies on its potential to connect the real world with the Internet, and overlaying the information on the world itself. Imagine you are at the supermarket, thinking about what brand of soap is better. Or maybe what wine you should get for tonight's dinner. Just look at objects, hold them on your hands, and Sixth Sense will show you if it's good or bad, or if it fits your preferences or not. Now take this to every aspect of your everyday life. You can be in a taxi going to the airport, and just by taking out your boarding pass, Sixth Sense will grab real time information about your flight and display it over the ticket. You won't need to do any action. Just hold it in front of your and it will work. The key here is that Sixth Sense recognizes the objects around you, displaying information automatically and letting you

access it in any way you want, in the simplest way possible.Clearly, this has the potential of becoming the ultimate "transparent" user interface for accessing information about everything around us. If they can get rid of the colored finger caps and it ever goes beyond the initial development phase, that is. But as it is now, it may change the way we interact with the real world and truly give everyone complete awareness of the environment around us. Or just make a few people look like dorks. It can go either way, really. Whatever it is, I want to try it..

around us and we can use natural hand gestures to interact with that information. Doesn't it sound so interesting!! We've evolved over millions of years to sense the world around us. When we encounter something, someone or some place, we use our five natural senses to perceive information about it; that information helps us make decisions and chose the right actions to take. But arguably the most useful information that can help us make the right decision is not naturally perceivable with our five senses, namely the data, information and knowledge that

INTRODUCTION
Everyday there is a new technology in the world. And how much of you remember the last one. But there are some technology which changes the thought of each and every person who are fond of new technology and accept it very well with its uses in their life. In the same way, recently, I got to know about the new technology and want to share with you all. It is not only unique but amazing technology which will change the entire world. It is none other than 'SixthSense'. This technology is very unique and who ever will know about it, will proud to be an Indian and the person who had developed it. The SixthSense technology is a wearable gestural interface that augments the physical world with digital information

mankind has accumulated about everything and which is increasingly all available online. Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world. Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. SixthSense frees information from its confines by seamlessly

integrating it with reality, and thus making the entire world your computer.

about topics which are as varied as they are interesting. For a long time these talks were for the

This SixthSense technology is developed by Pranav Mistry who is 28 year old and the most interesting part is that he is of Indian origin. He invented Sixth Sense' which one can also say WUW ( Wear UR World) in short. Actually, it is named so because you can operate this with a wearable gestural , it is user friendly interface which links the physical world around us and uses the hand gestures the same way you use other things like reading newspaper, turning pages, etc and in the same way you can interact with them. Well, this device will bring us closer to reality and assists us in making right decisions by providing the relevant information, thereby, making the entire world a computer. Isn't so?

privileged few who attended, however quite recently, these phenomenal talks were made freely downloadable from their website www.ted.com and made available to the public at large. This article is based on a talk that I downloaded from the TED website featuring Pattie Maes and a mesmerizing gadget that she, her student Pranav Mistry and her team are developing. They call it the Sixth Sense. And no, its got nothing to do with some deep spiritual stuff or Bruce Willis, but the end product is such that I wouldnt blame you if you said it was akin to magic.

SixthSense

Technology:

Life Beyond the Five Senses

TED(Technology,Entertainme nt,Design)
TED (Technology, Entertainment, Design) is one of the most interesting conferences that happen every year in the US where extraordinary people from all over the world come and share their insights and knowledge

which acts as the communication and The world has shrunk. Distances have dissolved. rendered Communication feasible. lines and this interaction with countless systems have been However technological overhaul has been peripheral and not so much related to the human body; researchers and innovators have constantly grappled with the issue of bridging the gaps which limit the human-environment contact. Well, looks like we finally may have stumbled upon an answer to that quagmire. Pranav Mistry, a student at the Media Lab of Massachusetts of Technology (MIT), has Institute developed a computation device. The entire hardware apparatus is encompassed in a pendantshaped mobile wearable device. Basically the camera recognises individuals, images, pictures, gestures one makes with their hands and the projector assists in projecting any information on whatever type of surface is present in front of the person. The usage of the mirror is significant as the projector dangles pointing downwards from the neck. To bring out variations on a much higher plane, in the demo video which was broadcasted to showcase the prototype to the world, Mistry uses coloured caps on his fingers so that it becomes simpler for the software to differentiate between the fingers, demanding various applications. The software program analyses the video data caught by the camera and also tracks down the locations of the coloured markers by utilising single computer vision techniques. One can have any number of hand gestures and movements as long as they are all reasonably identified and differentiated for the system to interpret it, preferably through unique and varied fiducials. This is possible only because the Sixth Sense device supports interaction. multi-touch and multi-user

gestural interface device which enables enrichment of the physical world with knowledge that is digital and allows a person to use natural motions to act together with this information so received. This device, tentatively name as the Sixth Sense, is a wearable machine that assists unexplored interactions between the real and the virtual sphere of data. It consists of certain commonly available components, which are intrinsic to its functioning. These include a camera, a portable batterypowered projection system coupled with a mirror and a cell phone. All these components communicate to the cell phone,

Now you might be thinking how does it work. Well, this SixthSense prototype is comprised of a pocket projector, a mirror and a camera and the hardware components are coupled in a pendant like mobile wearable device. To use this both the projector and the camera are connected to the mobile computing device in the users pocket. On the base of an information, I would like to share that projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces while the camera recognizes and tracks user's hand gestures and physical objects using computer-vision based techniques. The video stream data captured by the camera and processed by the video strea. After that they tracks the locations of the colored markers at the tip of the users fingers using simple computer-vision techniques, thus SixthSense also supports multi-touch and multi-user interaction. You can also zoom in and zoom out using intuitive hand movements. This technology also recognizes users freehand gestures or postures you can say.

Sixth Sense Commands Computer with Gestures

Pranavs Website defines is this a

gadget/experience

as SixthSense

wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information. By using a camera and a tiny projector mounted in a pendant like wearable device, SixthSense sees what you see and visually augments any surfaces or objects we are interacting with. SixthSense is the opposite of virtual reality. That is, virtual reality submerses a person in the digital world. SixthSense turns the tables and enables the digital world to shine a light on the real world. Communication in this computing environment happens with everyday human gestures. It projects information onto surfaces, walls, and physical objects around us, and lets interact with the projected information through natural hand gestures, arm movements, or our interaction with the object itself. SixthSense attempts to free

information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer.

Wearing sensors on your fingertips that understand gestures and a camera equipped, image projector system hanging around your neck. With SixthSense, you can draw on any surface, including your hand, check email by making an "@" symbol with your finger in the air, take pictures simply by framing an image with your hands, watch a video or read news displayed on a regular piece of paper, among other amazing feats.

'SixthSense' is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information

Devices or Components of Sixth Sense

The SixthSense prototype is comprised of a pocket projector, a mirror and a camera. The hardware components are coupled in a
7

pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the users pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks user's hand gestures and physical objects using computer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducially) at the tip of the users fingers using simple computer-vision techniques. The movements and arrangements of these fiducials are interpreted application number of into gestures that act as interaction instructions for the projected interfaces. tracked The fingers maximum is only

constrained by the number of unique fiducials, thus Sixth-Sense also supports multi-touch and multi-user interaction. The prototype device as it is right now, is assembled from available separate parts which together dont cost more than USD$ 350. The cost is bound to become lower and lower as they refine it and mass-produce it, and it cannot cost more tomorrow than it would cost today. Right now, it looks very clunky and quite heavy. Since you need to wear it around your neck, it might be quite uncomfortable to lug around. But again, its still very much in the research stage and as they get closer to a (possible) commercial release, they will for sure make much sleeker, lighter and cooler designs.

A wearable camera: think of the head strap lamp/camera that you

often see adventurers wearing while exploring a confined, remote cave on the National Geographic channel.

camera pocket.

are The

connected in projector

to the

the users visual

mobile computing device

projects

portable

projector:

This

information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks users hand gestures and physical objects usingcomputer-vision based techniques. The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tip of the users fingers using simple computer-vision techniques. It also supports multi touch and multi user interaction. The device has a huge number of

component is not much bigger than a thick BlackBerry, hanging from a neck cord, a little lower than chest height. The projector points straight down, so an angled mirror is attached to the bottom so that the image projects forward.

cell

phone/portable

computer: The first two component are wirelessly connected to a cell phone/portable computer. This unit has a wireless Internet connection.

Sixth Sense Technology will Revolutionize the World

applications. Firstly, it is portable and easily to carry as you can wear it in your neck. The drawing application lets user draw on any surface by observing the movement of index finger. Mapping can also be done anywhere with the features of zooming in or zooming out. The camera also helps user to take pictures of the scene he is viewing and later he can arrange them on any surface. Thats not it. Some of the more practical uses are reading a newspaper. Imagine reading a

The Sixth Sense prototype consists of a pocket projector, mirror and a camera. The device is pendant shaped like mobile wearing devices. Both the projector and the
9

newspaper and viewing videos instead of the photos in the paper. Or live sports updates while reading the newspaper. The device

can also tell you arrival, departure or delay time of your air plane on your tickets. For book lovers it is nothing less than a blessing. Open any book and you will find the Amazon ratings of the book. To add to it, pick any page and the device gives additional information on the text, comments and lot more add on features. While picking up any good at the grocery store, the user can get to know whether the product is eco friendly or not. To know the time, all one has to do is to just gesture drawing circle on the wrist and there appears a wrist watch. The device serves the purpose of a computer plus saves time spent on searching information. Currently the prototype of the device costs around $350 to build. Still more work is being done on the device and when fully developed, it will definitely revolutionize the world.

Ever thought of sending your vacation pictures within-a-wink-of-an-eye to friends and family? If yes, then Surface Computing is all what youd aspire to have.

Surface computing is a term used for a

Surface Computing
Ever wanted to move digital pictures around on the tabletop? Ever needed to resize your snaps without much irk? Ever imagined ordering a beverage during a meal with just a-tap-of-a-finger?

specialized computer GUI (Graphical User Interface), in which traditional GUI elements are replaced by everyday objects. It has no keyboard, no mouse, no trackball no obvious point of interaction, except its screen, as there are no cables or in external USB ports for peripherals. plugging

10

The term surface describes how its used. All interactions with the computer are done via touching the surface of the computers screen with hands or brushes, or via wireless interaction with physical objects like smartphones, digital cameras or PDAs.

Multi-user: The horizontal-form-factor makes it easy for several people to gather around them and experience a collaborative and face-to-face computing experience. Drag-and-drop: All interface components such as dialogs, mouse pointer, and windows are replaced with circles and rectangles outlining physical objects placed on the screen and are manipulated via drag and drop. Object different recognition: types of Users digital can place

Unveiled at the Wall Street Journals All Things Digital conference, at the hands of Microsoft Corporations CEO, Steve Ballmer, it blurs the lines between the physical and virtual worlds. Surface features a 30-inch diagonal square display built into a table configuration and runs on Windows Vista. It comprises a rear projection screen, five cameras that look through the screen to recognize and read items placed on the surface and to track hand gestures and touch, an Intel dualcore processor backed by 2 GB of RAM and a modest 256MB video card. It has in-built wired 10/100Mbit Ethernet and Bluetooth 2.0 support. Its features are: Direct interaction: Users can grab digital information with their touch and gesture. Multi-touch: It recognizes many points of contact simultaneously, not just from one finger like a typical touch-screen.

physical objects on the surface to trigger responses, including the transfer of digital content, for example, credit cards or hotel loyalty cards, or those having identification tags similar to bar codes. This means that when a customer simply puts a wine glass on the surface of a table, a restaurant could provide them with information about that wine and pictures of the vineyard it came from. No interface: With no interface text, it can be used by speakers of any language and any competency level. Harrahs Entertainment Inc., Starwood

Hotels & Resorts Worldwide Inc. and TMobile USA Inc. will be some of the first

11

companies

to

provide

unique

Surface

gaming concept take a leap farther than anticipated, the gaming interface has emerged from a mocked and rejected

experiences to their customers. Surface computing is going to revolutionize everyday lives, much like the way ATMs changed how we get money from the bank. Surface lets us manipulate a tremendous amount of information with our hands, so that the content works with you, rather than for you. It can compare product features, prices and phone plans. With Surfaces mapping application, you can manipulate a map and move it, shrink it and access personalized data for local sites, attractions and venues. It truly has the potential to tap the multi-billion dollar industry.

oscilloscope view to a 3-D user interactive view. The PlayStation, X-Box, Wii and et al have become a genre in gaming and redefined the art and passion involved. Apart from these LAN games, social networking games, online portals for games, PC based games, smart phones and a gamut of other categorically identifiable games have become a trendy frenzy.

Games Gamut
All work and no play makes Jack a dull boy Jack grew older with this proverb and has no penchant for play but loves to see his son play. Jack Jr however does not play but is not dull either; he games on his PlayStation 3. This form of entertainment has spawned many fads; video game culture has evolved in time, particularly in connection with internet culture. Gaming in all forms has got all the attention be it console based or web-based. Technological advances and commercial success helped the
12

Would you believe that as of July, 2008 The PlayStation has sold a 102 million units and to add to the trivia PlayStation 3 sold approximately 81000 units within 24 hours of its release. The ongoing hoopla in the entertainment industry is not about Michael Jacksons 3D comeback or the

2012s breathtaking CGI but the launch of a video game Call of Duty: Modern Warfare 2; a record 4.7 million copies of the game were sold on the first day raking up a whooping 186 million pounds. According to a report published in The Telegraph, UK, Mike Griffith of Activision admits that this can now be seen as a cultural phenomenon. The record shattering orders for launch of video games have proved that these games have emerged as a new medium for entertainment. The social networking websites are no exception on these numbers, Facebooks Farmville has a staggering 63.6 million users logging in almost every day across the globe and learning the art of farming; ironically the plight of the farmers all over is not cared or understood but a lost sheep or grown vegetable adds to the curiosity. People across all age groups are smitten by this gaming bug and left them in a frenzy over the blown-out-of-proportions technological advancements of these games. Five year old Mike plays as one of the cheeky monkeys in the Buzz! Junior: Jungle Party along with his other friends to figure out who is more cheekier. The Nielsen Active Gamer Study in November 2005 revealed that the average age in a group of 2000 regular gamers is 30 years

and this has been increasing ever since. The gender divide is more or less equal between males and females with the female percentage increasing. Apart from fun there are games that are thought provoking such as treasure hunts, castle quests and educational ones such as math blasters, word building which help the children build their mental math and vocabulary. From what started off as fun has over the ages transformed into passion and blossomed into art and culture. As we take a peek to learn about the effort that goes into making this ground breaking technology we surprise ourselves with the knowledge amassed. A multi billion dollar industry with big players like EA sports, DMA design etc have a wide range of products seeing the light of the day. Games like Halo, Age of Empires, Need for Speed, Grand theft Auto and Moto GrandPrix are on top of the charts. These games have become more user interactive with generations, A joystick to maneuver your car or swing your sword or take a shot gives you a real-time experience. The sixth sense concepts, real g advancements and game time developer interactions, internet, computer programmin

13

communities have ganged up to produce innovative games. Though technically savvy and financially attractive this industry has its own share of controversies; like related forms of media, computer and video games have been of the subject of frequent sexual drugs, tobacco, controversy and censorship, due to the depiction in graphic violence, of or themes, advergaming (a form of advertising games), consumption of alcohol consumption

Applications of Sixth Sense

The Sixth-Sense prototype implements several applications that demonstrate the usefulness, viability and flexibility of the system. The map application lets the user navigate a map displayed on a nearby surface using hand gestures, similar to gestures supported by Multi-Touch based systems, letting the user zoom in, zoom out or pan using intuitive hand movements. The drawing application lets the user draw on any surface by tracking the finger-tip movements of the users index finger. SixthSense also recognizes users freehand gestures ( postures ). For example, the Sixth-Sense system implements a gestural camera that takes photos of the scene the user is looking at by detecting the framing gesture. The user can stop by any surface or wall and flick through the photos he / she has taken. Sixth-Sense also lets the user draw icons or symbols in the air using the movement of the index finger and recognizes interaction the user those symbols For as instructions. to the map example, or

propaganda, or profanity in some games. Critics of video games include parents groups, politicians, organized religious groups, and other special interest groups, even though all of these can be found in all forms of entertainment and media. One of the most common criticisms of video games is that they allegedly increase violent tendencies among youth. games like death Race, Mortal Kombat are proponents of physical violence and have been banned for their content. Controversies are bound to be a part of any realm but the big picture this industry airs is more glamorous and attractive. A generation awakens to a new genre of entertainment and indulges in a non-existent world of exhilaration. Video Games in all formats are the future for fun and entertainment.

drawing a magnifying glass symbol takes application drawing an @ symbol lets the user

check his mail. The Sixth-Sense system also augments physical objects the user is
14

interacting

with

by

projecting

more

buy? You dont need to go searching for a computer, or even use google in your clunky blackberry, simply hold the book (game) up and you will get the amazon.com rating as well as excerpts from reviews (which you can expand with a gesture, in case you want to read the entire thing), even notes your other friends may have made about this particular product.

information about these objects projected on them. For example, a newspaper can show live video news or dynamic information can be provided on a regular piece of paper. The gesture of drawing a circle on the users wrist projects an analog watch. Imagine being able to walk into a super market and having the ability to decide which of the different varieties of products to buy based on preset personal preferences, simply by holding the product in your hand. Or, while you are taking a stroll, you see this fantastic photo opportunity and you dont need to fumble around for a camera, simply hold your hands up making a rectangular frame and the photo get taken And if you want to see how the photo looks like, you can project it onto any surface the road, a wall, even someones back, and zoom, resize and rotate the photo all with natural hand gestures.

You would never have to read a newspaper again, you could watch the up to date, relevant news projected right on top of the paper. And you could get a tag cloud of keywords pulled out of any individuals blog or website while you are talking to them to know more about them and their interests.

You are in a bookshop or a computer games store and want to know what other people think about the book or game you want to
15

Hear in the above figure the person checks out the road map using sixth sense technology.

Hear in the above figure the person checks out his image dump , resizes the images and arranges all the pictures using the sixthSense technology using figure tips. Hear in the above figure the user of SixthSense, projects the watch on his waist.

Dialing a call using sixth-sense technology, by projecting on palm and using cap for finger to dial a call.

You are driving to the airport and want to know your flight status and which gate you need to board from? No problem! Simply hold your boarding card up and the device will tell you if its delayed or on time, and which gate you need to board from! You will not ever need to wear a watch again or even carry your mobile You just (gesture) draw a circle on your wrist and voila, you will have the time. And you can project the number pad of a phone right on the palm of

16

your hand and use the other hands finger to dial a call Much better than the iPhone!

on a day to day basis would get annoying sooner than later. Today, we are all so proud of Pranav Mistry for developing this unique technology called 'Sixth Sense'

Conclusion
With this exciting technology just around the corner, the world will suddenly become more meaningful for everyone as they get relevant information about what they are looking at and interacting with right away. And maybe we will all know the Latin (and common) names of trees and flowers just by looking at them. The current prototype system costs

References:

http://www.engagementtheory.com/? p=50 http://www.pranavmistry.com/proje cts/sixthsense/

approximate $350 to build. He said it will be open source so that every one can use it at mass level without any problem. This is really a very big achievement and still more work is being done on the device and when fully developed, it will definitely revolutionize the world. The current prototype WUW sixth sense design is not sleek or somewhat attractive to be wearing around all the time but hopefully they can make a more sleeker design in the near future. Wearing a camera system around your neck or attaching it to your cap

17