Table Of Contents

1. Brief overview of Kinect 2. History of kinect 3. Why develop kinect? 4. Advantages/problems with kinect 5. How kinect works? 6. Kinect for windows 7. How to set up kinect with xbox 360 console 8. Technology in kinect 9. Projects on kinect 10. 1 of the most popular games involving kinect (star wars)

Kinect
Brief overview of kinect

kinect is basically a motion sensing input device developed by Microsoft for the Xbox 360 video game console (and now for PC). Based around a webcam-style add-on peripheral for the Xbox 360 console, players can navigate around and interact with the xbox 360 console via hand/body gestures and spoken command, completely eliminating the need of a game controller. Kinect was launched in North America on November 4, 2010 Europe on November 10, 2010 Australia, New Zealand and Singapore on November 18, 2010 Japan on November 20, 2010.

Kinect achieved the Guinness World Record of being the "fastest selling consumer electronics device" due to 10 million units of the Kinect sensor have been shipped as of March 9, 2011.

(Note : kinect was first known as Project Natal)

History of kinect June 6, 2009 Microsoft announces Project Natal at the annual E3 press conference. They demonstrate the new motion sensing technology they have developed. September 24, 2009 Microsoft provides more information about the games for Natal and who will be making them. June 13, 2010 Microsoft announces that Project Natal will now go by Kinect. July 20, 2010 Microsoft gives Kinect a price and release date of November 4, 2010 with a price of $149.99. November 4, 2010 Microsoft launches Kinect in North America. November 15, 2010 1 million units of Kinect have been sold less than two weeks after its North American launch. November 18, 2010 Kinect is launched in Australia. November 20, 2010 Kinect is launched in Japan. November 29, 2010 Microsoft has sold 2.5 million Kinect units in the 25 days since it was released. January 5, 2011 8 million Kinect units have been sold. February 21, 2011 Microsoft announces a plan to release new software for Kinect on Windows machines in the spring. March 3, 2011 10 million Kinect units have been sold.

Why develop kinect? A few important points : Microsoft wanted people to start using their full bodies, rather than just their hands,
while gaming good way for people to interact without controllers, broadening the typical gamer base encourages people to be active

released to compete with the Nintendo Wii and the Playstation eye and move.

Nintendo WII

PlayStation 3 Move

PlayStation 3 Eye

Advantages of kinect 1. 2. 3. 4. 5. 6. no controller voice recognition build up family bonds help you to exercise small and easy to use fun and entertaining

Problems with kinect fatigue players must take break in between pricy need a lot of space eliminate the use of traditional video game controller unable to recognized dark skinned peoples ( must manually sign in )

in such a case , users must adjust the lighting condition as the RGB camera embedded is unable to function in darkness comparable to dark skinned peoples

How kinect works?

Motion sensor

o o o o

Invisible light source illuminates subject Sensor chip measures distance and light travel Unique embedded imaging software uses depth map to perceive & identify objects End user device reacts appropriately

MiddleWare kinect firmware can recognize people distinguishing human body parts, joints, movements and faces the camera knows who you are, once u step in

Xboxs processor o translating movement captured by kinect camera to meaningful on screen events

o context-specific o easier time recognizing some gestures and postures than others

Skeletal tracking

Digital Skeleton

o creates a digital skeleton of you based on depth data. o If you move left or right or jump around, the sensor will capture it and put you in the game.

Facial recognition

o Players have to create a kinect ID (profile) o This ensure that the kinect will recognize the player in future Kinect has 2 camera : A traditional color video camera (light dependent) o Take pictures and enable video conference Infrared light sensor (not light dependent)

o Measure depth position and motion For facial recognition to work, it uses both cameras

Voice recognition

o Have 4 mics within the sensor to recognize and separate your voice from other noises o Allow you to control movies and etc with your voice (e.g. when playing a movie, u can say play kinect or when pausing a movie, pause kinect)

Kinect for Windows SDK Kinect is also window compatible Need to download some additional drivers specially for kinect (OpenNI, Sensor kinect drivers, NITE middleware) as the official driver havent been released yet Able to build kinect applications with C++, C#, or Visual Basic by using Microsoft Visual Studio 2010

Have the following features 1. Raw sensor streams: Access to low-level streams from the depth sensor, color camera sensor, and four-element microphone array. 2. Skeletal tracking: The capability to track the skeleton image of one or two people moving within the Kinect field of view for gesture-driven applications. 3. Advanced audio capabilities: Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API. 4. Sample code and Documentation.

How to set up kinect with xbox 360 console ?

Position the kinect sensor

. Kinect sensor needs to see your entire body.

Place the sensor near the edge on a flat, stable surface. Position the sensor between 2 feet (0.6m) and 6 feet (1.8m) from the floor. Ideally, the sensor should be within 6 inches (15 cm) above or below your TV. Avoid positioning the sensor in direct sunlight or within 1 foot (.3m) of audio speakers. Do not manually tilt the sensor, it adjusts automatically. Be careful not to drop the sensor

Set up the play space

Kinect needs to be able to see your entire body.

Clear the area between the sensor and the players. One player: Stand back 6 feet (1.8 m). Two players: Stand back 8 feet (2.4 m). Make sure that the play space is at least 6 feet (1.8 m) wide, and not wider or longer than 12 feet (3.6 m). Make sure the room has bright, even lighting.

Connect the cables

Original Xbox 360 console

Plug one end of the cable into the USB port on the back of the console and the other end into an electrical outlet. If you have a wireless networking adapter, plug it into the USB port on the front of the console by using the supplied USB extension cable. Move all other USB accessories to USB ports on the front of the console.

Xbox 360 S console

If the Kinect sensor is attached to a cable with a power plug, disconnect the power plug portion of the cable. A power cable is not needed with the Xbox 360 S console. Plug the sensor cable into the AUX port on the back of the console.

Update Console Software

To update your console software for Kinect Insert a Kinect game disc and follow the on-screen instructions to update the console software. Don't have a Kinect game disc? Sign into Xbox LIVE. If a console update is available, you will be prompted to update your console software.

Technology in kinect

3D Depth Sensor Emitter 3D Depth Sensors RGB Camera

Motorized Tilt MultiArray MIC

Front view

Back view

Side view

Bottom view

kinect sensor device consisted of 6 components 3D depth sensors RGB camera Multi-Array MIC Motorized Tilt Kinect sensor Microsoft software

3D Depth Sensors Consists of infrared laser projection combined with CMOS (complementary metal
oxidesemiconductor) sensor

captures video data in 3D under any ambient light conditions adjustable sensing range calibrating the sensor based on gameplay and players physical environment make facial recognition more accurate

RGB Camera a video camera that deliver 3 basic color components found in various webcams and mobile phones standard 640x480 resolution operating at 30 frames per second helps to enable facial recognition and more depend on visible light

Multi-Array MIC a microphone that able to locate voice and extract ambience noise. Enable use of voice command Able to navigate around with voice Enables chatting online (msn, skype)

Multi-Array Mic consisted of many microphones functioning as a single directional input device. Using sound propagation principles, the principal sound sources in an environment can be located and distinguished from each other

Kinect Sensor Combination of RGB Camera, 3D Depth Sensor and multiarray microphone Track full body movement and individual voice, turning you into a controller Microsoft Software A proprietary software layer Differentiates kinect from other technology in the market with its ability to enable human body recognition and extract other visual noise. Motorized Tilt

 Necessary as not every room is the same size and not every television has the same height Can tilt up or down 30 degree Zoom function

Projects on kinect

Project which involve hand manipulation

Object Manipulation

In the screenshot above, the man is manipulating the kettle by controlling the movement of the kettle with his 2 hands. More information can be found here : http://www.vimeo.com/17045326

Grabbing and letting go

Need to enter the contours and convexity defects to estimate when a hand opens or closes

in the screenshots above , the person is controlling the virtual hand grabbing the yellow ball and replacing it at another location with his own hands. More information can be found here : http://vimeo.com/17333026
Shadow Puppet

In the screenshot above, a program was created in the hacked kinect to recognize the elbow, forearm, wrist, fingers and thumb to in the bird shape. The bird movement is controlled by the movement of the hand , by opening and closing your hand like a beak causes the bird to do the same .

More information can be found here : http://www.popsci.com/diy/article/2010-11/five-hacks-free-microsofts-kinect-xbox?page=6

Making my room dance

In the screenshot above, the person is dancing and as the person dances, the room react to the dancing movement of the person by shaking and moving with the flow. More information can be found here : http://vimeo.com/17176078

Manipulating stuffed monkey

In the screenshot above, the person is controlling the monkey with his body movements More information can be found here :

http://technabob.com/blog/2011/08/11/kinect-hack-stuffed-monkey/

Project which help visually impaired to navigate around

In the screenshot above, the kinect is helping a visually impaired person to navigate around warning the user about any obstruction or obstacles in his way by a vibrator in the belt, and a Bluetooth headset which provide verbal feedback. More information can be found here :
http://medgadget.com/2011/03/project_navi_a_kinect_hack_that_helps_visually_impaired_navigat e_indoors.html

Projects that involve creativity and drawings

Ghost following you

In the screenshot above, the ghost is draw in the video frame and its designed in such a way that the ghost is always behind your blind spot (behind you) More information can be found here :
http://www.destructoid.com/kinect-hack-makes-the-mario-boo-ghost-really-follow-you195351.phtml

Projects which involve virtual effects

Dragonball Z-Style Super Saiyan

In the screenshot above, the person grow out fake virtual hair and has a yellow aura around his body and he can unleash the super saiyan which consisted of light beam and various lightning effects. More information can be found here :
http://www.pcworld.com/article/218681/go_super_saiyan_dragonball_zstyle_with_kinect_hack.ht ml

Comic book sound effect in fighting scene

in the screenshot above , when a person is being punched/kicked by another person , a

comic book-esque sound effects together with the wordings will be played more information can be found here : http://www.pcworld.com/article/221231/comic_kinect_turns_human_interaction_into_a_comic_b ook_fight_scene.html

cool lighting effects

In the screenshot above, the kinect sensor is reacting to the movement and gesture of the lady hand and thus triggering the lighting on the sofa More information can be found here : http://www.youtube.com/watch?v=ho8KVOe_y08

Interactive stage lighting with kinect

In the screenshot above, hands will be placed in the hole and the kinect sensor inside the box will read the hand movement and correspond to them

Some of the beautiful lighting effect projected out More information can be found here :
http://kinecthacks.net/interactive-stage-lighting-with-kinect/

project involve robots

kinect powered robots that could help earthquake victims

In the screenshot above, the robot have sensors which act as eyes to search for victims in an earthquake zone They are the 3rd team of peoples who revamped this rescue robots but using kinect as shown below

In the screenshot above, a kinect powered robot can help to locate earthquake victims and rescuing them out of the earthquake zone More information can be found here :
http://www.pcworld.com/article/221075/kinect_powered_robot_could_help_earthquake_victims.h tml

Kinect assembly robot

step 1 : worker first prepare the materials on the surface

Start button

Step 2 : initiate the robot by pressing the start button

Step 3 : robots began to assembly the parts as programmed by the person in charge More information can be found here : http://www.kinecthacks.com/kinect-assembly-robot/

Surgical robots

in the screenshot above, a kinect camera is used for hand gesture recognition to give commands to the robots and to browse through radiological images on a nearby computer screen Example of hand gesture recognition : cutting with index and middle finger to indicate scissors. more information can be found here :
http://medgadget.com/2011/02/kinectpowered_purdue_robot_to_replace_scrub_nurses.html

Kinect powered robot (jedibot) duelling

Kinect Sensor

In the screenshots above, the kinect sensor tracks the location of both lightsabers and has the
robot make an attack or defending move based on their positions. More information can be found here :

http://science.kqed.org/quest/2011/07/26/stanford-students-create-kinect-powered-jedibot/

1 of the most popular games involve kinect

Star Wars Kinect Star Wars allows fans to physically experience training as a Jedi, using the Force and battling with a lightsaber.

Gameplay Screenshots

Flashing and slashing of lightsabers

Using the force to move and carry objects