Proceedings of International Conference on Circuits, Communication, Control and Computing (I4C 2014)
Virtual Mouse
Ashish Mhetar 1 , B K Sriroop2, Kavya AGS3,Ramanath Nayak4, Ravikumar Javali5, Suma K V6
1, 2, 3, 4, 5
Student
M.Tech – VLSI Design & Embedded Systems
MSRIT, Bangalore
Email: mashish@gmail.com
Sriroop.91@gmail.com
Kavya.home@gmail.com
ramanathj@gmail.com
ravijavali13@gmail.com
ABSTRACT
The major aspect in today’s e-learning is the improvement
in the methods of teaching by using technology dependent
resourceful products to have a better communication and
interaction between the teacher and the student. In favor of
this, we have proposed a product which aims to be a virtual
marker and also has additional advantage that it has features
of a Mouse incorporated into it. The existing Virtual Marker
is modified to function more than a marker and act as a
mouse pointer also giving it the advantage of all the
functionality of a mouse. In this paper, we have proposed a
hardware implementation of a Virtual Mouse which has an
improvement in performance of the existing “Virtual
Marker” by making it highly responsive in real time.
Key Words: Virtual Marker, Virtual Mouse,
1. INTRODUCTION
A virtual mouse is a device used to air-browse the functions of
a system such as a computer, laptop or a smart-pad with the
functions corresponding to a mouse and a marker. It can be
used in class-rooms for teaching without the board and chalk.
This can also be used in conferences as a smart device to work
as a mouse. A person is just required to make the
corresponding gestures imitating a mouse or a marker in front
of an Infra-red (IR) camera using this hardware device which
is modeled as a pen and this avoids the use of wired mouse,
chalks and boards. A virtual marker is already implemented by
using softwares to process the images captured to get the co-
ordinates. A WII-remote is configured to act as a virtual
marker with a IR source. But since it uses software it involves
certain delay to process and obtain the co-ordinates and more-
over it acts as a marker alone.
Using this idea we can build virtual marker provide functions
of a mouse by using a high-end microcontrollers configured as
a Human Interface Device (HID) for faster response interfaced
MSRIT, BANGALORE, India, 21-22 NOVEMBER 2014
6
Assistant Professor
ECE Department,
MSRIT, Bangalore
Email: sumakv@msrit.edu
with an IR camera using I2C interface. IR Camera tracks IR
sources and sends information regarding its position which is
converted to X and Y co-ordinates by Teensy and moves
mouse pointer to the position pointed by co-ordinates.
2. LITERATURE SURVEY
Multi-point Interactive Whiteboards are available using the
Wiimote [4]. The components used are IR pen, computer with
Windows XP (installed with Microsoft .NET framework, the
Wiimote Connect program and the Wiimote Whiteboard
software), wiimote controller, a beamer capable of a 1024 x
786 pixel resolution. Here the wiimote controller tracks the
infra-red source on the white board and sends info to PC via
Bluetooth. The teaching platform comprises of a Wii-mote-
based multi-touch teaching station, a Wii-mote-based
interactive whiteboard and a Wii-mote-based stylus input
conversion tool [5]. According to the literature survey, most
people have used the Wii-mote to configure it as a virtual
marker.
2.1 MOTIVATION
A lot of virtual markers are available in market, but it can only
function as a marker alone. A high level processor is used to
process the data and used as a virtual marker, but it can do
even more function like mouse functions. Its capability is not
being utilized fully to its ability to function and hence the
product becomes under-loaded and hence costlier in terms of
market price versus function.
3. PROPOSED SOLUTION
In this paper we propose a method to improve the utilization of
resources without wasting its ability to create a virtual mouse
and decrease the cost when compared to the functions it can
perform. A virtual mouse is more compatible and useful than a
marker and performs all the functions of a mouse while it can
still act as a marker.
978-1-4799-6546-5/14/$31.00 © 2014 IEEE 69
Proceedings of International Conference on Circuits, Communication, Control and Computing (I4C 2014)
3.1 Block Diagram:
Figure 1 shows general block diagram of virtual mouse.
Infrared camera is interfaced with Teensy 3.1 using I2C
Communication which in turn is connected to laptop.
Figure 1: General block diagram
3.2 PIXART'S IR CAMERA:
The Wii Remote includes a 128x96 monochrome camera with
built-in image processing. The camera's built-in image
processing is capable of tracking up to 4 moving IR objects,
and these data are the only data available to the host. Raw
pixel data is not available to the host, so the camera cannot be
used to take a conventional picture. The built-in processor uses
8x sub pixel analysis to provide 1024x768 resolutions for the
tracked points.
Figure 2: IR camera
3.3 IR Camera Interfacing:
The camera outputs the data over an I2C line or what is
sometimes referred to as a Two-Wire-Interface (TWI).
Following Figure 3 shows pin configuration for PixArt IR
Camera.
Figure 3: Pin out of Pixart IR camera
The IR Camera is enabled by setting bit 2 on output ports 0x13
and 0x1a:
The first enables a 24MHz pixel clock on pin 7 of the camera.
The second pulls pin 4 low - probably an active low enable.
MSRIT, BANGALORE, India, 21-22 NOVEMBER 2014
3.4 Optical Characteristics of IR Camera:
The IR camera has an effective field of view is about 33
degrees horizontally and 23 degrees vertically (as measured on
one unit). With the IR-pass filter intact, 940nm sources are
detected with approximately twice the intensity of equivalent
850nm sources, but are not resolved as well at close distances.
If the filter is removed, it can track all IR frequencies such as
IR light from tube light, sunlight etc.
3.4.1 Data Formats
The IR Camera can return different sets of data describing the
objects it is tracking. When the IR camera identifies an object,
it assigns it to the first available object slot. If an object moves
out of view, its slot is marked as empty (returns 0xFF data),
but other objects retain their slots. For example, if the camera
is tracking two objects and the first moves out of view, the
data returned will be [empty, second object, empty, empty].
With more than four objects visible, the camera is prone to
rapidly switching between some of them. This could allow
perception of more than four objects, at a reduced response
speed and reliability.
IR Camera works in different operating modes which are as
follows: [2]
1) Basic Mode
2) Extended Mode
3) Full Mode
Each mode has its own data format which is explained below
The data format MUST match the number of bytes available in
the Reporting Mode selected. Even choosing a mode with
space for more bytes than necessary will not work, it has to be
an exact match. In this project we have chosen extended mode.
Extended Mode
In Extended Mode, the IR Camera returns the same data as it
does in Basic Mode, plus a rough size value for each object.
The data is returned as 12 bytes, three bytes per object. Size
has a range of 0-15.
The data format for each object is shown in Figure 4:
Figure 4: Extended mode
3.5 HUMAN INTERFACE DEVICE
We have used a microcontroller which can mimic HID mouse
function provided the coordinates. Using this HID
functionality we can implement Virtual IR Mouse.
A human interface device or HID is a type of computer device
that interacts directly with, and most often takes input from,
humans and may deliver output to humans. The term "HID"
most commonly refers to the USB-HID specification.
70
Proceedings of International Conference on Circuits, Communication, Control and Computing (I4C 2014)
In the HID protocol, there are 2 entities: the "host" and the
"device". The device is the entity that directly interacts with a
human, such as a keyboard or mouse, in this case device is
Virtual Mouse. Virtual Mouse consists of a IR LED (IR
Source), battery and three buttons for mouse operations. The
host communicates with the device and receives input data
from the device on actions performed by the human. The host
in this case is Teensy (ARM M4) [3]. Output data flows from
the host to the device and then to the human.
3.6 WORKING PRINCIPLE
The algorithm of virtual mouse is as follows:
i. Object recognition.
ii. Trace object.
iii. Co-ordinate calculation.
iv. Setting cursor position.
v. Event generation.
Here we are using PixArt's IR Camera from Nintendo Wiimote
device. This Camera is first initialized using initialization
sequence. For this application we have set the Camera in
Extended Mode for its operation. We will use IR Pen as IR
light source, made by IR LED and battery as shown in figure
5.
Figure 5: Virtual Mouse (IR Pen)
Sensor assembly contains IR Camera interfaced with a high
end microcontroller Teensy(ARM M4) using two wire
interface I2C protocol, Teensy in turn is connected with laptop
or computer by wired or wireless connection.
i. Object Recognition:
IR camera works with frequency of 24MHz which can be
generated using external crystal or using Teensy. This Camera
only Recognizes IR source. Hence it will recognize the IR
light from Virtual Mouse.
ii. Trace object.
We are using teensy for generating this 24MHz clock. IR
Camera uses this clock to sweep both X and Y axes. Sweeping
through X and Y axes gives exact position of IR blob (Object)
hence tracing the source. Camera then sends this position data
to a microcontroller through I2C communication as shown in
block diagram.
Figure 6 shows general setup required for implementing virtual
mouse, it shows general distance and angle requirement for
good results. Actually sunrays and tube light also contains IR
frequency light and our IR camera also detects these rays. But
as we don't want these rays to be detected by camera, we will
use a filter glass that will filter out IR rays from sunlight and
tube light. It will only detect IR Pen.
MSRIT, BANGALORE, India, 21-22 NOVEMBER 2014
Figure 6: General setup required for implementing virtual mouse
First presenter or tutor will hold the IR Pen(Virtual Mouse) in
hand to do required mouse gestures. To begin with, the
working area of the Virtual Mouse is defined by the first four
clicks at the corners of projector screen. IR camera gives one
co-ordinate data for each click. After getting the data co-
ordinates are calculated as follows.
iii. Co-ordinate calculation.
After getting the data the Coordinate conversion command is
sent, to read as follows.
[START condition] [0xB1]
It will give 3 bytes data for each co-ordinate position, those
bytes are referred as XX, YY & SS. X & Y co-ordinates are
calculated using these co-ordinate values as shown by
following equations
X coordinate = (SS & 0x30) << 4 + XX (1)
Y coordinate = (SS & 0xC0) << 2 + YY (2)
iv. Setting cursor position.
By using these calculations, it is possible to obtain the values
of the coordinates. As mentioned a microcontroller which has
HID function can be used to move the mouse pointer. The
Teensy library has inbuilt HID functions. The X and Y co-
ordinates that we got from IR camera is given as arguments to
HID functions which moves the mouse pointer to that position
shown by X and Y co-ordinates, hence setting the mouse
pointer. If we want to imitate mouse actions such as left click
and right click we can use inbuilt library and implement mouse
functions.
v. Event generation.
As you move the mouse, IR camera tracks the IR Led position
as co-ordinate of that point and reflects same motion on
projector. As we move the Virtual mouse; Camera will track
its new position and teensy moves the mouse pointer to new
position. Like this, as you keep on moving the mouse Camera
will go on tracking the blob and Teensy will move the mouse
to that position Hence imitating the gesture. This process is
done at 24MHz frequency, hence the actual series of points
appears as continuous line.
71
Proceedings of International Conference on Circuits, Communication, Control and Computing (I4C 2014)
4. EXPERIMENTAL RESULTS
Accurate marking of coordinates is crucial for the working of
Virtual Mouse. The Coordinate calculations are done using
equations (1) and (2). The Virtual Mouse can perform
different functions as follows:
1) Virtual Marker: Many times while doing presentation we
may require marker to explain some concept. Instead of actual
writing on board, virtual mouse can be used as marker to write
virtually on screen. As shown in Figure 7, we can write
anything in a Paint document which is reflected on screen.
Figure 7: Virtual Mouse used as a Virtual Marker
2) Mouse Functions (Right Click and Left Click): Virtual
Mouse can be used to perform mouse clicks virtually on screen
result of which is actually reflected in computer. As shown in
Figure 8 we can see that when we left click on windows button
on screen we can see on computer that it actually pops
windows option after click. As shown in Figure 9 when we
right click on desktop, a window pops on the screen showing
options after a click.
Figure 8: Left Click using Virtual Mouse
Figure 9: Right Click using Virtual Mouse
For right click in figure 9, IR camera tracks IR light and gives
co-ordinates of that point. The graphical representation of
Figure 9 is shown in the Figure 10.
MSRIT, BANGALORE, India, 21-22 NOVEMBER 2014
Figure 10: Interpretation of Figure 9
Then actual co-ordinates are calculated using the equation (1)
& (2) mentioned in the co-ordinate calculation. Then library
function will imitate right click at that point opening right click
dialog box.
5. CONCLUSION
The major aspect in today’s e-learning is the improvement in
the methods of teaching by using technology dependent
resourceful products to have a better communication and
interaction between the teacher and the student. In favor of
this, people have proposed virtual marker; but our proposed
product aims much more than a marker and can be used even
as a mouse. Thus exploring the HID functions of a high end
microcontroller and increasing its utility by amplifying its
functionality is done and the desired results are obtained.
REFERENCES:
[1] Hacking the Nintendo Wii Remote, www.computer.org/pervasive/
lee2008.pdf
[2] http://procrastineering.blogspot.com/2008/09/working-with-pixart-
camera- directly.html
[3] Teensy (ARM M4), https://www.pjrc.com/teensy/
[4] Johnny Chung Lee, http://www.wiimoteproject.com/ Johnny Chung Lee
- Projects - Wii.pdf
[5] Soon Nyean Cheong, Wen Jiun Yap and Mun Leong Chan, “Cost-
effective wiimote-based technology enhanced teaching and learning
platform”, 2nd International Conference of Teaching and Learning
(ICTL) INTI University College, Malaysia, 2009
[6] Nintendo, Consolidated Financial Highlights, www.nintendo.
co.jp/ir/pdf/2008/080124e.pdf, 24 Jan. 2008,
[7] A Murgia1, R Wolff2, P M Sharkey1 and B Clark1, “Low-cost optical
tracking for immersive collaboration in the CAVE using the Wii
Remote”, Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, pp
104-109, 2008
[8] Andreas Holzinger, Selver Softic, Christian Stickel, Martin Ebner.
“Nintendo Wii Remote Controller in for Higher Education:
Development and Evaluation of a Demonstrator Kit for e-Teaching”,
Graz University of Technology, Institute for Information Systems &
Computer Media (IICM)
72