Proceedings of the 2016 IEEE
International Conference on Robotics and Biomimetics
Qingdao, China, December 3-7, 2016
Interactive Aerial Projection of 3D Hologram Object
Jiono Mahfud and Takafumi Matsumaru, Graduate School of Information, Production and Systems
Waseda University, 2-7 Hibikino, Wakamatsu-ku Kitakyushu, 808-0135, Japan
mahfudjiono@akane.waseda.jp
Abstract— In this paper we present an interactive aerial
projection of 3D hologram objects by using the pyramid
hologram and parabolic mirror system (for 3D hologram
object reconstruction) and the Leap Motion sensor (as a finger
movement detector of a user). This system not only can
reconstruct and project the 3D hologram object in the mid-air,
but also provides a way to interact with it by moving a user’s
finger. There are three main steps: the reconstruction of 3D
object, the projection of 3D hologram object in the mid-air,
and the interactive manipulation of 3D hologram object. The
first step is realized by using pyramid hologram with LCD
display. The second step is achieved with the parabolic mirror
hologram. And Leap Motion sensor is used for the last step to
detect user finger movement. This paper traces the design
concept and confirms the system function of an interactive
aerial projection of 3D hologram objects by a prototype
demonstration.
Keywords— aerial projection; 3D hologram object;
interactive manipulation
I. INTRODUCTION
3D hologram object is a record pattern of the light
method. This object is presented by a projection technique
based on an illusion. Illusion of 3D objects is produced from
a light source projected into the object surface. The light
illuminates the object to create an interference between the
object and scattered light [1]. When two lights interact with
each other, the 3D hologram object namely aerial projection
is presented.
Aerial projection is an important aspect of digital
signage, control operation, entertainment, business, and
others. Among these applications of the aerial projection, the
effectiveness and reliability of control operation will be
improved for the movement of robots. In this paper, we
present an interactive aerial projection of 3D hologram
objects by using the pyramid and parabolic mirror hologram
system, and the Leap Motion sensor as a finger movement
*Research supported by Waseda University and Kayamori Foundation.
Jiono Mahfud is with Bio-Robotics & Human-Mechatronics Lab. ,
Graduate School of Information, Production and Systems, Waseda
University, Kitakyushu, Fukuoka, Japan (e-mail: mahfudjiono@
akane.waseda.jp).
T. Matsumaru is with Bio-Robotics & Human-Mechatronics
Lab. , Graduate School of Information, Production and Systems, Waseda
University, Kitakyushu, Fukuoka, Japan (e-mail: matsumaru@waseda.jp).
978-1-5090-4364-4/16/$31.00 ©2016 IEEE
detector. The pyramid hologram device is used to construct
the 3D object from 2D object projection. Parabolic mirror
hologram is used to project the 3D hologram object in the
mid-air. And Leap Motion sensor is used to detect user finger
movement, so that the 3D hologram object will be
interactively operated by the user.
This paper is divided into five sections: Section 1 here is
the introduction. Section 2 discusses about works and
researches related to the interactive aerial projection. Section
3 explains the design concept of the interactive aerial
projection of 3D hologram object. Section 4 presents the
construction of the prototype system. And Section 5
describes the conclusion.
II. RELATED WORKS
Prior to the actual processing of the interactive 3D
hologram objects, we introduce the related works concerning
the displaying of 3D objects in space (based on CGH and
SLM technique) and the 3D object manipulation (based on
the interactive object using Leap Motion sensor and the
augmented reality using HoloLens).
A. 3D Hologram Objects Projection in Mid-Air
There are two kinds of the aerial projection techniques:
the computer generated hologram (CGH) and the spatial
light modulator (SLM) technique [2-4].
1. Computer Generated Hologram
Computer Generated Hologram (CGH) is one of the
techniques to make a hologram object. In this kind of system,
a hologram object is projected by laser light, which is divided
into two types: object light and reference light by a half
mirror. While object light is projected from an illumination
to the object as scattered light, reference light is a
medium-recorded interfacing fringe formed by the object
light. The working principle of this system is shown in Fig.
1. Firstly, by illuminating the hologram using a laser source
with two mirrors placed at the same angle and a lens for
positioning the reference light, an image is generated at the
previous position of the objects in the recording process.
Then the holographic object can be reproduced with some
brightness (decided by the amplitude of light) and at some
direction (decided by the phase of light).
1930

Figure 1. Computer Generated Hologram [2]
2. Spatial Light Modulator Holograms
3D images can be built using the spatial light modulator
(SLM) based on the intensity of the light beam. SLM is used
in the holographic data storage setup to encode some
information data into the laser beam in the same way as
doing a transparent foil for an overhead projector. The
reconstruction of spatial image is made in a similar manner
when the objects are being observed directly. This method is
mainly used in the internal photography. The internal
photography is a special kind of method which can
reproduce discrete light rays from an object. The 3D
holographic object is produced with a newly-devised
space-saving image-readout optical system for multiple
reflection type SLMs [5].
There are many applications based on SLM which need a
computer device as the host. The function of the host
computer is to control the sending of command signal and
displaying signal of the SLM itself in real time. The device
needs a driver as the interface for computer and SLM. This
driver reconstructs the object pattern. Then the 3D hologram
object is generated as Fig. 2.
Figure 2. 3D Object based on SLM [3]
Although these two techniques produce a hologram
object, there is a considerable distance between the hologram
object and the user. We can see from Fig. 1 that the object is
projected in the opposite side of the user, therefore making it
impossible to be interacted by the user. Other problem of
SLM is about the capability of spatially-resolved intensity
modulation, the phase shifting, the beam steering or the
wave front shaping, and the total control of complex wave
fronts [6]. Therefore, it is difficult to apply SLM to our
intended system which can project 3D hologram object with
simple structure and uncomplicated mechanism.
1931
B. Interactive Sensing System
There are many research works to detect 3D objects for
an interactive system. The interactive system usually uses
two kinds of sensing system, the marker based system and
the non-marker based system.
1. Marker based System
Marker based system can detect the motion of user and
supports feedback vibration. One example of marker based
system is Haptic Workstation™. This device can detect
6-axis of hand movement through data gloves [7]. Another
example is based on embedded optical sensors onto a TFT
array substrate system [8]. The position of an object can be
detected by optical sensors with the same processes as a TFT
array. However, the bulk and weight of this device would be
inconvenient problem.
2. Non-marker based System
The recognition of hand movement and gesture is a kind
of interaction for communication between a user and a
machine. Non-marker based system normally uses the
camera system to calculate the position and direction of the
object. There are various kinds of sensing device for
movement and gesture recognition, such as Kinect and Leap
Motion sensor. Kinect sensor is a device which can be
applied in an interactive system such as remote operation
robot control system based on body gesture recognition [9].
Leap Motion sensor is especially used for hand movement
and gesture interaction [10]. Leap Motion sensor consists of
two monochromatic IR cameras and three infrared LEDs
that can detect hands and fingers.
The example of usage Leap Motion sensor is shown in
Fig. 3. It shows Japanese writing style on the table [11]. In
this system, user can do some Japanese writing exercises.
There are three kinds of exercises according to the learner's
ability: copying exercise, tracing exercise, and a
combination of them.
Figure 3. Leap Motion sensor used for calligraphy
exercise [11]
While the calligraphy-stroke learning support system
projects a stroke model on the table, the Leap Motion sensor
detects the brush movement. It is effective and efficient for
writing exercise. In this case, non-marker based system is
better to use in a limited work space of the application
system, because the bulk and weight of the non-marker
device is convenient.
III. DESIGN CONCEPT OF INTERACTIVE AERIAL PROJECTION
OF 3D HOLOGRAM OBJECT

This section explains the design concept of the

interactive aerial projection system. There are two sub
sections: the requirement and the design principle.

A. Requirement
The requirement of interactive aerial projection of 3D Figure 5. Working Principle of Pyramid Hologram
hologram objects system is divided into two categories. The
Four symmetrically opposite variations of the same object
first requirement is the aerial projection of 3D hologram
in LCD display are projected on the four faces of the
object reconstruction which is to be projected in the mid-air.
pyramid. By means of this projection, each side of the object
The second requirement is the user interaction of the 3D
falls at the center of the pyramid. Fig. 6 shows the system
hologram object. It means that the purpose of this system is
design of the pyramid hologram.
not only to reconstruct and project the 3D hologram object in
the mid-air but also to facilitate users to do some interaction
using his/her finger movement.

B. Design Principle
1. Aerial Projection of 3D Hologram Object
Reconstruction
Firstly, we explain the design principle of 3D hologram
object projected in the mid-air. The system mainly consists
of 3D object reconstruction device (pyramid hologram Figure 6. Design of the Pyramid Hologram
system) and aerial projection device (two concave mirrors).
The 3D object reconstruction device can create the 3D b. 3D Hologram Object Projection In Mid-air
holographic image as the original object from a LCD After displaying a 3D object on the pyramid hologram,
display. And aerial projection device makes the 3D the second part is aerial projection device (Fig. 4 part 2). The
hologram image projected in the mid-air. Fig.4 show the
basic element of this part is the concave mirror (parabolic
design principle of our system including two basic parts
mirror). The working principle of the concave mirror is
(Fig.4 part 1 and 2) of the aerial projection system.
shown in Fig. 7.

Figure 4. Design principle of Aerial Projection System

a. 3D Hologram Object Reconstruction
The first part is 3D object reconstruction by using
pyramid hologram device (Fig. 4 part 1). In this part a 3D
original object is built, which can be illuminated by the
parabolic mirror. Pyramid hologram is a simple device
which is made by cutting a transparent acrylic sheet into the
special shape of a hollow pyramid. The pyramid hologram
creates a 3D illusion for the viewer and makes an object to
appear in mid-air. Fig. 5 shows the working principle of the
pyramid hologram presenting a 3D hologram object.
Figure 7. Working Principle of Parabolic Mirror
In Fig. 7, a cube object is set at the bottom of the parabolic
mirror as the original object. The light (a, b and c) can
illuminate the 3D hologram object at the top of the parabolic
mirror. However, the object and the hole should be set near
the focal point both at the upper side and lower side of the
parabolic mirror, respectively. Therefore the aerial image of
the objects is focused near the hole at the upper side of the
parabolic mirror.

1932
2. Interactive 3D Hologram Object
After the 3D hologram object is created by the pyramid
hologram and parabolic mirror, the interactive aerial
projection of 3D hologram object can be done. In this
section, we describe the configuration of the interaction
aerial projection of 3D hologram object. The requirement is
divided into two parts: Non-marker sensor as finger
movement detector and the interactive 3D object
manipulation.
a. Non-marker Sensor as Finger Movement Detector
Based on the design principle of aerial projection system,
the aerial image of the 3D hologram object is projected at the
top of the parabolic mirror. However, the work space at the
top hole of the parabolic mirror has limited space with
maximum diameter 2 inch. Therefore, we use non-marker
based system, because it is favorable in a limited work space.
Also the bulk and weight of the non-marker device is
convenient to carry and setup like Leap Motion sensor.
Nevertheless, when the work space area become larger,
another sensor based on non-marker system can be
implemented such as Kinect or Xtion sensor. In this system,
we used Leap Motion sensor to detect the finger movement of
a user. The system configuration is shows in Fig. 8.
Figure 8. Interactive 3D Hologram Object System
In Fig. 8, a cube object is focused near the hole in the
upper side of the parabolic mirror. This hole is used for the
interaction (between the 3D object and user fingers
movement) by either position or direction. Leap Motion
sensor has the advantages that any position and direction of
the required finger movement can be measured.
b. Interactive 3D Object Manipulation
This part describes the reconstruction and manipulation
of the original 3D object for the aerial projection system. The
object is built by using Unity Assets programing with C#
programing language to control it. A computer gets feedback
from the Leap Motion sensor, which can detect the palm and
finger position and direction, as well as their velocity
information. Leap Motion sensor also provides preprocessed
data through the Unity programming interface. This data is
accessed by a frame object, querying per frame. There are
various properties of a frame object in the Leap Motion core
assets Orion version.
1933
Furthermore, some kind of documentation of augmented
reality is available for Unity programing. In the Unity
programing, the interaction between objects requires that at
least one object being set as rigid-body. In the interaction
process, the rigid-body must be detected when the two
objects begin to touch each other. For example, in Fig. 9, we
can see a rigid-body touching the cube object. The Unity
programing provides different collision region and some
information data. This data is usually used in some
application such as game, animation, etc. We have to choose
the most accurate type for when hands touch the object in
Unity programing to receive touching signal data from Leap
Motion sensor.
In Fig. 9, we made four scene objects in one view
position, which are four symmetrically opposite variations of
the same object and they are projected on the four faces on
the LCD. After that, the four faces are projected at the
pyramid hologram. To make easy to understand for users,
this object will change its color when user’s finger touches
the object. Also the shape of the object will change if the
depth of the finger position changes.
(a)
(b)
Figure 9. Four scene objects in one view position (white color is
before touching and blue color is after touching)
IV. CONSTRUCTION OF INTERACTIVE AERIAL PROJECTION
OF 3D HOLOGRAM OBJECT SYSTEM
A. Appearance of Interactive Aerial Projection of 3D
Hologram System
In this section we describe the components of the
interactive aerial projection of 3D hologram object system:
the pyramid hologram, parabolic mirror, and Leap Motion
sensor.
1. Aerial Projection of 3D Hologram Object
Reconstruction
a. 3D Hologram Object Reconstruction
For the reconstruction of the 3D hologram object we use
the pyramid hologram device. It is a simple device which is
made by cutting a transparent acrylic sheet into the special
shape of a hollow pyramid. Table 1 show the pyramid
hologram specification.
Table 1. The Pyramid Hologram Specifications
Material Transparent Acrylic 0.5
Upper Side 2.95 [inch]
Length
Bottom Side 0.4 [inch]
Height 3.5 [inch]
We made the pyramid hologram as featured in Fig.10.
The angle of pyramid hologram is almost 70 degrees. The
aim of this angle is to project the 3D hologram object near
the focal point of the pyramid hologram. It means that the
parabolic mirror can project the 3D hologram object more
clearly as well as focus near the hole at the upper side of the
parabolic mirror. The length of bottom side is only 0.4 inch
due to the limited size of the hole of parabolic mirror. It is
also designed to make the working space sufficient for the
object movement in the LCD display.
Figure 10. Pyramid Hologram
b. 3D Hologram Object Projection In Mid-air
The second device is the parabolic mirror. The
specifications of the parabolic mirror is shown in Table 2. It
consists outer diameter, inner diameter (hole diameter), and
height.
Table 2. Parabolic Mirror Hologram Specification
Product Desktop Mirage ®
Outer diameter 9 [inch]
Diameter
Hole diameter 2 [inch]
Height 3 [inch]
Figure 11. Parabolic Mirror Hologram Device
1934
The configuration setup between the pyramid hologram
and parabolic mirror hologram is shown in Fig.12.
Figure 12. Combination of the Pyramid Hologram and
Parabolic Mirror
2. Interactive 3D Hologram Object
The Leap Motion sensor is used to detect user’s finger
movement so that the 3D hologram object could be
interactively operated by a user.
a. Leap Motion as Finger Movement Detector
The Leap Motion sensor is especially used for detection
of the hand movement. This sensor consist of two
monochromatic IR cameras and three infrared LEDs which
can detect hands and fingers by accuracy of 0.01 mm in every
20 ms. In this system, the data of a finger position and
direction are extracted by Leap Motion sensor in every 60
ms.
Table 3. Leap Motion Specification
Depth 1.2 [inch]
Height 0.5 [inch]
Width 3.1 [inch]
Weight 1.6 [oz]
Figure 13. Leap Motion sensor dimensions
3. Appearance System of Interactive 3D Hologram Object
The combination of the pyramid hologram, concave
mirror and Leap Motion sensor as well as the PC makes up
the interactive aerial projection of 3D hologram object. Fig.
14 shows the appearance of the proposed system. The work
space of the system is about 2 inch x 2 inch x 2 inch which
depends on the hole diameter of parabolic mirror.
Figure 14. Appearance of the Interactive Aerial Projection of 3D
Hologram System
B. System Function of the Interactive Aerial Projection of
3D Hologram Object System
The interactive aerial projection of 3D hologram objects
system has two main functions: the reconstruction and aerial
projection of 3D hologram object and the interaction with the
3D hologram object.
1. 3D Hologram Object Projection in the Mid-air
The 3D hologram object shown inside the pyramid
hologram is the original object. The original image of the
object is positioned at the bottom hole, but it is formed at the
top hole, so that the 3D hologram object is projected in the
mid-air. Fig. 15 shows the interactive aerial projection of 3D
hologram objects in the mid-air.
Figure 15. Interactive Aerial Projection of 3D Hologram
Object in the Mid-air
2. Interactive aerial projection of 3D Hologram Object
In Fig. 15, the 3D hologram object is formed at the hole
of the upper side of the parabolic mirror. This object is a real
object in the mid-air. When the source of the object in the
pyramid hologram moves forward, backward, left or right,
the 3D hologram object in the mid-air will also move in the
same way while maintaining the focus on the hole in the
upper side of the parabolic mirror.
When the object is appeared, left-right direction means
the X position of the object and finger, forward-backward
direction is the Y position, and up-down direction is the Z
position. This object will change its color if the finger
touches the 3D hologram object. The shape of the 3D
hologram object will change if the Z position changes. If Z
position is higher than original position, the object will
become larger. In contrast, if Z position is lower than
original position, the object will turn to smaller. On the other
hand the 3D hologram object is moved forward-backward or
left-right following the finger position of a user.
1935
V. CONCLUSION
In this paper, we presented the interactive aerial
projection of 3D hologram objects by using the pyramid
hologram and parabolic mirror system. We set a
combination of the pyramid hologram and parabolic mirror
to reconstruct and project the 3D hologram object, and the
Leap Motion sensor as movement detector of user’s finger so
that the 3D hologram object can be interactively operated by
users. As a result, we succeeded in constructing the
interactive aerial projection of 3D hologram object.
Future prospect of this system is to make the larger work
space of hologram and to build more interactive
manipulation for remote operation which can control the
robot position and orientation.
ACKNOWLEDGMENT
This research is supported partially by Waseda
University (2016B-203) and Kayamori Foundation of
Informational Science Advancement (K26kenXIX-453), to
which we would like to express our sincere gratitude.
REFERENCES
[1] H. Ghuloum, “3D Hologram Technology in Learning Environment The
Historical Background of ICT in Education,” Proc. Inf. Sci. IT Educ.
Conf., pp. 693–704, 2010.
[2] T. Kawashima, K. Suzuki, T. Kakue, T. Shimobaba, and T. Ito, “Aerial
Projection System of Three-Dimensional Images Using
Computer-Generated Image Hologram and Concave Mirrors,” vol. 1, no.
1, pp. 1–5.
[3] T. Kakue, A. Yoshida, T. Kawashima, K. Suzuki, T. Nishitsuji, T.
Shimobaba, and T. Ito, “Real-Time Calculation and Reconstruction of
Computer- Generated Holograms for Aerial Projection System Based on
Electro-Holography with Parabolic Mirrors,” no. 1, pp. 5–7, 2015.
[4] T. Kakue, T. Nishitsuji, T. Kawashima, T. Shimobaba, and T. Ito,
“Aerial projection of three-dimensional images reconstructed from
computer-generated image holograms,” pp. 1–2.
[5] H. Sasaki, K. Yamamoto, K. Wakunami, Y. Ichihashi, R. Oi, and T.
Senoh, “Large size three-dimensional video by electronic holography
using multiple spatial light modulators,” pp. 4–11, 2014.
[6] W. Osten, C. Kohler, and J. Liesener, “Evaluation and application of
spatial light modulators for optical metrology,” Opt. Pura Apl, vol. 38,
no. 1998, pp. 228–233, 2005.
[7] K. N. Tarchanidis and J. N. Lygouras, “Data glove with a force sensor,”
IEEE Trans. Instrum. Meas., vol. 52, no. 3, pp. 984–989, 2003.
[8] W. d. Boer, A. Abileah, P. Green, T. Larsson, S. Robinson, and T.
Nguyen, “56.3: Active matrix LCD with integrated optical touch screen,”
in SID Symp. Dig. Tech. Papers, vol. 34, pp. 1494–1497, 2003.
[9] M. A. Hussein, A. S. Ali, F. A. Elmisery, and R. Mostafa, “Motion
Control of Robot by using Kinect Sensor,” vol. 8, no. 11, pp. 1384–1388,
2014.
[10] L. Shao, S. Ee, V. Reality, C. Report, I. G. Wetzstein, and R. Konrad,
“Hand movement and gesture recognition using Leap Motion
Controller,” pp. 2–6, 2015.
[11] M. Narita and T. Matsumaru, “Calligraphy - Stroke Learning Support
System Using Projection,” Proc. 24th IEEE Int. Symp. Robot Hum.
Interact. Commun., pp. 640–645, 2015.