See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/252002989

Food volume estimation from a single image using virtual reality technology

Article · April 2011

DOI: 10.1109/NEBC.2011.5778625

CITATIONS READS
22 184

6 authors, including:

Yongquan Yang John Fernstrom

Ocean University of China University of Pittsburgh
28 PUBLICATIONS 270 CITATIONS 225 PUBLICATIONS 14,909 CITATIONS

SEE PROFILE SEE PROFILE

Wenyan Jia Mingui Sun

University of Pittsburgh University of Pittsburgh
139 PUBLICATIONS 1,998 CITATIONS 376 PUBLICATIONS 4,892 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

eButton View project

Real-time image enhancement View project

All content following this page was uploaded by Wenyan Jia on 21 July 2022.

The user has requested enhancement of the downloaded file.

 Food Volume Estimation from a Single Image
Using Virtual Reality Technology
Zhengnan Zhang1,2, Yongquan Yang2, Yaofeng Yue1,2, John D. Fernstrom3, Wenyan Jia2, Mingui Sun1,2
Departments of 1Electrical & Computer Engineering, 2Neurosurgery, 3Psychiatry, University of Pittsburgh
Abstract—In dietary studies, an accurate tool for diet assessment
is highly required. In this paper, we present a new approach to
the estimation of the food volume from a single input image
based on the virtual reality (VR) technology. A virtual reality
model is built for the estimation process and an algorithm is
developed for the calculation of food volume. Experimental
results have indicated high accuracy and robustness in the food
volume estimation.
I. INTRODUCTION
Obesity has become an epidemic that threatens the health of
millions of Americans and costs billions of dollars in health
care. In studies of obesity and its potential treatments, accurate
dietary assessment is essential. Currently, the most common
type of dietary assessment used is derived from self-reports by
respondents [1]. However, this method is subject to significant
error because it depends on the accuracy of respondents’
memories and their willingness to report their true dietary
intake. With recent rapid advances in the fields of image
processing and computer vision techniques, some approaches
have been developed for volume estimation using image-based
computational algorithm to monitor dietary intake [2].
This paper proposed a new approach based on Virtual
Realty (VR) to accurately estimate food volume. The basic
principle of this approach is using computer vision and VR
technology to simulate the real world [3], in where a number
of 3D wireframe objects could be built to fit specific food
items in digital images. Different with traditional processes,
our approach estimated specific food volume directly,
producing accurate estimates of the volume of food from a
single image.
II. METHOD
The basic idea of our approach is creating virtual objects in
virtual world to simulate the real subjects from the real world
based on a single image. A virtual camera was constructed to
Fig.1. Description of application of the VR approach.
This work was supported by National Institutes of Health grant No. U01 HL91736.
978-1-61284-8928-0/11/$26.00 ©2011 IEEE
Authorized licensed use limited to: University of Pittsburgh. Downloaded on July 21,2022 at 14:23:21 UTC from IEEE Xplore. Restrictions apply.
Fig.2. Principle of the VR model for volume estimation.
simulate the real camera which capturing the real subjects.
These two cameras have coincided optical centers and image
planes. Both of these two camera models satisfies the classic
pin-hole camera model and perspective projection. Inside the
virtual world, virtual wireframe can be built to simulate the
interested object in real world. When the images of virtual
object and real object overlapped, the volume of the real
object can be estimated by the volume of the virtual wireframe
and scale ratio between the real world and the virtual world.
In real application, we set the digital food image as the
background of the scene for the virtual camera, and then the
virtual wireframe will be constructed between the virtual
camera’s image plane and the background as shown in Fig.1.
Through setting the virtual camera’s intrinsic parameters as
same as the real camera, an integrated image of real world and
virtual world could be produced and then the simulation and
estimation process can be operated as the following
description.
A. Construction of virtual world
Given a digital food image, we use a checkerboard
approach to calculate the intrinsic and extrinsic parameters of
the real camera model, and then constructed a virtual camera
with the same intrinsic parameters as the real one, so we start
from the point that we have already had the two cameras that
are identical and coincide at point  in Fig.2. The real world
coordinate frame ’ was constructed by the checkerboard
approach, in where the experiment surface was described as
the plane ′′′ and the camera target point (intersection of
the camera optical line and the experiment surface) was
indicated as the origin ′. Now, a virtual camera model should
be constructed and combined with the real camera model. We
set the origin  of the virtual coordinate frame, which is also
the virtual camera’s target point, on the real camera optical
axis ′, satisfies
 , 1 >  > 0
 = ′

 Then, based on the known point , we construct the virtual
frame , where the three principle axis , and  are parallel
with the frame ′ principle axis ′, ′and ′ of frame ′,
respectively.
B. Virtual wireframe construction and real subject volume
estimation
In order to obtain a perfect fitting projection on the image
Fig.3. Samples in the experiment. The LEGO
plane, the virtual wireframe and the real subject have a block is used for camera calibration.
proportional relationship. Considering a generic point of the
virtual wireframe with coordinates and its corresponding
generic point ’ of the real object with the coordinates ′ ′ in
order to have overlapped projections of these two points on
the image plane, points and ’ satisfies

 
=    =     =    (1) Fig.4. The effects of images after volume estimation.
 

integrated images after volume estimation are shown in Fig.4
Satisfying the above constrains, the image of virtual wireframe and the results of volume estimation are listed in Table1.
will perfectly fit the image of the real object, and the volume Experimental studies indicated that out system very accurately
′ of real object can be written as estimated the volume of foods.

′ =  ′ ′ ′

IV. CONCLUSION AND DISCUSSION
′ In this paper, we present a new approach for estimating the
Using Jacobian matrix to change the variables, we have volume of food from a single input image based on virtual
1⁄ 0 0 reality (VR) technology. A volumetric correspondence
 ′ =      0 1⁄ 0  = /  between virtual wireframes and real world food objects has
 0 0 1⁄ been mathematically proved. Algorithms for volume
where  indicates the volume of the virtual wireframe. estimation based on the VR mathematical model have been
Since we have set the table surface as the plane ’’ ’ and developed. Experiment results proved this VR model is
set the axis ′ to be perpendicular to the table surface in the satisfactory for dietary studies.
upward direction, the generic point ’ of the real object on the REFERENCES
table surface has a positive ′ value. Thus, according to (1)
[1] T. Trabulsi and D.A. Schoeller, “Evaluation of dietary assessment
and  > 0, it is necessary to set the Z value of any point of the instruments against doubly labeled water, a biomarker of habitual energy
virtual wireframe larger than or equal to 0, which means that intake,” Am. J. Physiol. Endocrimol. Metab. 2001; 281(5):E891-E899.
we “put” the virtual wireframe on the “virtual table surface”, [2] A.B. Koc, “Determination of watermelon volume using ellipsoid
the plane  . approximation and image processing,” Postharvest Biology and
Technology, Sep. 2007; 45:366-371.
A question arises whether we can construct a virtual [3] G.C. Burdea, P. Coiffet, Virtual Reality Technology, 2nd Ed, John Wiley
wireframe satisfying (1). The answer is yes. The reason lies in & Sons, New York, NY, 2003.
that the software only provides typical geometric objects to
estimate the volume of the real food in a digital image. When
we adjust the virtual wireframe in the virtual world until Table.1. Means and STDs of relative errors in volume estimates for eighteen
comprehensively the images of the virtual wireframe and an items obtained from four different camera positions.
interested real object are overlapped, we can guarantee that Subject Mean and STD of Subject Mean and STD of
their relationship will satisfy (1). The reason is that we use relative errors (%) relative errors (%)
Small Box -1.13 ± 3.86 Grapefruit -2.41 ± 4.89
typical geometric object to fit an approximately typical
Big Box -1.21 ± 2.63 Hamburger -3.49 ± 4.82
geometric object. If they did not satisfy (1), the virtual Bread -3.82 ± 4.40 Ice Cream -3.10 ± 5.57
wireframe would be an anomalistic 3D object which is in Cornbread -8.46 ± 2.97 Jello 6.06 ± 6.06
conflict with the fact that the software only provides typical Creamcake 4.90 ± 6.77 Juice 4.53 ± 2.51
geometric objects. Chicken -22.77 ± 2.61 Milk 9.14 ± 2.06
Rice -18.31 ± 7.95 Onion Slice -16.82 ± 3.36
III. RESULTS Potato 16.77 ± 5.06 Orange 3.78 ± 1.54
Egg -12.90 ± 2.50 Peach 0.54 ± 5.38
Experiments using eighteen regularly shaped objects and
food replicas were designed to test the performance of the
proposed approach and system. All the samples are shown in
Fig.3. The volumes of subjects have been estimated based on
readings from four different camera positions. The effects of

Authorized licensed use limited to: University of Pittsburgh. Downloaded on July 21,2022 at 14:23:21 UTC from IEEE Xplore. Restrictions apply.
View publication stats