Professional Documents
Culture Documents
Object Sensing For Fruit Ripeness Detection Using Wifi Signals
Object Sensing For Fruit Ripeness Detection Using Wifi Signals
WiFi Signals
Sheng Tan, Linghan Zhang, Jie Yang
Florida State University, Tallahassee, Florida, USA
Abstract—This paper presents FruitSense, a novel fruit maintain fruit quality through effective selling strategies based
ripeness sensing system that leverages wireless signals to enable on accurate categorization of the fruit ripeness. Monitoring
non-destructive and low-cost detection of fruit ripeness. Such a the ripeness of fruits throughout the supply chain thus can
system can reuse existing WiFi devices in homes without the
need for additional sensors. It uses WiFi signals to sense the
reduce waste as well as improve the consistency of quality for
physiological changes associated with fruit ripening for detecting consumers. Consumers also can benefit, for example, knowing
the ripeness of fruit. FruitSense leverages the larger bandwidth the degree of ripeness can avoid the unpleasantness of tart or
at 5GHz (i.e., over 600MHz) to extract the multipath-independent rotten fruit.
signal components to characterize the physiological compounds Existing approaches for inferring the ripeness of fruit mainly
of the fruit. It then measures the similarity between the extracted
features and the ones in ripeness profiles for identifying the
depend on penetrometer [16], refractometer [41], or spectrom-
ripeness level. We evaluate FruitSense in different multipath eter [58]. As the firmness and sugar content are two most
environments with two types of fruits (i.e, kiwi and avocado) widely used indicators of fruit ripeness, the food industry
under four levels of ripeness. Experimental results show that primarily relies on penetrometer and refractormeter in the
FruitSense can detect the ripeness levels of fruits with an field for the purpose of ripeness detection [31]. While the
accuracy over 90%.
penetrometer measures fruit firmness by quantifying the force
required to insert a probe into the fruit [16], refractometer
I. I NTRODUCTION
(e.g., Brix [17]) analyzes sugar content of the juice using
Recent advances in wireless technology have greatly ex- light refraction. Both methods, however, are destructive and
panded the WiFi usage from providing laptop connectivity to less acceptable to consumers. In contrast, spectrometer based
connecting mobile and smart devices to the Internet and home approach is non-destructive. It splits light signals into a fruit
networks. Such an evolution has resulted in the prevalence of and then measures the light that is emitted, absorbed or
WiFi devices, which provides opportunities to extend WiFi’s scattered by the fruit for ripeness inference [58]. However,
capabilities beyond communication, particularly in human traditional spectrometer instruments are bulky and expensive
sensing. As the wireless signals travel through space, they (i.e., costing several thousands of dollars) and are limited to
interact with human body and undergo wave phenomena controlled laboratory settings [38].
such as reflection and diffraction. These phenomena lead More recent work includes using advanced imaging tech-
to multipath effects, which carry a rich set of information niques and ultrasonic measurement systems to analyze the im-
about the physical environment including the human location age features (e.g., color and texture) and measure the ultrasonic
and activities. Indeed, there has been growing interest within attenuation of the fruit for ripeness detection [32], respectively.
the wireless and mobile computing communities in using However, these methods require specialized equipments, and
multipath effects to perform human sensing, ranging from are less accessible to ordinary consumers. Recently, due to
large scale movements (e.g., daily activities [48], [50], [14], advances in materials and fabrication techniques, portable
body movements [5], [37], [49], [54], hand gestures [26], spectrometers that work together with smartphones have been
[37]), to small scale motions (e.g., finger gestures [9], [43], realized. For example, Das et al. [13] propose a smartphone
[42], lip motions [47], breathing and heartbeats [30], [8], [14], based spectrometer that can measure UV fluorescence of
[29] and location [55], [18], [51], [46], [57], [56], [28]. Chlorophyll found in apples, whereas the company Consumer
In this work, we further expand the WiFi sensing capabilities Physics develops the sensor SCiO that can be integrated with
from human sensing to sensing bio-information of fruit crops. smartphones to analyze the molecular composition of food [2],
In particular, we seek to sense the degree of ripeness in fruits [1]. Although these portable solutions can be adapted by
with WiFi signals. Monitoring the ripeness of fruit provides consumers, there also are non-negligible costs incurred in
many benefits for several end-users, ranging from farmers, to purchasing dedicated spectral sensors (i.e., at around $200).
distributors, to retailers and consumers. It can help farmers In this paper, we introduce FruitSense, a new method for
to determine the optimal time to harvest fruit as the quality inferring the ripeness of fruit that is both non-destructive and
heavily depends on when they are harvested. It can also low-cost. FruitSense uses WiFi signals to sense the physi-
assist fruit distributors in performing rapid sorting in storage ological changes associated with fruit ripening for ripeness
facilities and deciding when to send their stock during post- detection. It enables users to reuse off-the-shelf WiFi devices
harvest period. Moreover, retailers can minimize losses and for ripeness monitoring, and thus can benefit a large number
of users, such as farmers, retailers and customers. stitch together the calibrated CSI measurements from each
In particular, fruit ripening involves a series of physiological individual channel. As the usable WiFi channels in 5GHz
changes leading to the development of a soft edible ripe fruit. are unequally and non-contiguous spaced, the inverse non-
Take avocado as an example, there are changes in total dry uniform Discrete Fourier Transform (NDFT) is used to derive
matter and moisture content during maturation and ripening. the fine-grained power delay profile for multipath removal.
As moisture content decreases, the dry matter increases ac- Next, we identify and extract the signals that directly went
cordingly. When wireless signals travel through a fruit, the through the fruit for ripeness detection. At last, we use the
changes in the physiological compounds of the fruit during Maximal Overlap Discrete Wavelet Transform (MODWT) to
ripening lead to distinct and measurable effects on the received extract signal features over multiple channels and compare
signals. We thus are able to infer the ripeness level based on the features against known ripeness profiles that identify the
the physiological changes interpreted by the received signals. degree of ripeness.
Accurately discerning the ripeness of fruit is challenging, We evaluate FruitSense in different multipath environments
however, when using a single pair of off-the-shelf WiFi de- with two types of fruits: kiwi and avocado. The fruit samples
vices. First, the measurable changes in the received signals due are purchased from two different venders with different levels
to fruit ripening are subtle, and the impact of fruit size on the of ripeness. The volume of each type of fruit is about 75 for
received signals might distort such changes. To address these each level of ripeness. We also experience the scenarios where
issues, we leverage frequency diversity by probing the fruit people are moving around and serving as multipath interfer-
with WiFi signals at multiple frequency channels. Leveraging ences during ripeness detection. We identify the ripeness of
frequency diversity not only provides a rich set of information fruit as one of the four levels: unripen, half ripen, ripen and
to capture the subtle changes, but also enables us to examine over ripen. Experimental results show that FruitSense is highly
the change pattern among multiple channels, instead of a effective in detecting fruit ripeness. It achieves an accuracy at
single channel, to mitigate the impact of fruit size. over 90% in identifying the ripeness level of fruit.
Second, the signal propagation is dominated by multipath The main contributions of our work are summarized as
in typical indoor environments when the line-of-sight (LOS) follows:
is blocked by the fruit that is located between the transmitter • We show that the WiFi signals can be utilized to capture
and the receiver during the ripeness sensing process. While the physiological changes of fruit for ripeness detection.
the WiFi based human sensing primarily relies on the signal Our system is non-destructive and low-cost. It allows
reflections, the reflected signals from surroundings represent reusing existing WiFi devices at homes without the need
interferences to the signal component indicating the ripeness of for additional sensors.
fruit. This is because once the indoor environment changes, the • We leverage the frequency diversity to accurately capture
changes in the received signals mainly reflect the differences the physiological changes of the fruit due to ripening, and
in multipath propagation instead of the physiological change utilize the larger bandwidth at 5GHz to combat multipath
of the fruit. To accurately capture the changes corresponding for extracting multipath-independent signal components
to fruit ripening, we propose to isolate the signal component for ripeness detection. We further utilize MODWT to
traveling through the fruit directly, from the ones reflected with extract features over multiple frequencies for identifying
longer paths. Specifically, we leverage the larger bandwidth at the level of fruit ripeness.
5GHz (i.e., over 600MHz) to derive fine-grained power delay • We conduct extensive experiments in different multipath
profile for multipath removal. environments with two typical fruits (i.e., kiwi and av-
Third, due to unavoidable residual synchronization errors ocado) under various conditions. Experimental results
of the off-the-shelf WiFi NICs, the sampled phase of the show that FruitSense achieves around 90% ripeness level
channel frequency response contains linear phase errors, which detection accuracy.
result in significant distortion in the derived power delay II. P RELIMINARIES
profile. To calibrate the phase errors, we utilize the fact that
A. Fruit Ripening
the power delay profile derived from each individual channel
should be identical as long as the probing packets across these There exists two types of fruits according to the regulatory
channels were sent out within coherence time. We thus propose mechanisms underlying their ripening process: climacteric
to formulate the phase error calibration as an optimization and non-climacteric fruits. Climacteric fruits, such as kiwi,
problem, in which we search for the best phase compensation avocado, apple and banana will continue to ripen after the
that minimizes the differences in derived power delay profiles fruit has left the plant; however, non-climacteric fruits, such
across multiple channels. as grape, orange and pineapple stop the ripening process the
More specifically, our system first probes the fruit with minute they leave the plant [36]. In this work, we focus on
WiFi signals hopping at all available 5GHz channels. The sensing the ripeness of climacteric fruits, as we do not have
sampled channel frequency response, which is exported by the access to the non-climacteric fruits that are on the plant.
the WiFi NICs in the form of Channel State Information Nevertheless, our system can be extended to sense the maturity
(CSI), then goes through the calibration process to correct of non-climacteric fruits during the growth and development
errors due to hardware limitation of WiFi NICs. We then process while they are on the plant.
Path2 Refraction
Reflection Path2
Path1
TX RX Path1
Power
Path3
Path3 Path4
Path1
Path4
TX RX
0 τ τ + 1.5 τ + 3 τ + 4.5
Time(ns)
Fig. 1. Multipath propagations in an indoor environment during fruit Fig. 2. Combining all the channels at 5GHz provides a power delay
ripeness sensing. profile with sufficient resolution to differentiate multipath propagations.
Fruit ripening is a highly co-ordinated and an irre- (i.e, the fruit in our case). In particular, the complex relative
versible phenomenon involving a series of physiological and permittivity ε∗ of a material to that of free space in frequency
organoleptic changes [36]. For example, some fruits, such as domain can be described as following [27]:
banana and apple, come in a wide array of colors that change
ε∗ = ε − jε
I II
throughout their ripening process, with the brightest colors (1)
often occurring when the fruit is optimally ripened. Such color I
changes enable people to predict the ripeness level based on The real part ε is referred to as the dielectric constant, which
external visual inspection. By contrast, some other fruits, like describes the ability of the material to store energyII
when it is
kiwi and avocado, do not exhibit obvious organoleptic changes exposed to an electric field. The imaginary part ε is referred
√
during the ripening process. It is hard for ordinary people to to as dielectric loss factor, and j = −1. The dielectric loss
tell the ripeness level based on organoleptic testing. We thus factor influences both energy attenuation and absorption, and
experience with the ripeness detection of kiwi and avocado in is commonly used to describe the ability of the material to
this work. We seek to sense the fruit ripeness based on their dissipate electrical energy as heat.
physiological changes, instead of organoleptic changes. As fruit ripening involves a series of physiological changes,
Existing work uses the physiological changes such as oil fruit at different ripeness levels results in different dielectric
context, dry matter and moisture content to determine the constants and loss factors. The permittivity of the fruit thus
ripeness of avocado [31]. Specifically, oil content and dry mat- could be used as an indicator of its internal quality. Indeed,
ter increase during development and continue to change during there exists prior work on using dielectric spectroscopy to
ripening [31]. As oil content increases, the moisture content measure the dielectric properties of fruit and vegetable for
decreases by the same amount, so that the total percentage internal quality analysis[25].
of oil and moisture content remains constant. Similarly, the As we use the off-the-shelf WiFi devices, we turn to
content change within dry matter can be used to track the exam how the received signal changes due to the changed
ripeness of kiwi fruit [36]. In particular, dry matter of kiwi physiological compounds of the fruit. Specifically, when the
is mainly comprised by the carbohydrates and starch, which WiFi signal travels through the fruit, the electric field strength
gradually transform to soluble solids such as sugar content decreases with the distance from its surface. To quantify such
during the ripening. The percentage of carbohydrates and an effect, the attenuation factor α, which depends on the
starch thus could be used as an indicator of fruit ripeness. dielectric properties of the fruit, could be leveraged. It is given
B. WiFi Sensing based Approach by: (
\1
Figure 1 illustrates a typical experimental setup of sensing 2π 1 1 ε (I1 + εII \2 1 1/2 (2)
α= I
λ0 2 −
fruit ripeness using a pair of WiFi devices in an indoor εI
environment. The testing fruit is placed in between a pair of where λ0 is the free-space wavelength of the WiFi sig-
closely spaced transmitter and receiver, thus blocking the line- nal [27]. Based on Equation 2, we know that the fruits
of-sight (LOS) signal propagation. We rely on the received at different ripeness levels lead to different attenuation fac-
signal component that directly travels through the fruit to tors, which could be measured by analyzing the received
sense its physiological change. The phenomenon where radio signals.Therefore, instead of using dedicated dielectric spec-
waves travel through and modulated by the fruit is commonly troscopy, we leverage the received signal that directly travels
referred as refraction, which describes the signal passes from through the fruit for ripeness detection.
one medium to another. In our case, it is the WiFi signal travels Excepting for the controllable experimental settings such
from air to fruit, and then from fruit to air. as the location of the fruit and the distance between the
To quantify the effect of the refraction, we could leverage transmitter and the receiver, the sizes of the fruit could be
the concept of permittivity, which is a measure of how an slightly different, which may affect the signal attenuation as
electric field affects, and is affected by, a dielectric medium well. Note that from Equation 2, we can observe that the WiFi
signals at different frequencies (wavelengths) result in different different countries before shipping the WiFi NICs. We thus
attenuation factors as well. We thus can leverage the frequency cannot simply use IFFT, which only works for uniformly-
diversity by probing the fruit at multiple channels, and then spaced frequency measurements. To overcome this issue, we
analyze the relative changes between multiple frequencies to adopt the inverse non-uniform Discrete Fourier Transform
mitigate the impact of fruit size. (NDFT), which is capable of deriving a fine-grained power
delay profile from non-uniformly spaced channels at 5GHz.
C. Challenges Another challenge lies in the use of off-the-shelf WiFi
While the intuition is simple, there are significant challenges devices. Although the WiFi NIC employs sampling frequency
to accurately extracting the received signal that directly travels offset (SFO) and carrier frequency offset (CFO) correctors to
through the fruit. As shown in Figure 1, besides the signal compensate the frequency offset errors, there still exists sig-
refraction, there exists signal reflections. The signals that nificant residual synchronization errors [20]. Figure 3 depicts
reflected from the walls, furniture, and human body will be the raw phases of two packets extracted at the NICs for the
combined with the signal travels through the fruit at the same multipath channel. Figure 3 (b) and (c) illustrate the
receiver. This leads to the fact that the measured signals at the derived power delay profiles with the raw phases presented in
receiver mainly reflect the multipath environments. To make Figure 3 (a). Ideally, these two derived power delay profiles
the system robust to the multipath environment, we need to should be identical as they reflect the same multipath channel.
separate the signal component that travels through the fruit However, Figure 3 (b) and (c) display the opposite observation:
from the reflected ones. one profile exhibits LOS propagation while the other shows
Intuitively, this can be done by leveraging the power delay no LOS propagation. In particular, Figure 3 (b) shows the first
profile, which gives the power intensity of received signals arriving signal has the strongest power, whereas Figure 3 (c)
as a function of propagation delay. By performing Inverse depicts the first arriving signal is weaker than the reflected
Fast Fourier Transform (IFFT) of the received signal mea- ones. To correct such an error, we utilize the fact that the
surements, we are able to extract the first arriving signal, power delay profiles derived from different WiFi channels
which travels through the shortest path among all the paths should be identical as long as the probe packets were sent out
from the transmitter to the receiver. However, the widely used within a coherence time. We thus could further compensate
bandwidth of a WiFi channel is either 20MHz or 40MHz, the frequency offset errors by searching for the best phase
which results in a power delay profile with resolution at either compensation that minimizes the differences in derived power
50ns or 25ns. Given that the wireless signal travels at the speed delay profiles across multiple channels.
of light, such resolutions correspond to distance resolutions of
15m and 7.5m, respectively. As the size of a typical room is III. S YSTEM D ESIGN
several meters by several meters, a majority of the reflected
A. System Overview
signals have path lengths smaller than 15m or 7.5m. Therefore,
the obtained first arriving signal from the power delay profile Our system uses a pair of WiFi devices to sense the
based on each channel is still a mixture of the signals that physiological compounds of fruit for ripeness detection. The
travel through the fruit and multipaths. Therefore, simply major advantage of our method is that it is non-destructive and
performing IFFT on the signal measurements at each WiFi allows users to reuse existing WiFi devices without additional
channel provides insufficient time or distance resolution for sensors. Figure 5 illustrates the flow of our system. It first
extracting the signal traveling through the fruit. probs the fruit with WiFi signals hopping through all usable
As WiFi spans multiple channels at both 2.4GHz and 5GHz, channels at 5GHz. The system then collects the sampled
we propose to probe the fruit at all available channels of channel frequency response in the form of Channel State
5GHz with larger bandwidth. Combining all the channels at Information (CSI) including the information of phase and
5GHz (i.e., from 5.18GHz to 5.825GHz) brings over 600MHz amplitude. The CSI measurements are reported by the WiFi
bandwidth, which corresponds to a 1.5ns power delay profile NIC at the receiver.
resolution or to a 0.45 meters distance resolution. Such a The CSI measurements are then preprocessed to calibrate
resolution is sufficient for us to separate the signal traveling both the phase and amplitude errors. In particular, the raw
directly through the fruit from the reflected ones, as shown phase contains residual synchronization errors, which are com-
in Figure 2. Given the signal measurements at each channel, posed of two types of errors: linear errors with respect to the
we then stitch these measurements together to derive a fine- subcarrier indexes and the constant errors across subcarriers.
grained power delay profile for multipath removal. Our system calibrates the linear phase errors by searching
One problem with that stitching lies in the fact that the us- for an optimum phase compensation value that minimizes
able WiFi channels at 5GHz are unequally and non-contiguous the differences in the derived power delay profiles across
spaced due to FCC regulation, as shown in Figure 4. For multiple channels. The constant phase errors and amplitude
example, the channels from 120 to 128 are partially occupied errors could be mitigated by averaging the measurements of
by the weather radar usage in the US, and different countries multiple packets that received in the coherence time window.
apply their own regulations. Venders usually disable some Given the calibrated CSI measurements, our system per-
of the 5GHz channels in compliance with the regulations of forms multipath removal for extracting the signal that directly
Power Level(db)
-1
Power Level(db)
Phase(radian)
Packet 1 40 40
Packet 2
-2 20 20
0 0
-3
0 20 40 60 0 0.25 0.5 0.75 1 0 0.25 0.5 0.75 1
Subcarrier Index Time Delay(ms) Time Delay(ms)
(a) Raw CSI phases (b) Power delay profile 1 (c) Power delay profile 2
Fig. 3. Raw phases of two CSI measurements and the corresponding derived power delay profiles.
Phase(radian)
Channel 1 Channel 40
Channel 3 0 Channel 44
Channel 5 Channel 48
0 -0.5
-1
-1 -1.5
0 20 40 60 80 100 120 0 50 100 150 200
Subcarrier Index Subcarrier Index
(a) CSI Phases after calibration at 2.4GHz (b) CSI Phases after calibration at 5GHz
Fig. 6. Calibrated CSI phases for three overlapping channels at 2.4GHz and three channels at 5GHz.
where / hp,q
k represents the phase rotation due to signal caused by imperfectly synchronized clock at both transmitter
propagation. Meanwhile both ϕl and ϕc represent the residual and receiver ends. Even after phase compensation at WiFi
synchronization errors of phase. NICs, there are still residual synchronization errors exist. In
particular, it can be further divided into two components:
C. Data Calibration
ϕl = ϕd + ϕ s , (8)
As our system relies on the CSI measurements to derive
fine-grained power delay profile for multipath removal, we first The packet boundary detection could introduce uncertain time
discover how the errors presented in the CSI measurements shift, which causes the phase error ϕd. Such time shifts
affect the derived power delay profile. First, different from the in reported CSI measurements are observed to follow the
phase errors, the error of the amplitude doesn’t affect the over- Gaussian distribution with zero mean [40]. Our experimental
all shape (or energy distribution with respect to time delay) of data also confirms such an observation. Furthermore, the
the power delay profile. The amplitude error is mainly caused error ϕd stays the same across subcarriers within a channel
by the power amplifier uncertainty. As the amplitude error but changes across different channels. Because ϕd follows
is discovered to follow Gaussian distribution [23], it could Gaussian distribution with zero mean, we are able to mitigate
be mitigated by averaging CSI measurements from multiple the error ϕd by averaging multiple CSI phase measurements
packets that within the coherence time. Among different types collected at each channel.
of phase errors, the constant phase error ϕc, is caused by the The second component of linear error ϕs is introduced
residual central frequency offset. It does not affect the energy by the residual sampling frequency offset due to the unsyn-
distribution with respect to time delay in the power delay chronized clock at the transmitter and the receiver. Although
profile as well. This is because the constant phase error is the clock skews between different transmission pairs vary
frequency independent. When performing IFFT, the phase shift significantly, the skew remains constant for a few minutes
ϕc would cause a constant rotation term for all the attenuation for the same pair of transmitter and receiver [21]. Since our
at different propagation delays. Thus, the derived power delay system probs the fruit with a single pair of WiFi device within
profile would stay the same after such a shift. Based on one second, the linear error ϕs would remain constant across
this observation, we select a reference channel from all the different packets and channels. To remove ϕs, we utilize the
available channels and mitigate ϕc through correcting the fact that the power delay profiles derived from different WiFi
phase difference from each channel according to the reference. channels should be identical, if the probe packets were sent out
The power delay profile distortion presented in Figure 3 within the coherence time. This is because the measurements
is mainly caused by the the linear phase error ϕl, which is of these packets from multiple channels reflect the same
a frequency dependent error. We next detail how we correct multipath environment. The phase error ϕs calibration thus
the linear phase error ϕl by using the CSI measurements could be formulated as an optimization problem, in which we
across multiple channels. The linear phase error ϕl is mainly search for the best phase compensation value that minimizes
55 40
Amplitude(db)
Power(db)
50
Channel 136 20
45 Channel 140
Channel 149
Channel 153
400 100 200 300 400 00 5 10 15 20 25
Subcarrier Index Time Delay(ns)
Amplitude(db) 50
45
40
Fig. 7. The CSI amplitude for four channels with gap at 5GHz before and after multipath removal.
the differences in derived power delay profiles across multiple together. Figure 6(b) shows three non-overlapping channels at
channels. 5GHz band after phase calibration.
Given the channel response, the power delay profile g at
D. Multipath Removal
channel q can be derived using IFFT:
L Multipath removal leverages a fine-grained power delay
'\" profile to extract the signal that directly travels through the
gq = alδ(t − tl), (9)
fruit for ripeness detection. Since a power delay profile derived
l=1
from one single WiFi channel provides insufficient time delay
where l denotes the sequence number of total L multipath resolution, our system stitches all the available channels at
channels, al and tl are the amplitude and signal propagation 5GHz together to improve the resolution for extracting the
time delay of lth path, δ(t) is the Dirac delta function. signal that travels through the shortest path.
After removing phase errors ϕc and ϕd and assuming the the Due to the regulations at many countries, the usable chan-
I
phase compensation for ϕs is ϕ s, Equation 7 can be rewritten nels at 5GHz are unequally and non-contiguous spaced. For
as: example, the usable channels at 5GHz on the Atheros NICs
I
/ c-sik = / k + kϕ − kϕ . (10) are separated into four segments: from 5.18GHz to 5.32 GHz,
p,q p,q s s
h I from 5.5GHz to 5.58 GHz, from 5.66GHz to 5.7 GHz, and
By gradually changing the value of ϕ s , different power delay
from 5.745GHz to 5.825GHz. The venders disabled the rest
profiles can be derived for each channel. The optimum phase
channels in compliance with the local regulations. We thus
compensation ϕs can be found when the differences in derived
cannot simply use IFFT for power delay profile derivation,
power delay profiles across multiple channels are minimized.
as it only works for uniformly-spaced frequency measure-
In particular, it can be formulated as an optimization problem
as following: ments. Instead, we adopt inverse Non-uniform Discrete Fourier
Transform (NDFT), which works for non-uniformly spaced
Q
channels. For fine-grained power profile derivation, given the
min '\" ||gq (ϕ ) − gqI (ϕs)|| , q /:= q , (11) calibrated CSI measurements from last step as:
I I
I 2
I
s CSI = [C-SI1 , ..., C-SIq , ...], (12)
ϕs I
q,q =1
I
where gq(ϕs) denotes the derived power delay profile at
the qth channel and Q is the total number of channels. By where q denotes the qth channel, we can formulate the inverse
I NDFT as following:
obtaining an optimum value of ϕ s, we are able to remove the
linear phase error ϕs from the CSI phase measurements. min ||CSI − F g|| 2, (13)
Figure 6 shows the phases across different channels after g
calibrating ϕs. As shown in Figure 6(a), after calibration, the where g represents the power delay profile that we search for,
phases of the overlapped subcarriers at three different 2.4GHz F is Fourier matrix. The goal is to find an optimum solution
band channels now demonstrate similarity and consistency. of g that minimizes the difference between CSI and Fourier
Thus we can pick either one channel as reference to stitch them Transform of g.
Such an optimization problem can be viewed as an under-
determined system, which would yield several possible solu-
Fruit Kiwi Avocado
tions. In order to pick out the best one, we need to add other
Ripeness Vendor 1 Vendor 2 Vendor 1 Vendor 2
constraints to filter out the less desired ones. To find such con-
straints, we look into the characteristic of signal propagation Unripen 48 32 45 35
in indoor environment. Based on previous observations [10], Half Ripen 43 31 43 33
even though multiple paths exist in typical indoor environment, Ripen 44 32 40 34
only a few paths would dominate the signal propagation. Over Ripen 42 30 44 34
It is because they travel through shorter paths and suffer
less attenuations when comparing to longer propagations. By
utilizing this observation, we add one constraint to the inverse Fig. 8. Volumes of fruits in our evaluation.
NDFT: among all the solutions of g that satisfy Equation 13,
our system favors the g with fewer dominating propagation
paths.
To solve the Equation 13 with the constraint, we adopt
the proximal gradient method that used to solve convex
optimization problem [19]. To be more specific, our system
takes CSI as input and computes the gradient of differentiable
term in Equation 13. After obtaining several possible solutions, Half Ripen Ripen Half Ripen Ripen
our system selects the one that with fewer dominating paths.
Given the derived fine-grained power delay profile, we remove
Fig. 9. Avocado and kiwi at two different ripeness levels.
the components from multipath propagations and only keep the
component that goes through the fruit directly. Then, we covert
the trimmed power delay profile back to frequency domain for that would be otherwise discarded by the DWT. Moreover, the
feature extraction. MODWT is shift invariant, which means the decomposition
Figure 7 shows the amplitude of the channels cover the output stay consistent across varies starting sample. This
same bandwidth before and after multipath removal process. enables us to retrieve consistent features across a large number
As shown in Figure 7(a), four channels at 5GHz band cover of channels. Given a signal X(i), each level of MODWT
100MHz bandwidth with a 40MHz gap in between as high- coefficients are computed based on the following equations:
'\"
lighted by a black dot rectangle. After performing inverse Wj,i = l = 0L−1hj,lXi lmodN ,
NDFT, we are able to derive a fine-grained power delay −
'\" L−1
(14)
profile shown in Figure 7(b), which indicated the lack of line Vj,i = l=0 gj,lXi −lmodN ,
of sight. Then we remove the multipath components from
the power delay profile, only reserve the signal propagation where Vj,i is the approximation coefficient and Wj,i is the
affect by fruit and convert it back to frequency domain. The detailed coefficient. Here hj,l and gj,l are the wavelet filter
resulting spectrum is shown in Figure 7(c). We observe that the and scaling filter respectively. j is the level of decomposition
and l = 1...L is the length of the filter. By applying MODWT
amplitude continue to decrease with increasing the frequency.
to the extracted signal at each channel, we obtain both large
It is consistent with Equation 2, where the attenuation factor
scale and detailed features.
increases with frequency. It means that a higher frequency
To identify the degree of ripeness, we utilize cross correla-
would suffer a larger attenuation when propagating through
tion to calculate the similarity between the extracted features
the fruit.
and the ones in the profile library. Cross correlation measures
E. Feature Extraction and Ripeness Identification the similarity based on the relative changes and is independent
of translations and scaling in the amplitude. The ripeness level
Our system next uses Maximal Overlap Discrete Wavelet
with the profile in the library that has the highest similarity
Transform (MODWT) to extract features based on the sig-
with the testing fruit is then identified as the detected ripeness
nals that directly travel through fruit at multiple frequencies
level.
(i.e., spectrum). Similar to discrete wavelet transform(DWT),
MODWT analyzes the signals in both time and frequency IV. P ERFORMANCE EVALUATION
domains by decomposing signals into successive approxima- In this section, we describe our experimental setup and eval-
tion coefficients along with detailed coefficients. The approx- uate the performance of FruitSense in detecting the ripeness
imation coefficients depict the large scale characteristic of level of Kiwi fruit and Avocado under different multipath
change pattern, whereas the detailed coefficients capture small environments.
scale components that represent the fine details of the change
pattern. A. Experiment Setup
Unlike DWT, the MODWT is a nonorthogonal transform, We conduct experiments with two laptops (i.e., Dell LAT-
which retains more information during decomposing process ITUDE E5540), each connecting with an external antenna.
The distance between two external antennas is about 20cm. 8’
card. And they fall into four non-contiguous segments. The WiFi Device
Fruit
first segment is from 5.18GHz to 5.32 GHz (i.e., the channels Location
13’
Detection Accuray
Unripen 0.06 0.03 0.03
0.8
Half Ripen 0.03 0.03 0.02 0.6
Improvement(%)
Unripen 0.06 0.03 0.03
Fig. 11. Confusion matrix of ripeness level detection for both kiwi and Fig. 13. Detection accuracy and improvement after separating the small and
avocado fruits. the large size fruits.
Detection Accuracy
1
the performance when the testing fruits have similar sizes as
0.95 that of the fruits used to build ripeness profiles. In particular,
the size of kiwi varies from 55mm to 63mm, whereas it is
0.9 from 69mm to 85mm for avocado. We classify each type of
fruit into two categories: small fruits and large fruits. Then,
0.85 we sense the ripeness of small (large) fruits with the profiles
Kiwi
0.8 Avocado built from the small (large) size fruits. Figure 13 shows the the
1 Day 2 Days 3 Days 4 Days
overall accuracy for separating small and large size fruits and
corresponding improvement with respect to mixed size fruits.
Fig. 12. Detection accuracy using the data collected from multiple days. We find that our system achieves slightly better performance
at each ripeness level detection for both kiwi and avocado.
In particular, it achieves an overall accuracy at around 92%
collected from a single day, two days, three and four days for for both kiwi and avocado. The overall improvement is at
both kiwi and avocado. For both kiwi and avocado, we observe around 2% when comparing to that of the mixed fruit case. The
that the performance increases when increasing the number results show that our system is not very sensitive to the fruit
of dates where the data are collected from. In particular, the size, although a more fine-grained fruit profile could slightly
detection accuracy increases from 93% to 97%, when the improve the accuracy.
collected data are increased from two days to four days for
kiwi. Similarly, the detection accuracy is increased from 94% D. Detailed Study on Multipath Environments
to 98% for avocado when the data is increased from two Although our overall performance in Section IV-B shows
days to four days. Overall, comparing to use data collected that our system works for different multipath environments,
from one day, the system achieves over 8% improvement by we further perform a more detailed study on whether the per-
combing the data from four days. Such an observation could formance changes when using the ripeness profile built at one
benefit the venders or distributors who constantly tracking and multipath environment to test the fruits at a different multipath
monitoring the ripeness of fruit for much longer period than environment. Specifically, we build the ripeness profiles when
that of customers. the fruits are at the location 1 in Figure 10. We then use such
profiles to detect the fruits that placed at the location 2 (in the
C. Impact of Fruit Size same room), and location 3 (in a different room). Figure 14(a)
In last section, the ripeness profiles are built by averaging and Figure 14(b) present the performance of ripeness detection
the features from the fruits with various sizes. We now evaluate for kiwi and avocado, respectively. We observe that our system
Detection Accuracy
Detection Accuracy 1 1
0.8 0.8
0.6 0.6
0.4 0.4
0.2 2nd Environment 0.2 Kiwi
3rd Environment Avocado
0 0
Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen
(a) Kiwi (a) Overall accuracy
Detection Accuracy
1 5
Improvement(%)
0.8 4
0.6 3
0.4 2
0.2 2nd Environment 1 Kiwi
3rd Environment Avocado
0 0
Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen
(b) Avocado (b) Accuracy improvement with respect to mixed-producing location
Fig. 14. Detection accuracy under two multipath environments when using Fig. 15. Detection accuracy and improvement after separating fruit based on
the ripeness profiles built under a different multipath environment. producing location.
Detection Accuracy
1
does not suffer from performance degradation due to the
training and testing environments are under different multipath
propagation. In particular, the overall detection accuracy still
0.5
maintains at around 90% for both kiwi and avocado. These
results further demonstrate the effectiveness of the multipath
Kiwifruit
removal that leverages the larger bandwidth at 5GHz. It shows Avocado
that our fruit ripeness detection system is location independent 0
Unripen Half Ripen Ripen Over Ripen
and environment independent. Once the profile is built, it can
be applied to different environments without profile updating Fig. 16. Detection accuracy when people walking around during ripeness
or calibration. detection.