Received: 6 January 2018 Revised: 18 October 2018 Accepted: 16 December 2018

DOI: 10.1002/stc.2321

REVIEW

A literature review of next‐generation smart sensing

technology in structural health monitoring

Sandeep Sony | Shea Laventure | Ayan Sadhu

Department of Civil and Environmental

Engineering, Western University, London,
Ontario, Canada
Correspondence
Ayan Sadhu, Assistant Professor,
Department of Civil and Environmental
Engineering, Western University, London,
Ontario, Canada, N6A 3K7.
Email: asadhu@uwo.ca
Funding information
Natural Sciences and Engineering
Research Council (NSERC) of Canada
Summary
Advent of computationally efficient smartphones, inexpensive high‐resolution
cameras, drones, and robotic sensors has brought a new era of next‐
generation intelligent monitoring systems for civil infrastructure. Vibration‐
based condition assessment has garnered as a prominent method of evaluating
the health of large‐scale infrastructure. The use of contact‐based sensors for
acquiring vibration data becomes uneconomical and tedious due to their
instrumentation cost, centralized nature, and densification required to collect
sufficient data for system identification of modern complex structures. A need
to advance and develop alternative methods for efficient sensing system results
in next‐generation measurement technology of structural health monitoring.
The abundance of handheld smartphones with easily programmable frame-
work has helped in modifying relevant software to acquire vibration data using
embedded sensors in the smartphone. The inexpensive cameras have been
used to capture images and videos that are utilized to understand the structural
behavior with the aid of advanced signal processing techniques. The inaccessi-
ble components of structures require noncontact sensors such as unmanned
aerial vehicles (UAVs) or so‐called drones and mobile sensors to acquire struc-
tural data. To the authors' knowledge, this paper first time presents a compre-
hensive review of a suite of next‐generation smart sensing technology that has
been developed in recent years within the context of structural health monitor-
ing. The state‐of‐the‐art methods have been presented by conducting a detailed
literature review of the recent applications of smartphones, UAVs, cameras,
and robotic sensors used in acquiring and analyzing the vibration data for
structural condition monitoring and maintenance.

KEYWORDS
camera, mobile sensors, SHM, smartphone, structural condition assessment, UAV

Abbreviations: BIM, building information modeling; CCD, charged coupled device; DIC, digital image correlation; DSLR, digital single‐lens reflex;
DSN, dynamic sensor network; FPS, frames per second; GPS, global positioning system; HOG, histograms of oriented gradients; iOS, iPhone
operating system; LADAR, laser detection and ranging; MAV, microaerial vehicle; MCS, motion capture system; MM, motion magnification; NDT,
nondestructive testing; OBIA, object‐based image analysis; OCM, orientation code matching; OFM, optical flow method; RGB‐D, red‐green‐blue‐
depth; ROI, region of interest; SHM, structural health monitoring; SI, system identification; UAV, unmanned aerial vehicle; UIS, unit influence
surface; VBI, visual‐based inspection

Struct Control Health Monit. 2019;e2321. wileyonlinelibrary.com/journal/stc © 2019 John Wiley & Sons, Ltd. 1 of 22
https://doi.org/10.1002/stc.2321
2 of 22 SONY ET AL.

1 | INTRODUCTION

Large‐scale civil infrastructure such as buildings, bridges, dams, wind turbines, and pipeline systems are exposed to var-
ious external loads throughout their lifetime. Vibration caused by earthquakes, wind, temperature, or human‐made
excitation initiates structural damage during their service lives and subsequently, triggers catastrophic failure. Structural
Health Monitoring (SHM) is an emergent and powerful diagnostic tool for damage detection and disaster mitigation of
large‐scale structures. The SHM comprises four key elements: data acquisition, system identification, condition assess-
ment, and decision making/maintenance. The traditional SHM methods use global responses such as vibration and
local responses such as strains or a combination of both to assess the structure during in‐service conditions or extreme
climatic events. Unlike displacement or strain‐based methods, vibration‐based SHM strategies1-3 are very effective in
evaluating global health state of structures and performing a rapid risk assessment and hazard mitigation. Most of these
techniques primarily rely on acceleration measurements that require the installation of either contact or noncontact
sensors collecting rich quality of data.
Traditional contact‐based sensing methods involve the use of sensors that are directly attached to the structure to
measure dynamic responses such as acceleration, velocity, displacement, or inclination. Such contact sensors include
a wide range of devices such as an accelerometer, linear variable differential transformer, strain gauge, fiber optic sen-
sor, piezoelectric sensor, an impedance sensor, telependulum, and ultrasonic wave sensor.4-11 However, they pose many
economic and practical challenges. For example, the contact‐based wired sensors involve time and labor‐intensive
installation process and seek substantial maintenance to achieve long‐term monitoring and maintenance. Moreover,
these sensors provide sparse and discrete point‐wise measurements and low spatial sensing resolutions limiting the
effectiveness of the SHM on a large‐scale structure. With the advent of wireless sensing technology, contact‐based
sensors have seen a paradigm shift into wireless sensors addressing several limitations of wired sensors by sending data
to the central workstation wirelessly.12-14 However, the wireless sensors still have several limitations that motivate
researchers to develop a newer class of smart sensors. For example, wireless data collection is still rigorous, time‐
intensive, and destructive in nature while hundreds of sensors are mounted on a large‐scale structure. Moreover, the
data acquisition remains challenging due to the complexity of data transmission and time synchronization and power
consumption. The last two decades have seen innovation in smart sensing network systems for data acquisition, which
is considered as the most critical component in SHM, and some of the benchmark studies15,16 have presented efficient
implementation of various data acquisition systems and smart sensing within SHM.
With the recent development of sensing and robotic technology, there has been a paradigm shift of next‐generation
sensing techniques that outperform the traditional contact‐based sensors including wired or wireless sensors in various
aspects. This paper reviews a class of such emergent smart sensing technologies that have been developed in the last few
years and highlight the use of next‐generation sensors such as cameras, drones, robotic sensors, and smartphones
specific to SHM. In this paper, nearly 140 papers were reviewed covering the recent applications of next‐generation
sensors and their potential in SHM in both academia and industry in the near future. The key contribution of this paper
is to present a brief literature review on this topic, a critical explanation and detailed architecture of individual sensors
are not attempted.
This paper is presented as follows. A brief overview of contact‐based sensing methods is presented in this section in
light of their limitations and drawbacks that are addressed through next‐generation sensing methods. The details of
smart sensing methods and their recent relevant applications in SHM are presented next. The literature review of
camera and unmanned aerial vehicles (UAVs)‐based sensing methods are discussed first followed by the smartphone
and mobile sensing‐based SHM applications.

2 | N E XT ‐GENERATION SENSING M ETHODS

To overcome the challenges of contact sensors, there has been a significant development of noncontact sensors since last
three decades. Sensors based on Global Positioning System (GPS)17-21 provided an easy‐to‐install remote nonintrusive
approach to the SHM. The GPS is reasonably accurate and gives dynamic measurements of 20 Hz or more in case of
high‐rate GPS.18,19 However, the GPS can be very sensitive to electromagnetic noise, environmental interference, and
weather conditions. Unlike the GPS, noncontact laser vibrometers and radar interferometry22-30 provide high‐quality
measurements. However, these instruments are expensive and have restrictions on weather conditions and maximum
SONY ET AL. 3 of 22

required measurement distance because longer distance measurements require a light with higher intensity. They could
also be dangerous to human health that is present in the structures under inspection during in‐service data collection.
Above challenges of conventional noncontact sensors are eliminated with the recent development of other types of
next‐generation sensors that are integrated with visual and mobile monitoring systems. Such alternate smart sensing
techniques include digital and high‐speed cameras, UAVs, smartphones, and mobile (robotic) sensors. These sensors
address several limitations of the conventional sensors and outperform over the latter. They are easier and convenient,
and allow faster installation and offer data acquisition that is more reliable with high‐resolution temporal and spatial
information of the structures. Moreover, they are less labor intensive and highly cost‐effective. In this paper, a thorough
literature review of the state‐of‐the‐art modern sensors including camera, UAV, smartphones, and mobiles sensors
are presented, respectively. A schematic of various next‐generation sensors used for the SHM is shown in Figure 1.

2.1 | Camera‐based literature

There is a significant growth in low‐cost vision‐sensing technology, and with the aid of image and video analysis,
high‐quality condition assessment of structures can be performed remotely. Consumer‐grade DSLR cameras and highly
sophisticated high‐speed cameras have been used for data acquisition. Cameras are primarily characterized using
frames per second (fps), pixels, bandwidth, and image stabilization. The existing camera‐based techniques are diverse
in nature and range from digital image correlation (DIC) to motion magnification (MM). The vision‐based SHM
methods primarily consist of four steps: camera calibration, image acquisition and rectification, displacement field
measurement, and damage detection. Even though the DIC has become very popular, low level of movement related
to high‐frequency excitation forms a challenge. A new tool, called motion magnification, is developed to understand
operational deflection shapes of a structure. The combination of the DIC and MM can help up to a particular frequency
to counteract the challenge of displacement measurement at low displacements under high‐frequency excitation.31
Helfrick et al.32 investigated a 3D DIC technique for detecting damage through changes in curvature using a pair of
stereo cameras. This approach was able to predict the location of varying levels of damage in the form of a crack of a
certain depth. Huňady et al.33 used Q‐450 Dantec Dynamics camera to estimate damping of a steel plate where the data

FIGURE 1 A schematic of various next‐

generation sensing technology used in the
SHM
4 of 22 SONY ET AL.

were captured at a rate of 1,000 fps. Trebuňa and Hagara34 proposed a modification of a high‐speed correlation system
to estimate the modal parameters of steel plates. In their study, Dantec Dynamics cameras were used with a sampling
frequency of 2000 fps, and the resulting files were saved as hierarchical data format through Intra4D. An automated tool
called Modan3D was developed to perform image processing directly from hierarchical data format files to undertake
modal identification. Feng et al.35 developed a new vision‐based sensor to measure dynamic displacements from the
video images without using a target‐marker panel. The proposed sensor consisted of one or multiple low‐cost charge‐
coupled video cameras (manufactured by Point Grey Research) with telescopic lenses for real‐time extraction of
displacements of video images collected at a rate of 150 fps. Orientation code matching (OCM) algorithm36 was used
for image processing that allowed tracking of existing bridge surface remotely. Feng et al.37 improved the vision‐based
sensors and tracked the structural vibrations by taking a sequence of video frames and analyzed them with the template
matching technique. The results showed that the vision sensors generated a high degree of accuracy for both the
artificial and natural targets.
A study by Yang and Yu38 presented a video‐based vibration monitoring system to monitor the velocity and displace-
ment field. The image processing was carried out using three different methods: frame difference method, particle
image velocimetry, and optical flow method. The experimental result of the study showed that optical flow method
reached higher accuracy compared to particle image velocimetry as it utilized a sliding window‐based correlation
algorithm that was highly dependent upon the quality of the input images. In a similar study, Walker39 studied the
condition assessment of timber structures using virtual visual sensors. This research was focused on the determination
of natural frequencies by monitoring intensity value of a single pixel coordinate over the course of a few seconds of a
video of vibration and then applying fast Fourier transform to extract the frequencies. The resulting natural frequencies
of the bridge were found to be comparable to the accelerometers. The additional benefits of the virtual visual sensors
include the ability to use multiple data points and not be constrained to a single plane of motion.
A work by Bell et al.40 incorporated a DIC technique into the instrumentation and testing program of a bridge. They
developed a protocol to use performance data recorded from strain sensors and digital images that bridge managers can
utilize for risk assessment of the critical bridges. It was found that camera's height did not have a significant impact on
the strain measurements. Apart from the bridge deck, the detection of pavement defect also formed an integral part of
the infrastructure integrity assessment. Radopoulou and Brilakis41 studied the application of semantic texton forest,
a supervised learning algorithm, to detect pavement defects through parking camera. The data were collected using
two cameras: an HP Elite Webcam (Hewlett Packard, Palo Alto, California) chosen to simulate a low‐resolution parking
camera, and a Point Grey Blackfly 05S2M‐CS camera that met the existing standards of the parking cameras. It was con-
cluded that the algorithm needed abundant samples to train the baseline data. Moreover, other defects such as rutting,
depressions, and elevations in pavements also needed to be incorporated to develop a fully automated pavement condi-
tion monitoring method. Subsequently, Radopoulou and Brilakis42 improved previous method considering the fact that
the parking camera captured unintended areas, which slowed down the speed of the monitoring method. Radopoulou
and Brilakis42 presented a process that identified the correct region of interest (ROI) using inverse perspective mapping
to map the road frame coordinates to world coordinates.
A need to understand the real‐time damage assessment led to work by Yang and Nagarajaiah,43 which investigated
real‐time close‐up imaging of structures to automate the detection of local structural damage by exploiting the spatio-
temporal data structure of the camera‐based images. Numerous frames were decomposed into a superposition of low‐
rank background components (i.e., irrelevant features) and dynamic sparse elements (i.e., damage‐related information)
using principal component pursuit without requiring any parametric model or prior structural knowledge. A Nikon
COOLPIX L18 camera was used to obtain the baseline data for the algorithm at a rate of 30 fps. The authors concluded
this method had significant potential for automated online damage surveillance and diagnosis of critical structural
components. In another study, Park et al.44 presented a 3D displacement measurement model using a motion capture
system (MCS) consisting of three VICON T‐160 cameras. It was showed that the MCS had a better degree of accuracy
compared to the laser displacement sensors in terms of higher efficiency and sampling rate. It was concluded that 3D
displacement measurement was necessary for displacement sensing of structures where torsional and lateral displace-
ments occurred together. However, this technique had several limitations such as maximum range requirements
(less than 50 m), the necessity of at least three cameras and reflective markers. Using the similar concept, Oh et al.45
utilized an MCS with multiple markers to measure dynamic responses and extract modal parameters of the structure
using frequency‐domain decomposition method.
Recently, several studies have been conducted with the combination of vision‐based sensors and various image
recognition algorithms. For example, Dworakowski et al.46 applied a vision‐based method for in‐plane displacement
SONY ET AL. 5 of 22

measurement of cantilever beams using Canon EOS 5D MKII camera. The DIC was used to produce deflection curve of
structure using two different algorithms: line segment method and voting method. Ye et al.47 combined vision‐based
system with digital image processing and proposed a hybrid approach using a multipoint pattern matching algorithm
to search the measurement targets in the subsequent images captured by Prosilica GE1050 camera. Although the vision
system was able to capture excellent results when compared to the magnetostrictive displacement sensor, it was found
that illumination and vapor had a critical effect on the overall performance. Santos et al.48 performed continuous
structural monitoring using the vision‐based system with a sequence of images as well as full‐motion tracking of the
structure. In another studies, Yang et al.49 and Yang and Yu50 underscored the importance of high‐density modeshapes
that were critical for damage localization as they contained local structural features. They developed an algorithm
that is capable of utilizing undersampled data from the digital camera (Sony NXCAM) at a frame rate of 240 fps.
The readily available consumer‐grade cameras have shown a promising future for its use in SHM. Fukuda et al.51
developed a vision‐based low‐cost system to monitor large‐size infrastructure using a digital camcorder and computer
with preinstalled image‐processing software. The proposed technique used displacement parameters based on a target
and movement point for dynamic analysis of the structures and was further improved by including time‐
synchronization measurement. Wu et al.52 presented a framework to conduct dynamic testing using a basic target track-
ing vision‐based sensing system. A study by Yoon et al.53 introduced a target‐free vision‐based approach using
consumer‐grade cameras. The proposed method utilized the advantage of three algorithms to consider the absence of
targets, that is, Kanade‐Lucas‐Tomasi algorithm for feature detection and determining pixel coordinates, MLESAC
algorithm for estimating image geometry, and eigensystem realization algorithm for system identification. In practice,
the continuous monitoring for timely maintenance of the structures requires autonomous monitoring techniques,
namely, a deep learning methodology called restricted Boltzmann machine was employed to train deep learning
network in Xu et al.54 for crack detection in steel box girder bridges by capturing images using the consumer‐grade
camera. The effect of the resolution of images and element size was studied to understand the potency of the proposed
method.
In some cases, for buildings and bridges, multipoint displacement monitoring is critical, and the development of a
low‐cost single camera vision‐based technique is essential for its implementation in the industry. In Feng et al.,55
the multipoint displacement for a three‐story frame structure was evaluated using two advanced template matching
techniques: the unsampled cross correlation and the OCM. The results from a single camera, laser displacement, and
accelerometers were compared. With a good data acquisition capability of a single‐camera, the need to develop an
integrated image and video analysis application was explored by Luo et al.56 A new technique, InnoVision, a video image
processing technique, was developed to address challenges associated with vision‐sensors for SHM. The three critical
difficulties, limited lighting, multipoint displacement, and camera‐vibration in the field associated with vision‐sensors,
were addressed. Along the similar lines, Xu et al.57 instrumented cable‐stayed bridge to evaluate the modal frequencies
using low‐cost, consumer‐grade camera. A comparable study58 was conducted using 2D DIC and fiber Bragg grating to
explore its efficacy in measuring dynamic displacement of rail bridges under various constraints such as lightning effect,
camera setup and its movement, and matric calibration. The challenges encountered such as out‐of‐plane movements
were observed and corrected for better results in exploring the 3D behavior of the structures. In Xu and Brownjohn,59
a collection of various articles considering vision‐based displacement measurement published up to date was reviewed
with regard to camera calibration, target tracking, and displacement calculation procedures. The present challenges
faced in vision‐based methods, such as robust tracking methods, noncontact sensing, and evaluation of measurement
accuracy in field, were discussed.
Apart from the digital cameras, the abundance of recent low‐cost and high‐definition video camera found its use in
various SHM applications where the video camera was used as a remote monitoring device. The MM technique by Chen
et al.60 utilized high‐speed cameras (with a frame rate of 5000 fps) to visualize and quantify the modeshapes of
structures. In a subsequent study, Chen et al.61 presented a video camera‐based vibration measurement study as a
proof‐of‐study in which an antenna tower on top of a building was measured from over 175 m. In a recent research,
Chen et al.62 focused on off‐the‐shelf RGB‐D cameras to monitor deformation fields of the structure. The test results
showed that the RGB‐D sensor is able to record vibration data at a maximum sampling rate of 30 Hz with proper
integration of hardware and software. In a different study, the effect of body motion of various subjects in footbridges
was studied using cameras.63 In another recent study, Abdelbarr et al.64 investigated the use of RGB‐D sensors for 3D
displacement field measurement of flexible structures under dynamic loads. The research studied the advanced use of
Kinect sensor for 3D translation motion along with rotational and torsional component. The experiments were con-
ducted in the laboratory as the field usage of RGB‐D sensors was restricted owing to infrared technology. It was showed
6 of 22 SONY ET AL.

TABLE 1 Summary of camera‐based SHM applications

Literature Camera type Mathematical technique Key issues investigated

Helfrick, 2009 Stereo camera pair Digital image correlation Curvature method for
damage detection
Fakuda et al., 2010 Panasonic PV‐GS35 TCP/IP protocol Multipoint displacement
measurement
Hunady, 2012 Point Grey High‐speed correlation Damping ratio
system Q450
Trebuna, 2014 Q‐450 Dantec dynamics Normal mode and complex Experimental modal
mode indicator function analysis
Feng, 2014 Point Grey Orientation code matching algorithm Bridge dynamic response
Chen et al., 2015 Phantom V10 Time‐frequency method Motion magnification
high‐speed camera
Dworakowaski Canon EOS 5D mark II Damage correlation method, Deflection curves
et al., 2015 segment and voting method
Feng et al., 2015 Point Grey Up‐sampled cross‐correlation Displacement measurement
Oh et al., 2015 Vision T‐160 with Frequency domain decomposition Motion capture system and
Nexus post‐processing system identification of
buildings
Park et al., 2015 Vision T‐160 model Coordinate transformation 3D structural displacements
Yang and Nikon COOLPIX L18 Principal component pursuit Real‐time detection of local
Nagarajaiah, 2015 damage
Feng and Feng, 2015 Point Grey The unsampled cross Multipoint displacement
correlation (UCC) and measurement using one
the orientation code camera
matching (OCM)
Rodapoulou and PGBFLY 0552M Parking Inverse perspective mapping Region of interest on
Brilakis, 2016 Camera plus pavements
Suney DSL 212 lens
Bell et al., 2015 Point Grey Digital image correlation Bridge health monitoring
Yang and Yu, 2016 Video camera Various image processing methods Building vibrations
Chen et al., 2016 Point Grey Time‐frequency methods Antenna tower vibration
Santos et al., 2016 IPX‐2M30H‐L, Imperx Extended Kalman filter Displacement of large structures
Yang et al., 2016 SONY NXCAM with Spatial–temporal uncoupling Full‐field vibration
Zeiss lens measurement
Ye et al., 2016 Prosilica GE1050 with Multipoint matching algorithm Vision‐based structural
Navidar 12X zoom lens displacement
Zaurin et al., 2016 Digital video camera Lagrange interpolation algorithm Bridge unit influence lines
Zheng et al., 2016 Grey Flea USB3.0 CMOS Object tracking algorithm Human‐induced vibrations
Nikfar and GoPro Hero4 Black Signal synchronization with wavelet Vibration response of
Konstantinidis, 2016 Action Camera transform nonstructural component
Yoon et al., 2016 GoPro Hero and LG Pro Kanade‐Lucas‐Tomasi, MLESAC, Target free vision‐based
smartphone camera and eigenvalue realization algorithm measurement
Chen et al., 2017 RGB‐ Kinect V1 camera Frequency method and 3D‐imaging and depth sensing
cross‐correlation
Yang et al., 2017 SONY NXCAM Fractal dimension algorithm Non‐visible damage detection
Yu et al., 2017 Y3C with Xenoplan ZNSSD plus Gauss–Newton algorithm Full‐field vibration measurement
lens

(Continues)
SONY ET AL. 7 of 22

TABLE 1 (Continued)

Literature Camera type Mathematical technique Key issues investigated

Yeum et al., 2017
Feng and Feng, 2017
Kim et al., 2017
Kromanis and
Al‐Habaibeh, 2017
Poozesh et al., 2017
Zhou et al., 2017
Xu et al., 2017
Javh et al., 2018a
Javh et al., 2018b
Khuc and Catbas, 2018
Luo et al., 2018
Xu et al., 2018
Acikgoz et al., 2018
Nikon D90 ROI image localization for visual Monitoring of highway sign
interrogation structures
Point Grey Orientation code matching Structural stiffness and
excitation forces
IMX‐5040FT Pattern matching and subpixel Cable tension
estimation
Lenevo A806 Image processing Deformation of structures
PHOTRON high Blind source separation with poly Dynamic response of wind
speed camera reference least‐square complex turbine blades
frequency domain
Monochrome CCD with Wavelet decomposition Temperature effect on
GigE Vision Interface vision measurement
Nikon‐D7000 Restricted Bolmann Crack detection
machines‐deep learning
FastCam SA‐Z Gradient‐based optical High‐frequency modal
flow methods identification
Nikon D5300, Spectral optical flow imaging Full‐field displacement
Photron FastCam spectrum
Logitech webcam HOG and UIS Damage detection
Point Grey Gradient based template InnoVision video processing
matching algorithm application
and interpolation
subpixel method
GoPro Hero C++ programming and Multipoint displacement of
OpenCV cable‐stayed bridge
Allied Vision GigE video 2D DIC Dynamic displacements in
camera bridge

Note. CCD: charged coupled device; HOG: histograms of oriented gradients; ROI: region of interest; UIS: unit influence surface.

that if the structure's axis is parallel to Kinect's depth axis, the error increases to 5% when the displacement is larger
than 10 mm, and the rotation angle is greater than 5°.
The use of noncontact methods also offers a promising way for modal identification. Feng and Feng65 aimed to link
the displacement measurement with vision method by simultaneous identification of structural stiffness and excitation
forces. The robust subpixel OCM algorithm was used to track the structural displacement of the bridge. Kromanis and
Al‐Habaibeh66 presented the use of smartphones to monitor movement in structures. It was concluded that with the
addition of an optical zoom lens, the deformation monitoring of a full‐scale bridge could be carried out efficiently.
In a similar study, Poozesh et al.67 proposed modal identification method using optically measured data. They used
complexity pursuit algorithm to recover the source signals, and then, poly‐reference least square frequency domain
technique was employed to estimate the modal information of a wind turbine blade. Lately, Molina‐Viedma et al.31
investigated the use of a combination of the MM and DIC for modeshape characterization at higher frequencies.
The experimentation was performed on a stepped aluminum bar where the natural frequency and modeshapes were
identified. The DIC method was used for modeshape characterization at lower modes, and the MM was used for higher
modes.
The problem associated with the use of cameras during various weather conditions were evaluated by Kim et al.68
using various image processing techniques (i.e., pattern matching and subpixel estimation). The results showed only
1% variation in camera‐based measurement with respect to accelerometers. Zhou et al.69 aimed at understanding the
use of videogrammetry under temperature variations on vision measurement systems. The study found a temperature
variation in axial displacement in different directions. The vertical displacement measurement error and temperature
8 of 22 SONY ET AL.

change were found to be in a linear relationship, whereas horizontal displacement measurement and temperature
change were more scattered. The continuous wavelet transform was used to identify the frequencies of the signals.
The temperature fluctuations of 1.9°C were shown to cause a displacement error of 16.5 mm in long‐term monitoring
under outdoor measurement.
With the advancement in camera‐based SHM methods, there has been a parallel development of reducing the cost of
assessing the full structural integrity by employing a fewer number of cameras. Yu and Pan70 applied a single camera
and a high‐speed stereo‐DIC method using a four‐mirror adapter that can eliminate two synchronized high‐speed
cameras and reduce overall cost of full‐field vibration measurement. Surface images of targets were separately imaged
onto two halves of the camera sensor through two different optical paths by using a four‐mirror adapter to estimate
the modal parameters of the plate. In another similar study, Yeum et al.71 performed automatic image collection using
the UAVs and developed a novel image localization technique that can automatically extract the ROIs on each of the
collected images. The ROIs on the images were computed based on the 3D geometry of the images concerning targeted
ROIs in the structure. Yang et al.72 focused on using advanced computer vision and video processing algorithms to
develop a novel output‐only modal analysis algorithm that did not require any surface preparation. This study used both
the DIC and the optical flow technique and was able to estimate very high‐resolution operational modeshapes while
accelerometers provided modeshapes only at limited discrete points. Continuing along the similar lines, Oh et al.73
developed a new modal identification method using the images of a Vicon 2016 video camera.
One of the primary disadvantages of using high‐speed camera is that the data may accumulate measurement noise
that interferes in the modal analysis. Javh et al.74 conducted modal identification on noisy high‐speed camera data using
least‐square complex‐frequency domain method. The combination of an accelerometer and camera produced excellent
results in obtaining modal parameters without the burden of noisy data from the high‐speed camera. Javh et al.75
assessed the use of photogrammetry measurement using DSLR camera that utilized spectral optical flow imaging to
measure the individual displacement components. Khuc and Catbas76 presented a novel framework integrating vehicle
load as input for damage detection and localization for a beam type or plate‐like structures. In this study, the machine
learning methods, namely, vehicle detection algorithm and histograms of oriented gradients, were used for vehicle
localization, and an image‐processing‐based iterative algorithm was used to process the measurement data. Recently,
Feng and Feng77 presented a thorough review of vision‐based SHM methods covering system identification to damage
detection while also implying its field applications and various errors associated with camera‐based techniques.
As presented in the above section, the use of camera and the relevant image processing algorithms form a wide
variety of noncontact methods for different SHM applications. A summary of various cameras and numerical methods
used by the several researchers are finally presented in Table 1. Despite the recent use of camera‐based sensors in
myriad applications of the SHM, there are several challenges and limitations that are currently affecting the perfor-
mances of vision‐based methods. Factors including weather conditions such as wind, rain, light, snow, fog, and the
surrounding vibrations as well as the accuracy of camera‐based measurements under small‐amplitude motion need to
be explored in the context of SHM.

2.2 | UAV‐based literature

Traditional contact‐based sensors create a hindrance when the surface is inaccessible, or a large number of sensors are
required to be installed limiting the affordability of contact‐based sensors. UAV, namely, drone, has recently garnered as
an alternative portable device in the field of noncontact measurement technology. Rapid advances in control theories,
computing facilities, communications, robotics, and automation technologies offer an excellent tool for the UAV
technology towards its myriad applications in the SHM. With the aid of lightweight cameras, the UAVs are now used
to capture pictures and subsequently estimate the local and global health of the structure. Typically, a UAV is built
around a vision system that includes an off‐the‐shelf camera (e.g., infrared camera, optical sensor, or LADAR),
navigation system, and GPS with the aid of visual servoing. It contains in‐flight data acquisition and postflight image
processing and can fly remotely that can be controlled by a navigator on the ground. Recently, the UAVs are used
for many civil engineering applications, especially in the field of transportation, including traffic monitoring and survey-
ing, construction inspection, and health monitoring of roads, pipelines, bridges, and buildings. The UAV sensors reduce
workplace accidents and decrease logistics and working hours. They are suitable for monitoring inaccessible areas and
provide great temporal and spatial resolutions in comparison to satellite images. One of the attractive features of the
UAV sensors is that they provide 3D information of the structure that could be beneficial for large‐scale structures.
SONY ET AL. 9 of 22

Initially, the UAVs were introduced78 for bridge inspection and traffic surveillance. A control law was developed for
accurate tracking of a planer target using the UAVs. A feasibility test was performed using a helicopter‐based UAV that
captured a video of a bridge while stabilizing it above the target using the 2½D visual servoing control law. Rathinam
et al.79 proposed an image sensor directly onto the UAV while the aircraft was flown with a closed‐loop tracking system
augmenting GPS‐based control to follow a curved path. However, the algorithms did not consider disturbances caused
by wind that could affect smaller UAVs. Zhang and Elaksher80 developed a UAV‐based digital imaging system to
acquire road surface data where with the resulting images were used to produce a 3D model to assess internal damages.
A helicopter‐type UAV performed an experimental flight, and the aircraft acquired images of several rural roads that
showed signs of deterioration such as potholes and ruts. Image processing techniques including image orientation were
applied to compute 3D coordinates of conjugate points using two stereo photos. In Roca et al.,81 UAVs equipped with
lightweight Kinect cameras were used to inspect building facades and roofs. Photos that were taken were greatly
overlapped to produce a 3D point cloud representation of the façade. Dobson et al.82 tested the usefulness of the
UAV in monitoring of unpaved road surfaces utilizing a single‐rotor helicopter equipped with a camera. The UAV
was capable of automated flight between predetermined waypoints and maintaining a steady flight in the wind speeds
up to 5 mph. The UAV took the images on a 200‐m stretch of road which was accomplished in 5 min at a speed of 2 m/s.
The images were then processed through the Canny edge detection and Hough circle transform algorithms to detect
circular or elliptic potholes as well as their radius.
In a study by Eschmann et al.,83 the UAVs, specifically microaerial vehicles (MAVs), were used to conduct nonde-
structive testing (NDT) of buildings. The MAV chosen for the building inspection was an octocopter having a diameter
of 1 m, and an attached camera having a resolution of 12 megapixels and 14× optical zoom. The MAV was operated
manually instead of using GPS navigation and took images of the building with an automatic picture‐taking sequence.
Automated crack detection was investigated using edge detection and Gaussian Blur techniques, although this proved to
be insufficient in detecting smaller sized cracks. They also looked at the horizontal and vertical movement of UAVs
towards the accuracy of data. To test the feasibility of UAV‐based SHM, Some authors84-86 performed an experimental
flight. The test flight was conducted on an industrial inspection of a 225‐m concrete chimney where the aircraft
surveyed the entire structure in a circular pattern at a constant distance using Point of Interest Technology. The second
test flight was undertaken for the inspection of a 56‐m‐tall historical building. The UAV was able to fly very close to the
building, with small cracks readily detectable in the resulting images.
A study by Ortiz et al.87 used UAVs for surveillance of heritage sites. The UAV recorded video, performed a thermo-
graphic analysis, and measured temperature. During the flight, the UAV successfully identified several different
weathering patterns including loss of material, cracks, discoloration, deposits, alveolarization, and erosion. Ellenberg
et al. 88 used the UAV with 3D simultaneous localization and mapping technology. They developed a proof‐of‐concept
field demonstration for crack detection in a pedestrian bridge using the UAV. Because the damages in building facades
were primarily overlooked in previous literature, a technique called oblique color imagery was suggested by Galarreta
et al.89 The UAVs were used as a remote sensor to capture high‐resolution oblique images. The photos were used to
perform a damage assessment by producing a 3D point cloud, as well as the object‐based image analysis technique
to analyze the roof and facade. However, the main problem encountered was the aggregation of the damage data
collected from various parts of the structure, which increased complexity by introducing topological connections of
the damages in the building. A potential solution was to perform an analysis of the 3D point cloud and object‐based
image analysis simultaneously, but more research needed to be conducted in this area. In Hallerman et al,90 the
UAV was investigated in the facade of an office building, which required navigation in small regions and visual
recognition of cracks using a short exposure time to reduce motion blur. The cracks with a width of 0.5 mm were
evidently visible.
In a study by Sankarasrinivasan et al.91 crack detection was accomplished using a grey scale filter known as the
hat transform approach. It was attempted to combine a bottom hat transform with a color‐based filter, such as
Hue‐Saturation‐Value with a proper threshold, to provide optimal crack detection. It was also commented that
advanced flight controllers and gyro‐stabilized cameras must be used to handle wind and image noise. In another
study, Zhou et al.92 performed road detection using the GraphCut algorithm because of its superior performance when
dealing with 2D color imagery. The experiments were performed on paved roads with 2,760 images being captured in
varying resolutions and methods of flight such as slow and fast UAV speeds, as well as high and low altitudes. It was
determined that the UAV system has a precision of 98.4%, which indicated that these aircraft have a high potential to
perform data collection by road detection and tracking methods. However, it was discovered that at low speeds, drift
errors and irregular contours introduced significant errors. Cho et al.93 performed a UAV‐based crack detection using a
10 of 22 SONY ET AL.

feature‐based image recognition algorithm, known as CornerHarris, which utilized Haar‐like features and then
converted the image from color to greyscale. The histogram equalization was used on the black and white images to
enhance the recognition rate, followed by the adaptive binary method, which automatically found a threshold based
on the contents of the image. The proposed system safely inspected high‐rise buildings, and can be applied in other
industries as well, such as the inspection of steep cliffs and docked vessels. Also, a study by94 developed simultaneous
localization and mapping techniques along with combining 4D point clouds within building information modeling to
detect locations on the job site where changes are likely to occur. Postdata collection techniques were also researched
including image processing, computer vision methods, and geometrical processing to determine the most critical
photos.
In Na and Baek,95 the UAVs were equipped with a vibration based nondestructive method to identify damages at an
earlier stage, thus reducing both maintenance costs and the number of sensors attached to the structure. The proposed
NDT method used a single piezoelectric material, acting as both exciter and sensor, which is connected to the UAV via
electrical wiring, and attached magnetically onto ferromagnetic structures, or onto a preinstalled magnet on wood or
concrete structures. The proposed technique was effective in identifying different types of damage, with one plate
having progressive damage, whereas the other involved thickness loss. In Franke et al.,96 the field reconnaissance
performance of small UAVs was tested on bridges at two different soil liquefaction sites in Chile. By comparing the data
obtained from UAV‐based measurements and actual measurements of the lateral spread displacement, the 3D model of
the pier had an average dimensional error of ±2.5%, whereas the 3D model of the bridge had an average dimensional
error of ±1.0%. In another study, Qidwai and Akbar97 combined UAV visual inspection with a robotic magnetic flux
leakage system to identify cracks in metallic structures. The imagery was performed with the aid of vision‐based
inspection and image processing techniques including edge detection and Hough transform.
In a study by Reagan et al.,98 a 3D‐DIC technology was proposed where a UAV performed crack detection in a
concrete bridge. It was concluded that the 3D‐DIC UAV system is a feasible SHM solution, with an accuracy compara-
ble to using dial calipers, as well as a higher accuracy than visual inspection. In a subsequent study Reagan et al.,99
explored the use of the UAVs and 3D‐DIC techniques for the SHM of bridges. The evaluation of the proposed system
was verified using laboratory and field studies. The laboratory studies were conducted on a fixed aluminum plate
whereas field studies were conducted on two real‐life bridges. It was concluded that adequate lighting is critical for
accurate data measurement using the 3D‐DIC system. The use of UAVs is most useful if they are capable of remotely
monitoring a large structure. Chiu et al.100 evaluated the use of UAVs for monitoring of large‐scale structures. The study
used a remotely operated aerial vehicle for displacement measurement of a massive floating membrane spanning over
170 × 420 m in a wastewater treatment plant. Although the UAV system only provided structural displacement relative
to the position of the camera, Yoon and Spencer101 provided a framework to obtain the absolute displacement of the
structure by combining the relative structural displacement and camera motion.
One of the thrust areas for UAV application in SHM is damage identification at inaccessible areas. Morgenthal and
Hallermann102 performed damage detection using UAVs to improve upon conventional inspection methods. Test flights
were carried out on a masonry building, a hanger, a wind turbine, and a chimney, all exhibiting various forms of dam-
age. Ellenberg et al.103 carried out a study on bridge and obtained quantitative details of civil infrastructure systems by
combining imagery acquired from the UAVs. Similarly, Kim et al.104 introduced a crack identification technique using
UAV‐captured photos combined with image processing. Field experiments were performed on a concrete wall with
various forms of cracks occurring due to loads, creep, and shrinkage. The captured images were then sent through a
hybrid image binarization approach to estimate the width of the cracks. The proposed image processing method was
successful in determining cracks with a width larger than 0.1 mm with an error of 7.3%. Advancing in UAV‐based
SHM research, Omar and Nehdi105 used UAVs equipped with infrared thermography capabilities to observe damages
of concrete bridge decks. The captured thermal images were processed using an algorithm that stitched the images
together to form a mosaic of the bridge deck, and k‐means clustering technique was used to categorize the defects into
severity groups. Recently, Duque et al.106 monitored a timber bridge using a UAV combined with a damage quantifica-
tion protocol. Germanese et al.107 proposed a SHM system for historic structures using UAV‐captured images. The test
setup was conducted in a laboratory by using markers to simulate crack openings; these markers were moved
throughout the test to simulate long‐term crack formations. Lei et al.108 introduced a new crack detection technique
to improve upon weaker image‐processing methods of photos taken from UAV systems. The technique, namely, the
crack central point method, was demonstrated on a concrete bridge. The proposed method accurately measured the
width of the crack with an error less than 5%. Finally, Table 2 shows the summary of UAV systems and their relevant
applications in SHM.
SONY ET AL. 11 of 22

TABLE 2 Summary of UAV‐based SHM applications

Literature Model Camera Application

Metni and Hamel, 2007
Rathinam et al., 2008
Zhang and Elaksher, 2012
Park et al., 2012
Dobson et al., 2013
Eschmann et al., 2013
Hallermann and
Morgenthal, 2013
Ortiz et al., 2013
Roca et al., 2013
Hallermann and
Morgenthal, 2014
Morgenthal and
Hallermann, 2014
Ellenberg et al., 2015
Galarreta et al., 2015
Hallermann et al., 2015
Sankarasrinivasan et al.,
2015
Zhou et al., 2015
Cho et al., 2016
Na and Baek, 2016
Ellenberg et al., 2016
Franke et al., 2017
Qidwai and Akbar, 2017
Reagan et al., 2017a
Reagan et al., 2017b
Chiu et al., 2017
Yoon and Spencer, 2017
Kim et al., 2017
Omar and Nehdi, 2017
Duque et al., 2018
Germanese et al., 2018
Lei et al., 2018
N/A N/A Inspecting defects in bridge deck
Sig Rascal N/A Autonomously tracking a road
N/A Canon EOS Digital Rebel Xti Measuring stresses on unpaved roads
N/A N/A UAV wing monitoring
Bergan Tazer 800 Nikon D800 Detecting stresses on unpaved roads
Fraunhofer IZFP Canon PowerShot SX220 HS Detecting defects in buildings
AscTec Falcon 8 Panasonic Lumix DMC TZ Inspecting building and chimney
22/Sony NEX 5 for damages
Pelican VI Canon IXUS i130/Raytheon Weathering analysis of building walls
2500 AS infrared
Octo XL‐Hisystems GmbH MK HiSight SLR1 Inspection of building facades
AscTec Falcon 8 Panasonic Lumix DMC TZ Inspecting bridges and retaining walls
22/Sony NEX 5
AscTec Falcon 8 Panasonic Lumix DMC TZ Visual inspection and damage
22/Sony NEX 5 detection of structures
Parrot AR 2.0 Two built‐in cameras Defect detection on a pedestrian bridge
Aibot X6 V1 Canon 600D Structural damage assessment after
disasters
AscTec Falcon 8 Sony Alpha7R/Sony Monitoring of aging buildings and dams
Camcorder/Panasonic
TZ61/FLIR thermal
N/A PAL Detecting cracks in buildings
CineStar Octocopter HL Sony PJ670 Road detection and tracking
N/A N/A Recognizing cracks in high‐rise structures
SYMA X5HW N/A NDT in buildings
DJI Phantom GoPro Hero 3 Damage quantification in bridges
Phantom II/Skyjib GoPro Hero 3+ HD/Nikon Reconnaissance of liquefaction sites
Super‐6 D7100
Phantom Professional‐3 Built‐in 4K camera Crack detection in laboratory models
InstantEye Gen4 Two Basler acA Monitoring concrete bridges for cracks
1600‐20um cameras
InstantEye Gen4 Two Basler acA Bridge monitoring
1600‐20um cameras
Sensefly eBee RPV Sony WX220 Condition assessment of a floating cover
DJI Phantom‐3 4K resolution camera Absolute structural displacement
Professional
Parrot AR. Drone 2.0 LS‐20150 Concrete crack identification
DJI Inspire 1 Pro Thermal camera‐FLIR Remote sensing of bridge deck
Vue Pro
DJI Phantom 4 GoPro Quantification of bridge deterioration
ISTI‐CNR Canon EOS M Crack patterns in historical structures
Quadrotor N/A Crack detection in bridge

Note. N/A: not available; NDT: nondestructive testing; UAV: unmanned aerial vehicle.
12 of 22 SONY ET AL.

2.3 | Smartphone‐based literature

Modern smartphones are equipped with accelerometers, gyroscopes, and GPS that can be efficiently used for condition
assessment of structures. Since 2010, there has been an increasing trend of using smartphone‐based sensors in SHM
applications due to the abundance of low‐cost smartphones, their mobility and large storage capacities, significant
computational power, and easily modifiable software. Initial investigations were attempted to identify human move-
ment before the Android era when fewer touchscreen smartphones were available on the market. With the popularity
and availability of affordable smartphones, it has now become prevalent to use them for monitoring and retrofitting
of structures. Owing to several attractive features of the smartphones, they have significant potential to be used for
SHM applications for large‐scale structures.
Initially, Lau and David109 made the first attempt in adopting mobile embedded 3D accelerometers for movement
recognition. A Python‐based programming language was used in a smartphone. Several human motions including
sitting, walking, standing, and walking up and down the stairs were performed and identified to understand the accu-
racy of smartphone sensors using different supervised learning methods. The highest sampling rate achieved with the
data from three axes of the accelerometer was around 60–70 Hz, which was enough to capture human motions. Yu
et al.110 utilized sensors embedded in the smartphones along with a data‐processing software installed in the iPhone
terminal. Wireless communication was designed in an external sensor board, and a swing test was carried out to verify
the inclination results using gyroscope from an iPhone and wireless inclinometer. Morgenthal and Höpfner111 used
smartphones for measuring dynamic displacements and natural frequencies. They used Android‐based smartphone that
did not allow the users to specify the sampling rate. It was concluded that the limited range of acceleration measured
by the smartphones hindered their full utilization in SHM applications. It was also concluded that significant inaccura-
cies might arise due to component calibration and fluctuations in temperature. In Höpfner et al.,112 mechanical
oscillations were measured using smartphone embedded sensors. The study showed that the Linux‐based operating
system provides hindrance for sensor data transmission. The kernel driver processed data at a frequency up to
1.5 kHz, but acceleration sensor driver of the Android device initialized the sensor hardware only up to 25 Hz.
The energy requirement required by three‐axis sensors were critical in understanding the use of each sensor for
vibration measurement.
Dashti et al.113 monitored earthquake‐induced ground motion using four iPhone 3GS and three iPod touchpads
where high‐quality accelerometers were used to collect vibration measurements under 3D shake‐table testing.
It was concluded that the major drawback of using smartphones as seismic sensors was their limited operational range
and performance variations among different models. Cimellaro et al.114 developed a versatile mobile application, named
as earthquake damage assessment manager, which was capable of running on Apple's iOS, Google's Android platform,
and BlackBerry. The earthquake damage assessment manager allowed users to collect and report damage information
and perform rapid building damage assessment. The mobile application was grouped into three categories: (a) resident's
personal data, (b) structural damages, and (c) location and features of buildings and associated damages. The applica-
tion was tested under a real earthquake and compared with the conventional damage assessment techniques with
more than 85% accuracy. Shrestha et al.115 introduced the possibility of using multiple smart devices for vibration
measurement and compared their results using high‐quality sensors. The proposed concept seemed analogous to
decentralized wireless sensors; wherein, the smart devices were used for vibration measurement in place of conven-
tional wireless sensors. The proposed method was validated using various laboratory and field studies. The versatile
usage of smartphones in monitoring a full‐scale building was explored in study,116 where an alternative idea to condi-
tion monitoring of a full‐scale building was presented. Millikan library at California Institute of Technology was
monitored using various smartphone sensors. Various problems associated with the deployment of smartphones such
as accurate location in the building, orientation, and its height were discussed. Oraczewski et al.117 adopted an acoustic
technique in an Android‐based system to assess the crack initiation under fatigue loading. Two transducers and one
modulation controller were used for two aluminum plates to identify damage. In another study, Feng et al.118 and Ozer
et al.119 developed a crowdsourcing platform in the form of an iOS application, namely, Citizen Sensors for SHM.
Three different smartphones were used: iPhone 3GS, iPhone 5, and Android‐based Samsung Galaxy 4. A field study
was carried out on a prestressed reinforced concrete pedestrian bridge with an error less than 1% by comparing results
with the conventional wired sensors.
In Zhao et al.,120 an iPhone‐based (iOS 7.0 or higher platform) cable force measurement software, named as Orion
Cloud Cell, was developed that was relatively inexpensive, convenient, more accessible, and time‐saving. The error
between wireless sensors and smartphone sensors was calculated as 0.54% for frequency, and 1.09% for the cable force
SONY ET AL. 13 of 22

measurement. The error in newer generation iPhone 5S (~0.40%) was negligible as compared with old version iPhone 4S
(~4.5%). The field test was carried out on Dalian Hualu Bridge showing more confidence in the use of smartphone
sensors towards SHM applications. Yu et al.121 designed an iPhone with two different patterns, one with the internal
sensor unit and other with external sensor board run by serial port or Wi‐Fi. The internal sensor unit utilized the
embedded sensors in the iPhone, whereas external sensor board was communicated with the serial port or Wi‐Fi.
Numerous experiments were conducted both on a laboratory scale using a shake table and actual field bridges to verify
the feasibility of the proposed sensor board.
Zhao et al.122 evaluated the use of laser‐projection technology for real‐time displacement measurement in bridges.
A laser device was installed on the monitored structure, and the laser spot was projected on a projection plate.
The smartphone recognized the movement of the laser spot on the projection plate, and structural displacement was
calculated using image processing methods. The iPhone 6 was used where two different mobile application of image
processing was used, namely, D‐viewer and Orion‐CC with a sampling rate of 30 and 100 Hz, respectively. Both the
applications processed the color images captured by the camera into grey images and later converted them into binary
images. In this study, Zhao et al.123 used two different smartphones, Samsung A5 and Meizu MX4, for monitoring a
series of static and dynamic displacements. Experiments were conducted to verify the stability of a newly developed
mobile application (i.e., D‐viewer) used for laser‐projection method of the SHM. Samsung A5 showed higher acquisition
frame rate due to its lower number of pixels as compared with Meizu MX4. The experimental result showed an error of
0.85% in displacement measurement, whereas for suspension bridge model, the error was 6.33%. The smartphone used
in this study had short monitoring range and low acquisition frame rate, although this can be improved in the future
with better processing units and camera of the smartphones.
Ozer and Feng124 developed a modal identification strategy that combined spatiotemporally sparse data collected by
the smartphone‐based wireless sensor network. They demonstrated the need for accurate estimation of the sensor
position and nodal information. The signal energy was converted into signal power to normalize the spectral peak
concerning time and thereby to estimate absolute modal displacements. Once the displacements were obtained, they
were integrated with phase information, and subsequently, modeshapes were determined. The errors between the fre-
quencies and modeshapes obtained from smartphone and the wired sensors were less than 2% for the first three modes
of a footbridge. Zhao et al.125 proposed a mobile‐based application for a group of smartphones working simultaneously
to acquire vibration and geospatial data for the SHM. They exploited a low‐cost testing procedure in the iPhone 4S with
higher storage capacity; computation power and good network connection working on a developed software made it
very easy to use smartphones in the SHM applications. The proposed method was successfully implemented for
cable‐force measurement in two cable‐stayed bridges.
Although very convenient, there were no guidelines to perform accurate vibration testing using the smartphones.
Ozer and Feng126 proposed an improved vibration testing method using smartphones with minimal training. They
exploited the coordinate transformation system to choose appropriate orientation of smartphone during data collection.
Two different frameworks were used to process data, namely, core motion and core location. The core motion frame-
work was used for acceleration, gravity, rotation rate, altitude, and magnetic field whereas the core location framework
was used to process latitude and longitude. All these parameters were later used for local to global structural axis trans-
formations. Zhao et al.127 measured displacement response of bridge by photogrammetry technique using a mobile
application, namely “D‐Viewer.” The application was based on characteristics of a 2D image regarding light intensity
and color. A laser projection method was used to evaluate the displacement by measuring variation in the position of
pixels in the picture using iPhone 6 and iPhone 6+. Several static and dynamic tests were performed to understand
the capability of the newly developed system, and its application was tested on a medium span bridge. A recent study128
explored the use of smartphone camera and DIC technique using a custom iOS mobile application, “D‐Viewer,” to
evaluate in‐plane, out‐of‐plane, and arbitrary direction displacement. The results were compared with the laser displace-
ment sensor and a piezoelectric acceleration sensor. Ozer et al.129 proposed and validated a hybrid sensor network for
modal analysis. Three different iPhones (iPhone 3GS, iPhone 5, and iPhone 6) along with piezoelectric accelerometers
and laser vibrometers were used as a heterogeneous system. Feldbusch et al.130 developed a software “iDynamics.” for
the application of SHM.
The low resolution of accelerometers in mobile devices presents a challenge for high‐frequency measurement in
structures. Recently, Zeng et al.131 assessed the feasibility of smartphone‐based application for cost‐effective roughness
monitoring of road surfaces. Two Android‐based smart tablets were used to collect data, and they were firmly kept in
the boxes sitting on the floor of a car during the experimental trips. They concluded that the minimum number of trips
required to achieve high accuracy was dependent on the type of highway and sampling frequency. Despite several
14 of 22 SONY ET AL.

TABLE 3 Summary of smartphone‐based SHM applications

Phone model and operating

Literature system Issues addressed Specific

Lau and David, Nokia N95 8GB, Nokia N800 Movement recognition Triaxial and uniaxial accelerometers
2010 tablet with Symbian S60 FP1 were used
Yu et al., 2012 iPhone 4S with iOS SHM applications Motion and inclination was measured
Morgenthal and HTC desire HD with Transient displacements Comparison of sonar, traditional
Hopfner, 2012 Android V2.3 accelerometers and smartphone
Hopfner et al., 2013 HTC Sensation with Mechanical oscillations Limitations of smartphones for
Android V2.3.3 oscillation measurement
Dashti et al., 2014 iPhone 3GS with iOS Seismic monitoring Reliability of phones for earthquake
monitoring
Camellaro et al., Android and iOS Software development Building damage assessment
2014
Oraczewski et al., HTC one with Android V4.1.2, Nonlinear acoustics for Fatigue crack detection in
2015 “Jelly Bean” SHM aluminum plate
Feng et al., 2015 iPhone 3GS, iPhone 5, with iOS Accelerometer data Producing big data from
and Samsung Galaxy S4 with smartphone accelerometers
Android 5.0.1, “Lollipop.”
Ozer et al., 2015 iOS platform Development of software Development of Citizen Sensor
for crowd computing for SHM
Zhao et al., 2015 iOS platform Development of Orion‐CC Cable force measurement
Yu et al., 2015 iPhone Cable force measurement Validation of smartphones for SHM
Zhao et al., 2016a iPhone 5 Bridge SHM Use of D‐viewer and Orion‐CC
Zhao et al., 2016b Samsung A5 and Meizu MX4 Displacement measurement Development of D‐viewer Android
with Android 4.4.4 and 4.4.2 using laser‐projection‐sensing application
Ozer and Feng, iPhone 5 Modal identification Synthesizing spatiotemporally
2016 sparse sensor data
Zhao et al. 2017a iPhone 4S Quick bridge cable force Mobile application D‐viewer
measurement and Orion‐CC
Ozer and Feng, iPhone 5 Direction sensitive data Coordinate system transformation
2017a correction
Zhao et al., 2017b iPhone 6 and iPhone 6+ Displacement measurement Bridge displacement measurement
“D‐viewer” application
Ozer et al., 2017b iPhone 3GS and iPhone 5 Vision‐based structural Multi‐sensory hybrid sensing
and iPhone 6 with iOS displacement system using smartphone and
traditional accelerometers
Feldbusch et al., N/A idynamics application Application for semi‐professional
2017 vibration measurement
Zeng et al., 2017 Samsung galaxy note 10.1 Development of an Android Road roughness measurement
tablets with Android 4.0.4 application
Kong et al., 2018 Various smartphones Vibration measurement Condition monitoring of the building
including Samsung,
LG, HTC, Sony, Nexus
Wang et al., 2018 iPhone 6 Displacement measurement Smartphone camera and DIC
Shrestha et al., 2018 Multiple smart device systems High‐quality vibration Earthquake‐induced response
measurement with of a building
no data loss

Note. N/A: not available; DIC: digital image correlation.

SONY ET AL. 15 of 22

advantages of the smartphones, several following limitations may hinder their use in various SHM applications of
structures. The use of smartphones requires specialized training in software and computer programming along with
extensive storage instrumentation to process big data efficiently. Finally, Table 3 shows a summary of smartphone‐based
research and its applications relevant to SHM applications.

2.4 | Mobile sensor‐based literature

With recent development in robotic technology, mobile sensing can be used to perform accurate real‐time data acqui-
sition with minimal setup cost. It is considered as an enhancement and sophisticated application of wireless sensing
technology (i.e., next revolution of wireless sensing technology) that can offer higher data rates and accurate time
synchronization. Like smartphones, mobile sensors offer several attractive features that are amenable to SHM
applications. Apart from these sensors being cost‐effective, time‐saving, and compact, it can easily be implemented with
camera and wireless sensors for easy and faster data acquisition.
Initial development of the mobile robotic systems was carried out to develop robots for automation in construction
as well as NDT in bridges. Lorenc et al.132 developed a robotic bridge maintenance system primarily for remote
inspection, spray washing, paint removal, and painting the bridges. Tung et al.133 developed a mobile manipulator
image system for bridge crack inspection to avoid difficulty in reaching the site or locate the cracks that are not visible
to naked eyes. The systems consisted of two charged couple binocular cameras, and a four‐axis manipulator mounted on
a mobile vehicle. A projection algorithm was used to compare the images and crack detection. Experimental and field
studies were carried out to verify the proposed system. After few years, Oh et al.134 developed a seven degrees‐of‐
freedom robotic system equipped with a hydraulic actuator system. The system was designed for safety assessment
and crack detection in bridges. A machine vision system was integrated into the robotic system for tracing, detecting,
and evaluating the dimensions of the cracks in the bridge. The proposed machine vision system was composed of a
charged couple binocular camera, a digital video recorder board, and a vision processing program on the server
computer. A supervised manipulation algorithm for image processing was used to trace and detect the cracks in the
images. A field study was carried underneath the bridge deck, and crack detection was efficiently carried out using
the proposed system.
Zhu et al.135 developed and validated a new mobile sensing system as a proof‐of‐concept study for the SHM.
The proposed mobile sensing device was consisted of two 2‐wheeled cars connected with a beam carrying the acceler-
ometers. Two infrared sensors and two hall effect sensors measuring angular velocity were used in each wheel car.
The wheels were surrounded by thin magnets to provide attraction between the instrument and steel portal frame.
The goal of this study was to detect the structural damage in steel portal frame. Two damage scenarios were used to
understand the utility of mobile sensing device. Lim et al.136 proposed a robotic crack inspection and mapping system
for accurate assessment of cracks in the bridges. The Laplacian of Gaussian algorithm was used to detect cracks, and the
global crack map is obtained through camera calibration and robot localization. Multiple mobile sensing systems137
were used to collect modal parameters of pedestrian bridges wirelessly. It was assured that the devices could move over
the structure for a definite period and batteries can sustain during the time of testing. A single axis accelerometer was
used for this study with a frequency bandwidth of 0–300 Hz. The accuracy of modal identification was assessed under
free‐vibration testing. In this study, the motion of the mobile sensor was limited to unidirectional; the left and right
movements were not permitted. Moreover, the sensor can move only over the smooth surface.
In this study,138 an autonomous robot was used as a mobile sensing device for inspection and evaluation of the
bridge deck. The robotic system was built with electrical resistivity sensor, impact echo sensors, and the panoramic
camera was mounted on computer‐controlled, extendable mast. The measurements from various sensors were fused
using Extended Kalman filters. The motion of the robotic system was so designed to cover the bridge area appropriately,
and an “ox ploughing motion” was used for motion control of the system. Matarazzo and Pakzad139 studied the use of
mobile sensors for system identification with missing observations. A stochastic state‐space‐based modified Kalman
filtering was used to account for the missing observations in the ambient vibration data collected at the Golden
Gate bridge.
Recently, Marulanda et al.140 evaluated the use of a mobile sensor to understand the densification of modal
coordinates instead of a broad array of sensors. The proposed spatial densification method used a mobile sensor
comprising of Imote2 and a stationary sensor namely SHM‐A. The mobile sensor moved over the structure and was
attached with an accelerometer onto specific points where modal coordinates were required. The robustness of this
16 of 22 SONY ET AL.

method was checked by performing sensitivity analysis regarding the errors on the identification of natural frequencies,
the location of the stationary sensor, and signal‐to‐noise ratio. Goorts et al.141 aimed at developing a deployable
autonomous control system for short‐term vibration mitigation. The research used an unmanned ground vehicle for
accurate positioning of sensors, a modified husky A200 robot mounted with electromechanical mass damper for better
control and onboard vision sensors to facilitate self‐sufficient positioning of the device as well as sensing. Landmark
extraction and data acquisition were carried out with the help of Kinect sensors mounted on the deployable
autonomous control system. An experimental study was conducted to verify the functionality of the proposed solution
on a pedestrian bridge.
Apart from their several advantages, the mobile sensing systems come with several challenges that require
future advancement of mobile sensing technology to make them more popular in a wide variety of SHM applications.
For example, these sensors require sophisticated robots and control systems through a continuous multidisciplinary
collaboration of researchers from computer science, electrical engineering, mechanical engineering, and civil
engineering.

3 | CONCLUSIONS

This paper reviews a suite of fundamental literatures that demonstrate the recent development of next‐generation
sensing technologies including cameras, UAVs, smartphones, and mobile sensors towards their application specific to
SHM. With this thorough review, it is clear that there is a significant trend in utilizing smart sensing techniques for
monitoring, retrofitting, and control of large‐scale structures. Unlike traditional contact‐based sensors, the noncontact
sensors with the aid of advanced signal and image processing techniques have become very powerful and cost‐effective
sensing systems for remote and autonomous monitoring of structures.
Although these next‐generation sensors have finally begun to produce robust monitoring of small‐scale structures,
there is still a need for studying the reliability of these sensors on real‐time monitoring of the large infrastructure
under adverse weather conditions. The use of cloud computing to harness the utility of computationally efficient smart
sensors for online monitoring is still in nascent stage. Finally, the key findings of above mentioned next‐generation
sensing techniques are presented in Figure 2 in the context of the SHM.

FIGURE 2 Key findings of various next‐generation sensing technologies

SONY ET AL. 17 of 22

A C K N O WL E D G E M E N T
The authors would like to thank Natural Sciences and Engineering Research Council (NSERC) of Canada to provide
financial support to conduct this research through the last author's NSERC Discovery Grant.

ORCID
Ayan Sadhu https://orcid.org/0000-0001-5685-7087

R EF E RE N C E S
1. Doebling S, Farrar C, Prime M, Shevitz D. Damage Identification and Health Monitoring of Structural and Mechanical Systems from
Changes in Their Vibration Characteristics: A Literature Review. Los Alamos, NM: Technical Report LA‐13070‐MS, Los Alamos National
Laboratory; 1996.
2. Maia NMM, Silva JMM. Modal analysis identification techniques. Phil Trans Biol Sci. 2001;359(1778):29‐40.
3. Carden EP, Fanning P. Vibration‐based condition monitoring: A review. Struct Health Monit. 2016;3(4):355‐377.
4. Doebling SW, Farrar CR, Prime MB. A summary review of vibration‐based damage identification methods. J Appl Mech.
1998;111(2):270‐278.
5. Kuang KSC, Quek ST, Koh CG, Cantwell WJ, Scully PJ. Plastic optical fibre sensors for structural health monitoring: A review of recent
progress. J Sensors. 2009;2009:1‐13.
6. Lynch JP. A summary review of wireless sensors and sensor networks for structural health monitoring. Shock Vib Digest.
2006;38(2):91‐128.
7. Mascarenas DL, Todd MD, Park G, Farrar CR. Development of an impedance‐based wireless sensor node for structural health monitor-
ing. Smart Materials and Structures. 2007;16(6):2137‐2145.
8. Park G, Sohn H, Farrar CR, Inman DJ. Overview of piezoelectric impedance‐based health monitoring and path forward. Shock Vib
Digest. 2003;35(6):451‐463.
9. Raghavan A, Cesnik CES. Review of guided‐wave structural health monitoring. Shock Vib Digest. 2007;39(2):91‐114.
10. Liu C, Park J, Spencer BF, Moon D, Fan J. Sensor fusion for structural tilt estimation using an acceleration‐based tilt sensor and a
gyroscope. Smart Materials and Structures. 2017;26(10):105005.
11. Wu L, Casciati F. Local positioning systems versus structural monitoring: A review. Struct Control Health Monit. 2014;21(9):1209‐1221.
12. Spencer BF, Ruiz‐Sandoval M, Kurata N. Smart sensing technology: opportunities and challenges. Struct Control Health Monit.
2004;11(4):349‐368.
13. Lynch JP. An overview of wireless structural health monitoring of civil structures. Philosophical Transactions of the Royal Society of
London a: Mathematical, Physical and Engineering Sciences. 2007;365(1851):345‐372.
14. Cho S, Yun C, Lynch JP, Zimmerman AT, Spencer BF, Nagayama T. Smart wireless sensor technology for structural health monitoring
of civil structures. Steel Struct. 2008;8:267‐275.
15. Farrar CR, Park G, Allen DW, Todd MD. Sensor network paradigms for structural health monitoring. Struct Control Health Monit.
2005;13:210‐225.
16. Rice JA, Mechitov KA, Sim SH, Spencer BF, Agha GA. Enabling framework for structural health monitoring using smart sensors.
Struct Control Health Monit. 2010;18(5):574‐587.
17. Beskhyroun S, Wegner LD, Sparling BF. New methodology for the application of vibration‐based damage detection techniques.
Struct Control Health Monit. 2011;19(1):88‐106.
18. Häberling S, Rothacher M, Zhang Y, Clinton JF, Geiger A. Assessment of high‐rate GPS using a single‐axis shake table. J Geodes.
2015;89(7):697‐709.
19. Im SB, Hurlebaus S, Kang YJ. Summary review of GPS technology for structural health monitoring. J Struct Eng. 2013;
139(10):1653‐1664.
20. Knecht A, Manetti L. Using GPS in structural health monitoring. Smart Structures and Materials 2001: Sensory Phenomena and
Measurement Instrumentation for Smart Structures and Materials. 2001;4328:122‐129.
21. Yi TH, Li HN, Gu M. Recent research and applications of GPS based technology for bridge health monitoring. Sci China Technol
Sci. 2010;53(10):2597‐2610.
22. Park HS, Lee HM, Adeli H, Lee I. A new approach for health monitoring of structures: terrestrial laser scanning. Comput Aided Civ Inf
Eng. 2007;22(1):19‐30.
18 of 22 SONY ET AL.

23. Kuester F, Chang B, Olsen J, Hutchinson. Terrestrial laser scanning‐based structural damage assessment. J Comp Civil Eng.
2010;24(3):63‐72.
24. Staszewski WJ, Lee BC, Mallet L, Scarpa F. Structural health monitoring using scanning laser vibrometry: I. lamb wave sensing. Smart
Materials and Structures. 2004;13(2):251‐260.
25. Mallet L, Lee BC, Staszewski WJ, Scarpa F. Structural health monitoring using scanning laser vibrometry: II. Lamb waves for damage
detection. Smart Materials and Structures. 2004;13(2):261‐269.
26. Leong WH, Staszewski WJ, Lee BC, Scarpa F. Structural health monitoring using scanning laser vibrometry: III. Lamb waves for fatigue
crack detection. Smart Materials and Structures. 2005;14(6):1387‐1395.
27. Gentile C. Deflection measurement on vibrating stay cables by non‐contact microwave interferometer. NDT and E International.
2010;43(3):231‐240.
28. Gu C, Rice J a, Li C. A wireless smart sensor network based on multi‐function interferometric radar sensors for structural health
monitoring. IEEE Topical Conference on Wireless Sensors and Sensor Networks. 2012;2012:33‐36.
29. Pieraccini M. Monitoring of civil infrastructures by interferometric radar: a review. Sci World J. 2013;2013:1‐8.
30. Pieraccini M, Fratini M, Parrini F, Atzeni C, Bartoli G. Interferometric radar vs. accelerometer for dynamic monitoring of large
structures: an experimental comparison. NDT and E International. 2008;41(4):258‐264.
31. Molina‐Viedma A, Felipe‐Sesé L, López‐Alba E, Díaz F. High‐frequency modeshapes characterisation using digital image correlation and
phase‐based motion magnification. Mech Syst Signal Proces. 2018;102:245‐261.
32. Helfrick M, Niezrecki C, Avitabile P. Curvature Methods of Damage Detection Using Digital Image Correlation. 2009 Health Monitoring of
Structural and Biological Systems 2009:72950D.
33. Huňady R, Hagara M, Schrötter M. Using high‐speed digital image correlation to determine the damping ratio. Proc Eng.
2012;48:242‐249.
34. Trebuňa F, Hagara M. Experimental modal analysis performed by high‐speed digital image correlation system. Measurement.
2014;50:78‐85.
35. Feng MQ, Fukuda Y, Feng D, Mizuta M. Nontarget vision sensor for remote measurement of bridge dynamic response. J Bridge Eng.
2015;20(12):4015023.
36. Fukuda Y, Feng MQ, Narita Y, Kaneko S, Tanaka T. Vision‐based displacement sensor for monitoring dynamic response using robust
object search algorithm. IEEE Sensors J. 2013;13(12):4725‐4732.
37. Feng D, Feng MQ, Ozer E, Fukuda Y. A vision‐based sensor for noncontact structural displacement measurement. Sensors.
2015;15(7):16557‐16575.
38. Yang Y, Yu XB. Image analyses for video‐based remote structure vibration monitoring system. Front Struct Civ Eng. 2016;10(1):12‐21.
39. Walker K. Use of virtual visual sensors in the determination of natural frequencies of timber structures for structural health monitoring.
In: Master of Science Thesis. Oregon State University; 2015.
40. Bell SE, Gaylord S, Goudreau A, White D. “Instrumentation, digital image correlation, and modelling to monitor bridge behaviour
and condition assessment,” FHWA‐NH‐RD‐15680L, U.S. Department of Transportation, Federal Highway Administration. 2015;
A001(267):15680L.
41. Radopoulou SC, Brilakis I. Automated detection of multiple pavement defects. J Comp Civil Eng. 2016a;31(2):4016057.
42. Radopoulou SC, Brilakis I. Parking camera calibration for assisting automated road defect detection. In: 16th International Conference on
Computing in Civil and Building Engineering; 2016b.
43. Yang Y, Nagarajaiah S. Dynamic imaging: real‐time detection of local structural damage with blind separation of low‐rank background
and sparse innovation. J Struct Eng. 2015;142(2):1‐9.
44. Park SW, Park HS, Kim JH, Adeli H. 3D displacement measurement model for health monitoring of structures using a motion capture
system. Measurement. 2015;59:352‐362.
45. Oh BK, Hwang JW, Kim Y, Cho T, Park HS. Vision‐based system identification technique for building structures using a motion capture
system. J Sound Vib. 2015;356:72‐85.
46. Dworakowski Z, Kohut P, Gallina A, Holak K, Uhl T. Vision‐based algorithms for damage detection and localization in structural health
monitoring. Struct Control Health Monit. 2015;23(1):35‐50.
47. Ye X, Yi T, Dong C, Liu T. Vision‐based structural displacement measurement: system performance evaluation and influence factor
analysis. Measurement. 2016;88:372‐384.
48. Santos CA, Costa CO, Batista J. A vision‐based system for measuring the displacements of large structures: simultaneous adaptive
calibration and full motion estimation. Mech Syst Signal Process. 2016;72–73:678‐694.
49. Yang Y, Dorn C, Mancini T, et al. Blind identification of full‐field vibration modes from video measurements with phase‐based video
motion magnification. Mech Syst Signal Process. 2017;85:567‐590.
SONY ET AL. 19 of 22

50. Yang Y, Yu XB. Image analyses for the video‐based remote structure vibration monitoring system. Front Struct Civ Eng. 2016;10(1):12‐21.
51. Fukuda Y, Feng MQ, Shinozuka M. Cost‐effective vision‐based system for monitoring dynamic response of civil engineering structures.
Struct Control Health Monit. 2010;17(8):918‐936.
52. Wu L, Casciati F, Casciati S. Dynamic testing of a laboratory model via vision‐based sensing. Eng Struct. 2014;60:113‐125.
53. Yoon H, Elanwar H, Choi H, Golparvar‐Fard M, Spencer BF. Target‐free approach for vision‐based structural system identification using
consumer‐grade cameras. Struct Control Health Monit. 2016;23(12):1405‐1416.
54. Xu Y, Li S, Zhang D, et al. Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm
based on consumer‐grade camera images. Struct Control Health Monit. 2017;25(2):e2075.
55. Feng D, Feng MQ. Vision‐based multipoint displacement measurement for structural health monitoring. Struct Control Health
Monit. 2015;23(5):876‐890.
56. Luo L, Feng MQ, Wu ZY. Robust vision sensor for multi‐point displacement monitoring of bridges in the field. Eng Struct.
2018;163:255‐266.
57. Xu Y, Brownjohn J, Kong D. A non‐contact vision‐based system for multipoint displacement monitoring in a cable‐stayed footbridge.
Struct Control Health Monit. 2018;25(5):e2155.
58. Acikgoz S, DeJong MJ, Soga K. Sensing dynamic displacements in masonry rail bridges using 2D digital image correlation. Struct Control
Health Monit. 2018;25(8):e2187.
59. Xu Y, Brownjohn JM. Review of machine‐vision based methodologies for displacement measurement in civil structures. J Civil Struct
Health Moni. 2017;8(1):91‐110.
60. Chen JG, Wadhwa N, Cha YJ, Durand F, Freeman WT, Buyukozturk O. Modal identification of simple structures with high‐speed video
using motion magnification. J Sound Vib. 2015;345:58‐71.
61. Chen JG, Davis A, Wadhwa N, Durand F, Freeman WT, Büyüköztürk O. Video camera – based vibration measurement for civil infra-
structure applications. J Infrast Syste. 2016;23(2014):1‐11.
62. Chen YL, Abdelbarr M, Jahanshahi MR, Masri SF. Color and depth data fusion using an RGB‐D sensor for inexpensive and contactless
dynamic displacement‐field measurement. Struct Control Health Monit. 2017;24(11):1‐14.
63. Zheng F, Shao L, Racic V, Brownjohn J. Measuring human‐induced vibrations of civil engineering structures via vision‐based motion
tracking. Measure J Int Measur Confe. 2016;83:44‐56.
64. Abdelbarr M, Chen YL, Jahanshahi MR, Masri SF, Shen W, Qidwal UA. 3D dynamic displacement‐field measurement for structural
health monitoring using inexpensive RGB‐D based sensor. Smart Mate Struct. 2017;26(12). 125016
65. Feng D, Feng MQ. Identification of structural stiffness and excitation forces in time domain using noncontact vision‐based displacement
measurement. J Sound Vib. 2017;406:15‐28.
66. Kromanis, R., and Al‐Habaibeh, A. (2017), “Low‐cost vision‐based systems using smartphones for measuring deformation in structures
for condition monitoring and asset management,” The 8th International Conference on Structural Health Monitoring of Intelligent
Infrastructure, December 2017.
67. Poozesh P, Sarrafi A, Mao Z, Niezrecki C. Modal parameter estimation from optically‐measured data using a hybrid output‐only system
identification method. Measurement. 2017;110:134‐145.
68. Kim SW, Jeon BG, Cheung JH, Kim SD, Park JB. Stay cable tension estimation using a vision‐based monitoring system under various
weather conditions. J Civil Struct Health Monit. 2017;7(3):343‐357.
69. Zhou HF, Zheng JF, Xie ZL, Lu LJ, Ni YQ, Ko JM. Temperature effects on vision measurement system in the long‐term continuous
monitoring of displacement. Renew Energy. 2017;114:968‐983.
70. Yu L, Pan B. Single‐camera high‐speed stereo‐digital image correlation for full‐field vibration measurement. Mech Syst Signal Proces.
2017;94:374‐383.
71. Yeum CM, Choi J, Dyke SJ. Autonomous image localization for visual inspection of civil infrastructure. Smart Materials and Structures.
2017;26(3):35051.
72. Yang Y, Dorn C, Mancini T, et al. Blind identification of full‐field vibration modes of output‐only structures from uniformly‐sampled,
possibly temporally‐aliased (sub‐Nyquist), video measurements. J Sound Vib. 2017;390:232‐256.
73. Oh BK, Kim D, Park HS. Modal response‐based visual system identification and model updating methods for building structures.
Comput Aided Civ Inf Eng. 2017;32(1):34‐56.
74. Javh J, Slavič J, Boltežar M. High‐frequency modal identification on noisy high‐speed camera data. Mech Syst Signal Proces.
2018a;98:344‐351.
75. Javh J, Slavič J, Boltežar M. Measuring full‐field displacement spectral components using photographs taken with a DSLR camera via an
analogue Fourier integral. Mech Syst Signal Proces. 2018b;100:17‐27.
76. Khuc T, Catbas FN. Structural identification using computer vision–based bridge health monitoring. J Struct Eng. 2018;144(2):04017202.
20 of 22 SONY ET AL.

77. Feng D, Feng MQ. Computer vision for SHM of civil infrastructure: from dynamic response measurement to damage detection –
a review. Eng Struct. 2018;156:105‐117.
78. Metni N, Hamel T. A UAV for bridge inspection: visual servoing control law with orientation limits. Autom Construct. 2007;17(1):3‐10.
79. Rathinam S, Kim ZW, Sengupta R. Vision‐based monitoring of locally linear structures using an unmanned aerial vehicle. J Infr Syst.
2008;14(1):52‐63.
80. Zhang C, Elaksher A. An unmanned aerial vehicle‐based imaging system for 3d measurement of unpaved road surface distresses.
Comput Aided Civ Inf Eng. 2012;27(2):118‐129.
81. Roca D, Laguela S, Diaz‐Vilarino L, Armesto J, Arias P. Low‐cost aerial unit for outdoor inspection of building facades. Autom Construct.
2013;36:128‐135.
82. Dobson RJ, Brooks C, Roussi C, Colling T. Developing an Unpaved Road Assessment System for Practical Deployment With High‐
Resolution Optical Data Collection Using a Helicopter UAV. 2013 International Conference on Unmanned Aircraft Systems:235‐243.
83. Eschmann C, Kuo CM, Kuo CH, Boller C. Unmanned Aircraft Systems for Remote Building Inspection and Monitoring. 6th European
Workshop on Structural Health Monitoring; 2013:1‐8.
84. Park CY, Kim JH, Jun SM. A structural health monitoring project for a composite unmanned aerial vehicle wing: overview and
evaluation tests. Struct Control Health Monit. 2012;19(7):567‐579.
85. Hallermann N, Morgenthal G. Unmanned aerial vehicles (UAV) for the assessment of existing structures. 36th International Association
for Bridge and Structural Engineering Kolkata Symposium. 2013;101(14):1‐8.
86. Hallermann N, Morgenthal G. Visual inspection strategies for large bridges using unmanned aerial vehicles (UAV). In: 7th International
Conference on Bridge Maintenance, Safety and Management; 2014.
87. Ortiz P, Vazquez MA, Martin JM, et al. The diagnosis of the royal tobacco factory of seville assisted by quad‐rotor helicopter. In: RICH
2012 1st Conference on Robotics Innovation for Cultural Heritage; 2013.
88. Ellenberg A, Branco L, Krick A, Bartoli I, Kontsos A. Use of unmanned aerial vehicle for quantitative infrastructure evaluation. J Infrast
Syst. 2015;21(3):04014054.
89. Galarreta JF, Kerle N, Gerke M. UAV‐based urban structural damage assessment using object‐based image analysis and semantic
reasoning. Natur Haza Earth Syst Sci. 2015;15(6):1087‐1101.
90. Hallermann N, Morgenthal G, Rodehorst V. Unmanned aerial systems (UAS) – case studies of vision‐based monitoring of ageing structures.
International Symposium Non‐Destructive Testing in Civil Engineering; 2015.
91. Sankarasrinivasan S, Balasubramanian E, Karthik K, Chandrasekar U, Gupta R. Health monitoring of civil structures with integrated
UAV and image processing system. Proce Comp Sci. 2015;54:508‐515.
92. Zhou H, Kong H, Wei L, Creighton D, Nahavandi S. Efficient road detection and tracking for unmanned aerial vehicle. IEEE
Transactions on Intelligent Transportation Systems. 2015;16(1):297‐309.
93. Cho OH, Kim JC, Kim EK. Context‐aware high‐rise structure cracks image monitoring system using unmanned aerial vehicles.
Int J Control Autom. 2016;9(9):11‐18.
94. Ham Y, Han KK, Lin JJ, Golparvar‐Fard M. Visual monitoring of civil infrastructure systems via camera‐equipped unmanned aerial
vehicles (UAVs): a review of related works. Visual Eng. 2016;4(1):1‐8.
95. Na WS, Baek J. Impedance based non‐destructive testing method combined with unmanned aerial vehicle for structural health
monitoring of civil infrastructures. Appl Sci. 2016;6:1‐9.
96. Franke KW, Rollins KM, Ledezma C, et al. Reconnaissance of two liquefaction sites using small unmanned aerial vehicles and structure
from motion computer vision following the April 1, 2014, Chile Earthquake. J Geot Geoenvi Eng. 2017;143(5):04016125.
97. Qidwai U, Akbar MA. Coordinated robotic system for civil, structural health monitoring. MATEC Web of Conferences. 2017;120:01003.
98. Reagan D, Sabato A, Niezrecki C. Unmanned Aerial Vehicle Acquisition of Three‐Dimensional Digital Image Correlation Measurements for
Structural Health Monitoring of Bridges. SPIE Smart Structures and Materials + Non‐destructive Evaluation and Health Monitoring;
2017a 1016909.
99. Reagan D, Sabato A, Niezrecki C. Feasibility of Using Digital Image Correlation for Unmanned Aerial Vehicle Structural Health
Monitoring of Bridges. Structural Health Monitoring: An International Journal; 2017b 147592171773532.
100. Chiu WK, Ong WH, Kuen T, Courtney F. Large structures monitoring using unmanned aerial vehicles. Proced Eng. 2017;188:415‐423.
101. Yoon H, Spencer BF Jr. Structural displacement measurement using unmanned aerial system. Comput Aided Civ Inf Eng. 2017;0:1‐10.
102. Morgenthal G, Hallermann N. Quality assessment of unmanned aerial vehicle (UAV) based visual inspection of structures. Adv Struct
Eng. 2014;17(3):289‐302.
103. Ellenberg A, Kontsos A, Moon F, Bartoli I. Bridge related damage quantification using unmanned aerial vehicle imagery. Struct Control
Health Monit. 2016;23(9):1168‐1179.
SONY ET AL. 21 of 22

104. Kim H, Lee J, Ahn E, Cho S, Shin M, Sim SH. Concrete crack identification using a UAV incorporating hybrid image processing. Sensors.
2017;17(9):2052.
105. Omar T, Nehdi ML. Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography. Autom Construct.
2017;83:360‐371.
106. Duque L, Seo J, Wacker J. Bridge deterioration quantification protocol using UAV. J Bridge Eng. 2018;23(10):04018080.
107. Germanese D, Leone GR, Moroni D, Pascali MA, Tampucci M. Long‐term monitoring of crack patterns in historic structures using
UAVs and planar markers: a preliminary study. J Imaging. 2018;4(8):99.
108. Lei B, Wang N, Xu P, Song G. New crack detection method for bridge inspection using UAV incorporating image processing. J Aero Eng.
2018;31(5):04018058.
109. Lau SL, David K. Movement Recognition Using the Accelerometer in Smartphones. Future Network and Mobile Summit; 2010:1‐9.
110. Yu Y, Zhao X, Ou J. A new idea: mobile structural health monitoring using smartphones. In: 2012 Third International Conference on
Intelligent Control and Information Processing; 2012.
111. Morgenthal G, Höpfner H. The application of smartphones to measuring transient structural displacements. J Civil Struct Health
Monitor. 2012;2(3–4):149‐161.
112. Höpfner H, Morgenthal G, Schirmer M, Naujoks M, Halang C. On measuring mechanical oscillations using smartphone sensors. ACM
SIGMOBILE Mobile Computing and Communications Review. 2013;17(4):29‐41.
113. Dashti S, Bray JD, Reilly J, Glaser S, Bayen A, Mari E. Evaluating the reliability of phones as seismic monitoring instruments. Earthq
Spectra. 2014;30(2):721‐742.
114. Cimellaro GP, Scura G, Renschler CS, Reinhorn AM, Kim HU. Rapid building damage assessment system using mobile phone technol-
ogy. Earthquake Eng Eng Vib. 2014;13(3):519‐533.
115. Shrestha A, Dang J, Wang X. Development of a smart‐device‐based vibration‐measurement system: effectiveness examination and
application cases to existing structure. Struct Control Health Monit. 2017;25(3):e2120.
116. Kong Q, Allen RM, Kohler MD, Heaton TH, Bunn J. Structural health monitoring of buildings using smartphone sensors. Seismol
Res Lett. 2018;89(2A):594‐602.
117. Oraczewski T, Staszewski WJ, Uhl T. Nonlinear acoustics for structural health monitoring using mobile, wireless and smartphone‐based
transducer platform. J Intel Mater Syst Struct. 2015;27(6):786‐796.
118. Feng M, Fukuda Y, Mizuta M, Ozer E. Citizen sensors for SHM: use of accelerometer data from smartphones. Sensors.
2015;15(2):2980‐2998.
119. Ozer E, Feng MQ, Feng D. Citizen sensors for SHM: towards a crowdsourcing platform. Sensors. 2015;15(6):14591‐14614.
120. Zhao X, Han R, Ding Y, et al. Portable and convenient cable force measurement using a smartphone. J Civil Struct Health Monitoring.
2015;5(4):481‐491.
121. Yu Y, Han R, Zhao X, et al. Initial validation of mobile‐structural health monitoring method using smartphones. Int J Distri Sensor
Net. 2015;11(2):274391.
122. Zhao X, Ri K, Han R, Yu Y, Li M, Ou J. Experimental research on quick structural health monitoring technique for bridges using a
smartphone. Adv Mater Sci Eng. 2016a;1‐14.
123. Zhao X, Liu H, Yu Y, et al. Displacement monitoring technique using a smartphone based on the laser projection‐sensing method. Sen-
sors and Actuators Phys. 2016b;246:35‐47.
124. Ozer E, Feng MQ. Synthesizing spatiotemporally sparse smartphone sensor data for bridge modal identification. Smart Materials and
Structures. 2016;25(8):085007.
125. Zhao X, Han R, Yu Y, et al. Smartphone‐based mobile testing technique for quick bridge cable–force measurement. Journal of Bridge
Engineering. 2017a;22(4):06016012.
126. Ozer E, Feng MQ. Direction‐sensitive smart monitoring of structures using heterogeneous smartphone sensor data and coordinate
system transformation. Smart Materials and Structures. 2017a;26(4):045026.
127. Zhao X, Zhao Q, Yu Y, et al. Distributed displacement response investigation technique for bridge structures using smartphones.
J Perform Constr Fac. 2017b;31(4):04017029.
128. Wang N, Ri K, Liu H, Zhao X. Structural displacement monitoring using smartphone camera and digital image correlation. IEEE Sens J.
2018;18(11):4664‐4672.
129. Ozer E, Feng D, Feng MQ. Hybrid motion sensing and experimental modal analysis using collocated smartphone camera and
accelerometer. Meas Sci Tech. 2017b;28(16):105903.
130. Feldbusch A, Sadegh‐Azar H, Agne P. Vibration analysis using mobile devices (smartphones or tablets). Procedia Engineering.
2017;199:2790‐2795.
22 of 22 SONY ET AL.

131. Zeng H, Park H, Smith BL, Parkany E. Feasibility assessment of a smartphone‐based application to estimate road roughness. KSCE
Journal of Civil Engineering. 2017;1‐10.
132. Lorenc SJ, Handlon BE, Bernold LE. Development of a robotic bridge maintenance system. Automation in Construction.
2000;9(3):251‐258.
133. Tung PC, Hwang YR, Wu MC. The development of a mobile manipulator imaging system for bridge crack inspection. Autom Construct.
2002;11(6):717‐729.
134. Oh JK, Jang G, Oh S, et al. Bridge inspection robot system with machine vision. Autom Construct. 2009;18(7):929‐941.
135. Zhu D, Yi X, Wang Y, Lee K‐M, Guo J. A mobile sensing system for structural health monitoring: design and validation. Smart Materials
and Structures. 2010;19(5):55011.
136. Lim RS, La HM, Shan Z, Sheng W. Developing a crack inspection robot for bridge maintenance. In: Proceedings of IEEE International
Conference Robotics and Automation. China: Shanghai; 2011:6288‐6293.
137. Zhu D, Guo J, Cho C, Wang Y, Lee KM. A wireless mobile sensor network for the system identification of a space frame bridge. IEEE/
ASME Trans Mechatron. 2012;17(3):499‐507.
138. La, HM, Lim, RS, Basily, B, Gucunski, N, Yi, J, Maher, A, Parvardeh, H. (2013), “Autonomous robotic system for high‐efficiency non‐
destructive bridge deck inspection and evaluation”, 2013 IEEE International Conference on Automation Science and Engineering (CASE).
139. Matarazzo TJ, Pakzad SN. Structural identification for mobile sensing with missing observation. Journal of Engineering Mechanics,
18. 2016;142(5):1‐18.
140. Marulanda J, Caicedo JM, Thomson P. Modal identification using mobile sensors under ambient excitation. J Comput Civil Eng.
2016;31(2):04016051.
141. Goorts K, Phillips S, Ashasi‐Sorkhabi A, Narasimhan S. Structural control using a deployable autonomous control system. International
Journal of Intelligent Robotics and Applications. 2017;1(3):306‐326.

How to cite this article: Sony S, Laventure S, Sadhu A. A literature review of next‐generation smart sensing
technology in structural health monitoring. Struct Control Health Monit. 2019;e2321. https://doi.org/10.1002/
stc.2321