drones
Article
Real-Time Registration of Unmanned Aerial Vehicle Hyperspectral
Remote Sensing Images Using an Acousto-Optic Tunable
Filter Spectrometer
Hong Liu 1,2,3,4 , Bingliang Hu 1,3,4, *, Xingsong Hou 2 , Tao Yu 1,3,4 , Zhoufeng Zhang 1,3 , Xiao Liu 1,3 ,
Jiacheng Liu 1,3,4 and Xueji Wang 1,3
Citation: Liu, H.; Hu, B.; Hou, X.; Yu, T.;
Zhang, Z.; Liu, X.; Liu, J.; Wang, X.
Real-Time Registration of Unmanned
Aerial Vehicle Hyperspectral Remote
Sensing Images Using an Acousto-Optic
Tunable Filter Spectrometer. Drones
2024, 8, 329. https://doi.org/10.3390/
drones8070329
Academic Editor: Pablo
Rodríguez-Gonzálvez
Received: 22 April 2024
Revised: 13 July 2024
Accepted: 14 July 2024
Published: 17 July 2024
Copyright: © 2024 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Drones 2024, 8, 329. https://doi.org/10.3390/drones8070329
1 Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China;
liuhong@opt.ac.cn (H.L.); yutao@opt.ac.cn (T.Y.); zhangzhoufeng@opt.ac.cn (Z.Z.); liuxiao@opt.ac.cn (X.L.);
liujiacheng@opt.ac.cn (J.L.); wangxueji@opt.ac.cn (X.W.)
2 School of Electronic and Information Engineering, Xi’an Jiao Tong University, Xi’an 710049, China;
houxs@xjtu.edu.cn
3 Key Laboratory of Spectral Imaging Technology, Chinese Academy of Sciences, Xi’an 710119, China
4 University of Chinese Academy of Sciences, Beijing 100049, China
* Correspondence: hbl@opt.ac.cn
Abstract: Differences in field of view may occur during unmanned aerial remote sensing imaging
applications with acousto-optic tunable filter (AOTF) spectral imagers using zoom lenses. These
differences may stem from image size deformation caused by the zoom lens, image drift caused
by AOTF wavelength switching, and drone platform jitter. However, they can be addressed using
hyperspectral image registration. This article proposes a new coarse-to-fine remote sensing image
registration framework based on feature and optical flow theory, comparing its performance with
that of existing registration algorithms using the same dataset. The proposed method increases the
structure similarity index by 5.2 times, reduces the root mean square error by 3.1 times, and increases
the mutual information by 1.9 times. To meet the real-time processing requirements of the AOTF
spectrometer in remote sensing, a development environment using VS2023+CUDA+OPENCV was
established to improve the demons registration algorithm. The registration algorithm for the central
processing unit+graphics processing unit (CPU+GPU) achieved an acceleration ratio of ~30 times
compared to that of a CPU alone. Finally, the real-time registration effect of spectral data during
flight was verified. The proposed method demonstrates that AOTF hyperspectral imagers can be
used in real-time remote sensing applications on unmanned aerial vehicles.
Keywords: acousto-optic tunable filter; image registration; real-time processing; spectral imaging;
UAV remote sensing
1. Introduction
Image registration technology refers to the process of geometric alignment of two
images acquired at different times, from different perspectives, and from different sen-
sors [1,2]. Image registration in the field of remote sensing is an essential part of many
remote sensing image processes, such as object detection, urban development assessment,
geographic change assessment, hyperspectral image stitching, precision agriculture, and
biomedical applications [3–8]. These tasks need to be carried out to achieve successful reg-
istration; thus, further research on remote sensing image registration is deemed necessary
and crucial.
Acousto-optic tunable filter (AOTF) spectral imagers are frame scanning-based im-
agers with an adjustable number of spectral channels, and they capture the entire data
cube by sequentially exposing each band. However, in remote sensing flight experiments,
changes in sensor position and attitude are caused by airflow disturbances and drone vibra-
tions, resulting in spatial positional deviations in different spectral bands. Different spectral
https://www.mdpi.com/journal/drones
Drones 2024, 8, 329 2 of 28

bands cannot be fully matched in the original data cube, leading to spectral registration
errors. Therefore, it is necessary to implement post-processing registration correction.
AOTFs present advantages that include small size, light weight, no moving parts,
flexible wavelength selection, and strong environmental adaptability [9], thus making
them suitable as spectroscopic devices for high-resolution imaging spectrometers in aerial
remote sensing [10,11]. However, in remote sensing imaging based on AOTF hyperspectral
instruments, it is imperative to address any issues that may arise in the collected data.
Sharikova et al. [12] studied the spatial spectral distortion caused by acousto-optic
diffraction and performed spatial and spectral calibration on imaging spectrometers based
on AOTFs to develop a technique for balancing the transmittance of AOTFs throughout
the entire working spectral range. To obtain undistorted data, spectral and spatial non-
uniformity must be corrected. The calibration process included hardware spectral tuning
of acousto-optic devices and mathematical corrections to software. The proposed method
can perform real-time hardware calibration of hyperspectral devices based on AOTFs and
software calibration of display results. This method is universal and suitable for other
imaging spectrometers based on acousto-optic monochromaticization.
Zhang and Zhao [13] focused on the design and analysis of non-collinear AOTFs, and
they used a refractive index correction to resolve the geometric parameter introduction error
(0.5◦ or greater) that cannot be ignored in tellurium dioxide (TeO2 ) non-collinear AOTFs. This
basic theoretical research is crucial for the design and application of non-collinear AOTFs.
Zhao et al. [14] studied the spectral and spatial characteristics of AOTFs, such as
tuning curves, spectral resolution, angular aperture, and diffraction efficiency, determined
by the device’s acousto-optic crystal configuration and piezoelectric transducers. For high-
throughput spectral imaging applications, it is crucial to expand the spectral bandwidth
and angular aperture during the design phase of AOTFs. Therefore, this article analyzes
and studies the phase mismatch caused by incident angle or wavelength using phase
diagrams. In addition, a performance parameter analysis model was established for the
design of large angle aperture AOTF devices based on mercuric bromide crystals, and
the effects of crystal and transducer design parameters on spectral bandwidth and angle
aperture were evaluated.
Yu et al. [15] investigated spectral drift, which is a unique challenge when using an
AOTF spectrometer to observe moving targets, and revealed that an online spectral calibration
method based on acousto-optic interaction is required. A reverse ray tracking model was
constructed using the imaging position and driving frequency of the target spectrum, and
it achieved real-time calibration of spectral data and ensured the stability and accuracy
of subsequent target detection, recognition, and tracking. Experimental verification was
conducted using the developed parallel incident light mid-infrared AOTF spectral detection
system. The results showed that for simulated moving targets with different fields of view,
the correction accuracy of spectral drift was greater than 4.45%. This improvement enhanced
the application capability of the spectral detection of moving targets.
In our previous research on the design of an electric zoom lens AOTF spectrometer,
we addressed the issue of image blurring, which causes image size deformation due to
zoom, image drift due to wavelength switching, and drone platform shaking in remote
sensing applications [16]. The issue of cube registration based on AOTF spectrometer
imaging data has not been previously studied. According to general image processing and
remote sensing image processing methods, it is widely recognized that current methods
of solving remote sensing spectral data image registration mainly include grayscale and
template-based, feature-based, and domain transformation-based methods, as well as
machine learning and artificial intelligence-based methods [17]. Therefore, the methods for
addressing remote sensing spectral data image registration can be roughly divided into
these four categories.
Grayscale and template-based methods directly use correlation operations and other
approaches to calculate the correlation value to identify the best matching position. Block-
ing matching is performed to search for sub-images similar to the template image in
Drones 2024, 8, 329 3 of 28

another image based on a known template image. Li et al. [18] proposed a deep learning
semantic template matching framework for remote sensing image registration. Driven by
learning-based methods, reference images and template images are taken as inputs and
mapped to the semantic distribution positions of the corresponding reference images. Ruiqi
et al. [19] proposed a template-matching method based on a deep global feature-based
template-matching method (GFTM) to achieve fast and accurate multimodal image regis-
tration. The method performs fast template matching on global deep features to search for
positions with maximum similarity. A large number of experimental results on optical and
synthetic aperture radar (SAR) images have shown that the proposed method is effective
for multimodal image registration.
In feature-based methods, the features of the image are extracted, feature descriptors
are generated, and the features of the two images are matched based on the similarity
of the descriptors. The features of an image can mainly be divided into points, lines
(edges), regions (faces), and other features, as well as local features and global features.
The extraction of regional (surface) features is relatively cumbersome and time-consuming;
therefore, point features and edge features are mainly used. Point features include the
Harris, histogram of oriented gradient (HOG), local binary pattern (LBP), scale-invariant
feature transform (SIFT), speeded up robust features (SURF), binary robust independent
elementary features (BRIEF), smallest univalue segment assimilating nucleus (SUSAN),
features from accelerated segment test (FAST), fast retina keypoint (FREAK), binary robust
invariant scalable keypoints (BRISK), oriented FAST and rotated BRIEF (ORB) algorithms
and classifiers [6–8]. Edge features include the LoG operator, Robert operator, Sobel opera-
tor, Prewitt operator, and Canny operator. Ye et al. [20] proposed a new keypoint feature
detector aimed at the simultaneous extraction of corners and spots and the calculation
of SIFT descriptors for detected corners and spots and applied them jointly for remote
sensing image registration. Wu et al. [21] proposed a robust and accurate feature point-
matching framework. An improved SIFT method was first proposed for feature detection
and matching, and it was applied to automatic remote sensing image registration. Zhang
et al. [22] proposed an improved algorithm for the SURF classic algorithm, which is a
short time and high-precision image registration algorithm that can meet the registration
requirements of remote sensing image stitching. Chen et al. [23] proposed an iterative
image registration method for remote sensing images, known as iterative scale-invariant
feature transform (ISIFT). This method extends the registration system based on SIFT to a
closed-feedback SIFT system that includes a correction feedback loop, iteratively updating
the correction parameters. The experimental results show that compared with traditional
SIFT-based methods and state-of-the-art methods, ISIFT improves performance and yields
better registration accuracy. Jhan et al. [24] proposed a normalized SURF (N-SURF), which
can substantially increase the number of correct matches between different multispectral
image pairs, enabling one-step image registration. Additionally, they developed an auto-
matic multispectral image registration tool suitable for multi-lens multispectral cameras.
Wu et al. [25] proposed a two-step remote sensing image registration method based on
local and global constraints. Experiments on multiple remote sensing image datasets have
shown that this method is more robust and accurate than existing methods.
Based on domain transformation, phase correlation (Fourier Merlin transform), Walsh
transform, and wavelet transform are used for registration in new domains. Ordóñez
et al. [26] proposed a phase correlation algorithm based on FFT and developed a graphics
processing unit (GPU) to register two remote-sensing hyperspectral images. The proposed
algorithm is based on multi-layer fractional Fourier transform and logarithmic polar plots.
Liu et al. [27] proposed a multi-constraint registration algorithm based on variational
inference for complex remote sensing image registration problems. The experimental
results showed that compared with other point set matching algorithms, their method
demonstrated strong performance in terms of robustness and matching accuracy. Zhou
et al. [28] proposed a novel image registration algorithm using wavelet transform and
Drones 2024, 8, 329 4 of 28

matrix multiple discrete Fourier transform, and the results showed that it can perform
sub-pixel registration via full image-based methods but with shorter computation times.
Regarding machine learning and artificial intelligence-based methods, Lee et al. [29]
proposed a remote sensing registration framework based on convolutional neural networks
to improve the registration accuracy between two remote sensing images obtained from
different times and viewpoints. The proposed high-precision registration framework was
evaluated using the KOMPSAT-3 dataset and obtained a minimum root mean square
error value of 34.922 based on all control points and improved the matching accuracy
by 68.4% compared to traditional registration frameworks. Zeng et al. [30] proposed an
image registration method based on hierarchical convolutional features and applied it to
improve the efficiency of large-scale forestry image stitching generation. This method uses
a deep learning architecture to adaptively obtain image features from deep convolutional
neural networks. The experimental results showed that this method can detect and match
image feature points with marked spectral differences and effectively extract feature points
to generate accurate image registration and stitching results. Ye et al. [31] proposed a
multi-scale framework with unsupervised learning called MU Net. Without expensive
ground truth labels, MU Net directly learns end-to-end mapping from image pairs to their
transformation parameters. The experimental results indicated that MU Net achieved more
comprehensive and accurate registration between image pairs with geometric and radiative
distortions. Chen et al. [32] proposed a dense connected neural network (RBDCNN) that
was improved by residual blocks to extract feature values, which improved registration
performance and utilized the distance difference between the transformation matrices of
reference images and floating images. Compared with existing methods, the registration
results were closer to that of the reference image.
With the increasing demand for high accuracy in remote sensing image registration,
the complexity of remote sensing image registration algorithms is also increasing, which
has led to calculation times that are too long to be used in situations with high real-time re-
quirements. Hyperspectral image registration is a real-time application-related task, such as
environmental disaster management or search and rescue scenarios [33]. The compute uni-
fied device architecture (CUDA) has advantages such as low cost, low power consumption,
good portability, and flexible code modification. Therefore, by combining the advantages
of a GPU in general computing and the processing speed issues faced by remote sensing
image registration, a fast remote sensing image registration algorithm based on CUDA
parallel computing can be developed. Liu et al. [34] utilized swarm intelligence GPUs to
accelerate SAR image registration in parallel and achieve fully parallelized registration.
The experimental results showed that this method can achieve approximately 40-times
higher acceleration. Ordóñez et al. [35] proposed the first method of implementing hyper-
spectral KAZE (HSI-KAZE) with multiple nodes and GPUs for jointly registering band and
multispectral images. In this method, different multispectral datasets are distributed be-
tween available nodes in the cluster using MPI, and CUDA utilizes the parallel flow-based
capabilities of the GPUs within each node. Subsequently, HSI-KAZE was used in cluster
systems to achieve multi-GPU registration of high-resolution multispectral images. Zhang
et al. [36] proposed a multi-frame image registration algorithm and its parallel design
method for high-resolution images. Compared with feature point algorithms and deep
learning algorithms, the proposed algorithm and its parallel design considerably improve
the registration accuracy and speed of high-resolution optical remote sensing images.
To solve the issues of image size deformation, image drift, and image jitter in the
unmanned aerial vehicle remote sensing spectral data cube collected using the AOTF
spectrometer based on the electric zoom lens, we used image registration methods to solve
these problems. Compared with the current methods for hyperspectral image registration,
the predominant approaches include grayscale and template-based, feature-based, and
domain transformation-based methods, as well as some machine learning and artificial
intelligence methods. A coarse-to-fine remote sensing image registration method was
proposed based on feature and optical flow theory. In the coarse registration stage, the
Drones 2024, 8, 329 5 of 28

feature-based registration method was used to solve the registration problem of scale trans-
formation, rotation, and other overall transformations. In the fine registration stage, the
image registration method (based on optical flow theory) was used to solve the registration
problem of the local details in the image, such as issues with image zoom produced by the
use of a zoom lens and differences in the field of view caused by small jitter movements.
The main objectives of this study were as follows:
1. To propose a new coarse-to-fine remote sensing image registration framework based
on feature and optical flow theory. The data cube composed of collected spectral
segments registers the data of each spectral segment (registration between adjacent
spectral segments) after resolving image blurring and spectral segment drift of the
AOTF spectrometer using a fast zoom lens. The performance of the proposed method
was compared with that of other advanced algorithms.
2. To construct a VS2023+CUDA+OPENCV development environment for the improved
demons registration algorithm based on optical flow theory, which is highly suitable
for distributed and parallel processing. Parallel processing using the compute unified
device architecture (CUDA) was performed to achieve rapid registration and enable
real-time processing.
3. To verify the proposed method based on the existing AOTF unmanned airborne
spectrometer platform. The data cubes on each shooting waypoint were registered
online and met the requirements of real-time registration on unmanned aerial vehicles
(UAVs) and real-time processing on aircraft.

2. Related Work
This section provides information about the proposed method. First, the airborne AOTF
spectrometer based on a zoom lens is introduced. Then, the image registration algorithm
based on the optical flow theory is described, constituting the main part of the fine registration.
Finally, the image processing algorithm acceleration based on GPU use is introduced.

2.1. Airborne AOTF Spectrometer for UAVs Based on the Zoom Lens
With the rapid development of material technology and the increasing maturity of
optical device manufacturing technology, optical splitters have transformed from tradi-
tional prisms, gratings, etc., to new devices with higher spectral resolution, miniaturization,
integration, and intelligence [37]. AOTF represents an all-solid-state filtering and polariza-
tion modulation device that can achieve fast electronic control tuning without mechanical
moving components. This technology has many advantages, such as compact structure,
high diffraction efficiency, and wide tuning range. Moreover, it has been applied in the
development process of spectral imaging systems [38,39].
The airborne imaging system consists of an AOTF imaging spectrometer based on an
electric zoom lens, an AOTF driver, a MINI-PC with GPU, and a battery. The composition
diagram of the unmanned aerial vehicle hyperspectral imaging system based on AOTF is
shown in Figure 1.
The core optical path structure of the AOTF spectrometer based on an electric zoom
lens is shown in Figure 2.
In Figure 2, the core optical path structure of the AOTF spectrometer is presented, il-
lustrating (1) electrically controlled zoom lens; (2) front objective lens; (3) aperture diagram;
(4) collimating lens; (5) linear polarizer; (6) AOTF, composed of tellurium dioxide (TeO2 )
crystals and piezoelectric transducers; (7) linear polarizer; (8) secondary imaging lens;
(9) CMOS detector; (10) MINI-PC control and data acquisition system; and (11) RF driver.
The incident beam is refracted by an electrically controlled variable angle lens (1), a
front objective lens (2), aperture diagram (3), and a collimating lens before being vertically
incident on the surface of a linear polarizer (4). After polarization, the beam is vertically
incident on the surface of the AOTF module. The incident light and ultrasound interact
with each other inside the AOTF module to produce a diffracted beam. After passing
through the linear polarizer (6), the beam is focused by a secondary imaging lens (8) on the
 Drones 2024, 8, 329 6 of 28

Drones 2024, 8, x FOR PEER REVIEW
s 2024, 8, x FOR PEER REVIEW
imaging surface of the CMOS detector. Subsequently, the collected data are processed by
the MINI-PC. Specifically, the polarization direction of the linear polarizer (5) is parallel to
6 of 2
the acoustic optical interaction plane of the AOTF and perpendicular to the polarization 6
direction of the linear polarizer (7). The purpose of using a polarizer (7) is to filter out the
0-level transmitted light.

Figure 1. Composition diagram of the unmanned aerial vehicle hyperspectral imaging system based
on AOTF. (a) Components of the AOTF imaging system based on the zoom lens: AOTF imaging
spectrometer, AOTF driver, MINI-PC with GPU, and battery. (b) AOTF spectrometer airborne im
Figure 1. Composition
Figure diagram
1. Composition ofthe
diagram of
aging system based on a zoom lens.
theunmanned
unmannedaerialaerial
vehiclevehicle hyperspectral
hyperspectral imaging
imaging system based system
on AOTF. (a) Components
on AOTF. (a) Components of theAOTF
of the AOTF imaging
imaging system
system based onbased on the
the zoom lens:zoom
AOTF lens: AOTF im
imaging
spectrometer,
spectrometer, AOTFAOTF driver,MINI-PC
driver, MINI-PC with
withGPU,
GPU,andand
battery. (b) AOTF
battery. (b) spectrometer
AOTF airborne airborn
spectrometer
The core optical path structure of the AOTF spectrometer based on an electric zoom
imaging system based on a zoom lens.
aging
lenssystem
is shownbased on a zoom
in Figure 2. lens.

The core optical path structure of the AOTF spectrometer based on an electric
lens is shown in Figure 2.

Figure 2. Core optical path structure diagram of AOTF spectrometer based on electric zoom lens.
Figure 2. Core optical path structure diagram of AOTF spectrometer based on electric zoom lens.

In Figure 2, the
The design core for
materials optical pathspectrometer
the AOTF structure ofprototype
the AOTF spectrometer
based is presented
on an electric zoom
lens are shown in Table 1.
illustrating (1) electrically controlled zoom lens; (2) front objective lens; (3) aperture dia
gram;
Figure 2. (4) collimating
Core optical pathlens; (5) linear
structure polarizer;
diagram (6) AOTF,
of AOTF composed
spectrometer of on
based tellurium
electric dioxide
zoom l
(TeO2) crystals and piezoelectric transducers; (7) linear polarizer; (8) secondary imaging
lens;
In (9) CMOS
Figure 2, detector;
the core (10) MINI-PC
optical path control
structureandofdata
the acquisition system; andis(11)
AOTF spectrometer RF
prese
driver.
illustrating (1) electrically controlled zoom lens; (2) front objective lens; (3) apertur
Drones 2024, 8, 329 7 of 28

Table 1. Materials for the AOTF spectrometer prototype using an electric zoom lens.

Component Parameter Specification Component Parameter Specification

Wavelength 400–1000 nm Focal length 16 mm
FWHM ≤8 nm Objective lens Image plane 1/1.2′′
(M112FM16)
Diffraction
≥75% Aperture F2.0–F16.0
efficiency
AOTF filter Separation angle ≥4◦ Focal length 50 mm
(SGL30-V-12LE)
Aperture angle ≥3.6◦ Collimating lens Image plane 1′′
(V5014-MP)
Primary deflection
≥2.17◦ Aperture F1.4–F16.0
angle
Optical aperture 12 × 12 mm Wavelength range 300–2700 nm
Linear polarizer
Frequency range 43–156 MHz (R5000490667) Extinction ratio >800:1
Stability frequency 10 Hz Size 25.4 mm
AOTF driver
Frequency
0.1 MHz Detector PYTHON5000
resolution
Aperture 16 mm CMOS camera Pixel size 4.8 × 4.8 µm
Motorized zoom (MV-CA050-
lens (EL-16-40-TC- Response time 5 ms 20UM) Resolution 2592 × 2048
VIS-5D) −10 to +10
Focal range Interface USB 3.0
diopters

2.2. AOTF Spectral Characteristics

The main material of the AOTF crystal is TeO2 . Due to the characteristics of Bragg
diffraction in TeO2 crystals, the light passing through the crystal has different deflection
angles for different wavelengths of diffracted light, resulting in angular displacement
errors [40]. Previous studies have demonstrated the relationship between the diffraction
angle and wavelength of AOTF crystals [41,42], which is described below.
The geometric relationship is satisfied by the incident light angle θi , diffracted light
angle θd , and external diffraction angle β is defined as follows:

sinβ = sin(θi − θd ) (1)

where nd is the refractive index of TeO2 crystals corresponding to the wavelength of the
diffracted light. In addition, nd = no , where no is the refractive index of o light when the
optical rotation of the crystal is not considered. The order angle of the diffracted light can
be written as follows:  2
no
tan θd = tan θi (2)
ne
Equation (2) is a function of incident light wavelength. The geometric relationship is
satisfied in (3) when the emitting end face of the TeO2 crystal is not parallel to the incident
end face:
sin β = nd sin(θi − θd − θω ) (3)
where θω is the increased wedge angle of the exit end face of the crystal. Generally, the
relationship between the refractive index and wavelength is shown below:

2.5488λ2 1.1557λ2
no 2 = 1 + + (4)
λ2 − 0.13422 λ2 − 0.26382

2.8525λ2 1.5141λ2
ne 2 = 1 + + (5)
λ2 − 0.13422 λ2 − 0.26312
 where n is the refractive index of e-light, and both n and n are functions of
length of light λ. Based on the above equations, the expression of the external
angle of the e-light crystal can be obtained as follows:
Drones 2024, 8, 329 8 of 28
2.5488𝜆 1.1557𝜆
𝛽 = 𝑎𝑟𝑐 𝑠𝑖𝑛 1+ +
where ne is the refractive index of e-light, and both𝜆no − 0.1342
and 𝜆 −
ne are functions of 0.2638
the wavelength
of light λ. Based on the above equations, the expression ∗ sin (𝜃 external
of the 𝜃 )
− 𝜃 −diffraction angle of
the e-light crystal can be obtained as follows:
In this equation, the following
 assumptions were  made: the incident ligh
light, the incident angle 𝜃 ≈ 122.7°, − 1
and the1.1557λ
wedge angle of the crystal exit end
 2 2

2
β = arcsin + λ22.5488λ
−0.13422
+ λ2 −0.26382 (6)
0.6°. The varying relationship  between
∗sin(θi −the
θd −diffraction
θω )  angle 𝛽 and incident l
length is shown in Figure 2. Based on the figure, 𝛽 changes to 0.0419° over
In this equation, the following assumptions were made: the incident light is the e-light,
wavelength range. The translation of the central image plane is appr
the incident angle θi ≈ 22.7◦ , and the wedge angle of the crystal exit end face θω ≈ 0.6◦ . The
𝑓varying (0.0419°) between
× tanrelationship ≈ 11.7the μ𝑚diffraction
if the focal
angle βlength 𝑓 = light
and incident 16 𝑚𝑚. We can
wavelength infer that t
is shown
in the diffraction
in Figure 2. Based onangle withβ changes
the figure, respecttoto0.0419 ◦
the wavelength
over the whole over the whole
wavelength range.wavelen
The translation of the central image plane is approximately f × tan ( 0.0419 ◦ ) ≈ 11.7 µm
affects the imaging quality of the AOTF spectrometer.
if the focal length f = 16 mm. We can infer that the change in the diffraction angle with
From Figure 3, it can be seen that when the angle displacement error is
respect to the wavelength over the whole wavelength range affects the imaging quality of
small
the AOTF spectral range, such as 500–550 nm), the image captured on the detector f
spectrometer.
will experience3,spectral
From Figure drift.that
it can be seen When
when the angledisplacement
the angle displacement error error
is smallis(in
large
a (wid
small spectral range, such as 500–550 nm), the image captured on the
range, such as 500–900 nm), and the focal length of the imaging objective is fixdetector focal plane
will experience spectral drift. When the angle displacement error is large (wide spectral
cause defocusing
range, such as 500–900ofnm),
the and
scene
theon the
focal detector
length of the focal
imaging plane, resulting
objective is fixed, in image blu
it will
cause defocusing of the scene on the detector focal plane, resulting in image blurring [16].

Figure 3.3.Trend
Figure of variation
Trend betweenbetween
of variation the diffraction angle of the AOTF
the diffraction crystal
angle andAOTF
of the the wavelength
crystalofand the
incident light.
of incident light.
To address the issue of image blurring in certain spectral bands, we added a fast and
Toelectric
tunable address the issue
focusing lens inof image
front of theblurring in certain
imaging objective. Thespectral bands,
principle of we added
an electric
zoom mirror is to change its focal length through driving current, and the
tunable electric focusing lens in front of the imaging objective. The principle of zoom process
can be completed in milliseconds, which is in the same time order of magnitude as the time
zoom mirror
required is to
to switch change
bands its focal
by changing the length through
AOTF driving driving
frequency. current,
In the and the zoo
development
can be of
process completed in milliseconds,
the spectrometer, it is calibratedwhich
based onisthe
in imaging
the same timesoorder
distance that it of
canmagnit
capture clear images in every spectral range of 400–1000 nm.
Drones 2024, 8, 329 9 of 28

Using a fast zoom lens AOTF spectrometer design, clear images can be captured in
each spectral segment; however, the size of the images will vary due to the different focal
lengths of the zoom lens when capturing images in different spectral segments. To obtain
aligned data cubes, registration is a crucial step that must be performed.
In addition, the registration of data cubes can also address the issue of spectral drift
and the small movement of captured images caused by drone platform jitter in unmanned
aerial remote sensing applications.

2.3. Image Registration Algorithm Based on Optical Flow Theory

The optical flow method is commonly used for object motion estimation in video
images. It considers the inconsistency of the local motion of objects and uses the information
of pixels to individually estimate their motion in a 2D space. It is a high-precision pixel-by-
pixel model. The displacement field of image registration is similar to the optical flow field
of moving objects, and the inconsistent spatial dislocation between remote sensing images
is similar to the local motion of objects. Therefore, the optical flow method can be used for
remote sensing image registration in different time phases [43].
Ideally, the object’s brightness in the image before and after the motion is assumed to be
constant for small motions [44]. This assumption was followed in the proposed algorithm,
rendering it similar to other optical flow algorithms. If the brightness value of point (x, y) in
image I at time t is I (x, y, t), then according to the constant brightness during movements:

I ( x, y, t) = I ( x + dx, y + dy, t + dt). (7)

The Taylor expansion on the right-hand side of the above equation gives the following:

∂I dx ∂I dy ∂I dt
I ( x, y, t) = I ( x, y, t) + × + × + × + ε. (8)
∂x dt ∂y dt ∂t dt

According to the prerequisite, the motion should be small, and dx, dy, and dt denote
small quantities; therefore, the remainder ε can be ignored:

∂I dx ∂I dy ∂I
× + × + = 0. (9)
∂x dt ∂y dt ∂t

Note that
→
h i → h iT T ∂I
dy

u = dx , ∇I = ∂I ∂I
= Ix Iy , It = . (10)
dt dt ∂x ∂y
∂t
Thus,
→ →
u × ∇ I = − It , (11)
→
→ − It × ∇ I
u = . (12)
→ 2
∇I

The above formulas are the core concept of the image registration algorithm based
on optical flow theory. It shows that the offset of points between two images can be
calculated using gradient and difference information of the image over time, i.e., the
difference between the reference and the floating images. The remainder (ε) in the equation
is only ignored under the condition of small motion, while ignoring a large movement will
substantially impact the accuracy of the results, affecting the registration accuracy.

2.4. Basic Steps of GPU-Based Image Processing-Accelerated CUDA Program

The program can be divided into two aspects in CUDA architecture: the host side
(responsible for completing complex instructions) and the device side (responsible for
parallel completion of simple instructions). The host side runs on the central processing
 to be processed on the host side, allocating storage space in the v
ferring data to the video memory. The device side then perform
Drones 2024, 8, 329
the completed results to the host side, and finally releases 10 of 28
the v
The host side cannot directly manage the GPU video memory;
mission between the host and the device side must be realize
unit (CPU), whereas the device side runs on the GPU computing core. Programs running
onruntime application
the host side can be writtenprogramming
in C and C++, whereasinterface.
those on the The data
device sideare
must transm
be built into the kernel. The general CUDA calculation process involves preparing the
ultaneously. As frequent back-and-forth data transmission betw
data to be processed on the host side, allocating storage space in the video memory, and
vice is time-consuming
transferring data to the video memory. andThesubstantially reduces
device side then performs the GPU executio
calculation,
returns the completed results to the host side, and finally releases the video memory
operation should be avoided.
space [33]. The host side cannot directly manage the GPU video memory; therefore, the
The host
data transmission includes
between the hostthe
andCPU and
the device host
side mustmemory,
be realized byand the
calling thedevice
CUDA runtime application programming interface. The data are transmitted back and
memory.
forth TheAs
simultaneously. GPU can
frequent help accelerate
back-and-forth the CPU.
data transmission betweenThe program
the host and r
instructs
the the GPU to
device is time-consuming andrun whenreduces
substantially it encounters
GPU execution aefficiency,
devicethis program.
type
of operation should be avoided.
TheThe basic the
host includes steps
CPU of
andanhostaccelerating CUDA
memory, and the device program
includes GPU andwith video GPU
of allocating
memory. The GPU can CPU helpmemory and
accelerate the CPU.GPU videoruns
The program memory,
on the hosttransmitting
first and
instructs the GPU to run when it encounters a device program.
GPU, and
The basic using
steps the allocated
of an accelerating grid and
CUDA program block
with GPU imageto start the
processing kernel
consist of fu
trievesCPU
allocating the results
memory from
and GPU the
video GPU,
memory, the CPU
transmitting datamemory
from the CPUandto theGPU
GPU, vide
and using the allocated grid and block to start the kernel function. After the CPU retrieves the
ure from
results 4). the GPU, the CPU memory and GPU video memory are freed (Figure 4).

Figure 4. Basic steps of the GPU-based image processing-accelerated CUDA program.

Figure 4. Basic steps of the GPU-based image processing-accelerated CU
3. Methodology
This section introduces a detailed coarse-to-fine remote sensing image registration
3. Methodology
algorithm framework based on feature and optical flow theory and provides steps to imple-
ment the specific algorithm. The ORB feature point extraction and descriptor construction
This section introduces a detailed coarse-to-fine remote se
selected in the coarse registration stage are also introduced. Finally, the modified algorithm
ofalgorithm framework
demons correlation based
selected at the on feature
fine registration stage isand optical flow theory a
introduced.
plement the specific algorithm. The ORB feature point extraction
tion selected in the coarse registration stage are also introduced.
gorithm of demons correlation selected at the fine registration st

3.1. Algorithm Description

Drones 2024, 8, 329 11 of 28

3.1. Algorithm Description

The framework of the registration algorithm is divided into two stages: a coarse
Drones 2024, 8, x FOR PEER REVIEWregistration stage based on the feature method and a fine registration stage based on 11 of 28
optical
flow theory (Figure 5).

Figure 5. Framework of the proposed registration method.

method.

The
The coarse
coarse registration
registration stage
stage involves
involves the the following:
following:
1.
1. At the beginning of registration, features
At the beginning of registration, features of the of the floating
floating and
and reference
reference images,
images, which
which
can be one or a combination of Harris, Moravec, Haar-like, HOG, LBP, SIFT, SURF,
can be one or a combination of Harris, Moravec, Haar-like, HOG, LBP, SIFT, SURF,
BRIEF, SUSAN, FAST, CENSUS, FREAK, BRISK, ORB, etc., are extracted.
BRIEF, SUSAN, FAST, CENSUS, FREAK, BRISK, ORB, etc., are extracted.
2. Extracted features are matched to obtain feature pairs of floating and reference images.
2. Extracted features are matched to obtain feature pairs of floating and reference im-
This can be completed by brute force matching, which calculates the distance between
ages. This can be completed by brute force matching, which calculates the distance
a feature point descriptor and all other feature point descriptors, ranks the obtained
between a feature point descriptor and all other feature point descriptors, ranks the
distances, and selects the closest distance as the matching point.
obtained distances, and selects the closest distance as the matching point.
3. The feature alignment exception matching is deleted using the exception elimination
3. The feature alignment exception matching is deleted using the exception elimination
algorithm. Common methods include using a Hamming distance of less than twice
algorithm. Common methods include using a Hamming distance of less than twice
the minimum distance, cross-matching, k-nearest neighbor matching, and random
the minimum distance, cross-matching, k-nearest neighbor matching, and random
sampling consistency.
sampling consistency.
4. The transformation model from the floating image to the reference image is calculated
4. The
using transformation model from
the matching feature the floating
of removing imagepairs.
abnormal to the reference image is calcu-
5. lated
The floating image is transformed to match the referencepairs.
using the matching feature of removing abnormal image by transforming the
5. The
model floating image is transformed
and adopting appropriate to match the reference
interpolation image by
transformation, transforming
thus the
obtaining the
model and adopting
coarse registration result.appropriate interpolation transformation, thus obtaining the
coarse registration result.
The fine registration stage involves the following:
The fine registration stage involves the following:
1. The registration parameters are initialized. This may be the number of registration
1. The registration
cycles, parameters
the similarity betweenarethe initialized.
registered and Thisreference
may be images,
the number of registration
or other parameters.
2. cycles, the similarity between the registered and reference images,
It is determined whether the registration optimization conditions are met. or other parame-
3. ters.
If yes, the image is obtained after fine registration, and the process is completed.
2.
4. It
If is determined
not, the method whether
based the registration
on optical optimization
flow theory is usedconditions
to calculatearethe
met.
deformation
3. If yes, the image
displacement field. is obtained after fine registration, and the process is completed.
4. If not, the method based on optical flow theory is used to calculate the deformation
displacement field.
5. The deformation displacement field is subjected to Gaussian filtering.
6. The filtered deformation displacement field is used to interpolate the floating image.
Drones 2024, 8, 329 12 of 28

5. The deformation displacement field is subjected to Gaussian filtering.

6. The filtered deformation displacement field is used to interpolate the floating image.
7. The registration optimization conditions are calculated using normalization cross-
correlation, mutual information, structure similarity index measure (SSIM), and dif-
ference in RMSE.
8. Return to (2) and continue to judge.
The registration problems associated with scale transformation, rotation, and other
global transformations are mainly addressed at the coarse registration stage. In contrast,
those associated with local image details are primarily resolved at the fine registration
stage. The specific algorithm is described in the pseudo-code in Algorithm 1.

Algorithm 1: A specific algorithm for coarse-to-fine remote sensing image registration based on
feature and optical flow theory
Input: floating image “image01”, reference image “image02”
Output: image after registration
1: Detect ORB feature point position
2: Calculate descriptor according to ORB feature point position
3: Perform feature point matching
4: Constrain the matching points to obtain excellent matching points
5: Using the matching points, calculate the projection mapping matrix from the floating image to
the reference image
6: Use projection mapping matrix to complete rough image registration
7: Normalize the floating image and reference image after coarse registration and resize the square
8: Find the gradient of the reference image
9: Perform iteration operations as follows
10: Calculate coordinate offset code
11: Determine the gradient of floating image and improve the demons algorithm to determine
the gradient
12: Perform Gaussian smoothing for coordinate offset to reduce burrs
13: Apply the pixel resampling code
14: Perform until convergence
15: Complete fine image registration

3.2. ORB Feature Point Extraction and Descriptor Construction

ORB is a rapid local feature detection operator proposed by Rublee et al. [45]. The
ORB algorithm is an improved version of the original FAST and BRIEF algorithms [46],
constructing a Gaussian pyramid and gray centroid to compensate for the scale and rotation
invariances of the FAST algorithm and the rotation invariance of the BRIEF algorithm. The
rotated BRIEF (RBRIEF) algorithm is used to construct the feature descriptor. The ORB
algorithm has a fast-computing speed, strong real-time performance, and is insensitive to
image noise.
ORB uses the FAST algorithm to detect image feature points. The FAST corner de-
tection algorithm initially selects a central pixel (x, y) to draw a circle with a radius of
3 pixels and sets a threshold σ. The size of pixel points (x, y) is comparable to the 16-pixel
values determined by the circumference in sequence. If the gray value of z pixel points on
the circumference is >I (x, y) + σ or <I (x, y) − σ, then the selected pixel (x, y) is an image
feature point, where I (x, y) represents the gray value of the pixel (x, y), and z is 9 or 12 [47].
It is necessary to obtain a principal direction for the feature to achieve feature points
with rotation invariance. First, the feature points in the FAST corner set are taken as the
center, and the gray centroid within a certain range of the feature is calculated. Then, a
vector is constructed with the feature point and its gray centroid, and the main direction of
the FAST feature point is obtained using the vector direction.
The neighborhood moments of feature points are defined as:

m pq = ∑ x,y x p yq I (x, y), (13)

Drones 2024, 8, 329 13 of 28

where I ( x, y) represents the gray value of each pixel ( x, y) in the image: p, q ∈ (0, 1).
The centroid of the neighborhood moment of the feature point is:
 
m10 m01
C= , . (14)
m00 m00

The main direction of FAST feature points is expressed as:

 
m01
θ = arctan . (15)
m10

The traditional ORB algorithm uses RBRIEF descriptors to extract binary descriptors
from feature points, employing the main direction of feature points to determine the
direction of binary descriptors. This addresses the issue of the lack of rotation invariance in
BRIEF descriptors. The specific steps of this method are as follows:
First, a pixel block p with the size of N × N is selected after smoothing so that the gray
values at the midpoint x, y of the pixel block are p( x ) and p(y). The binary test criterion τ
is defined as: 
1, p( x ) < p(y)
τ ( p; x, y) = . (16)
0, p( x ) ≥ p(y)
A total of n pairs of position coordinates ( xi , yi ) are selected around the feature points.
An n-dimensional binary feature code string is obtained by comparing the coordinates with
the binary test criterion:
f n ( p) = ∑ 1≤i≤n 2i−1 τ ( p; xi , yi ). (17)
A matrix S of order 2 × n is defined as:
 
I, x2 , ..., xn
S= . (18)
I, y2 , ..., yn

The principal direction of feature point θ corresponding to the rotation matrix Rθ is:
 
cosθ sinθ
Rθ = . (19)
−sinθ cosθ

The matrix S is rotated to obtain a new matrix, Sθ :

Sθ = Rθ S = I. (20)

Finally, the binary descriptor with a rotation invariant property is obtained as follows:

gn ( p, θ ) = f n ( p)|( xi , yi ) ∈ Sθ . (21)

3.3. Demons Correlation Algorithm

The demons algorithm is a registration method developed based on optical flow theory,
which treats the iterative process of the algorithm as the process of each pixel in the floating
image gradually spreading to the corresponding position in the reference image. The
gray-value difference between the corresponding points of the two images is the external
force of diffusion, whereas the gradient of the corresponding points of the reference image
is the internal force of diffusion.
The offset (Ux, Uy) of point (x, y) in the reference image is calculated assuming that
the reference image is S and the floating image is M, where Sx and Sy are gradients in
the x and y directions, respectively, at point (x, y) in the reference image, and ∇ f is the
gray-value difference between the reference image and the floating image at point (x, y).
Drones 2024, 8, 329 14 of 28

Sx
Ux = ∇ f ×   , (22)
S2x + Sy2 + ∇ f 2
Sy
Uy = ∇ f ×   , (23)
S2x + Sy2 + ∇ f 2

∇ f = S( x, y) − M( x, y). (24)
Sx and Sy may be solved by various methods, including a Riemann–Liouville fractional
differential algorithm [48], a Grumwald–Letnikov fractional differential edge detection al-
gorithm [49], a Prewitt gradient operator, Scharr gradient operator, and Sobel operator as
the gradient algorithm. The Sobel operator was selected here based on a comparison of
performances.
The coordinate offset of the whole image is calculated to make the offset smooth and
continuous in the global range and is Gaussian smoothed during each iteration to avoid
image blurs after resampling.
Based on the original demon algorithm, Thirion proposed to increase the diffusion
velocity coefficient α to control the size of coordinate offset (diffusion velocity), as shown
in Formula (25) [50]. Registration accuracy in the iterative process of the algorithm usually
increases with the number of iterations α:
S
Ux = ∇ f ×   x , (25)
S2x + Sy2 + α2 × ∇ f 2

Sy
Uy = ∇ f ×   . (26)
S2x + Sy2 + α2 × ∇ f 2

The active demons algorithm adds the gradient of the floating image to the offset
calculation [51]. The internal force driving diffusion is the gradient of the reference image
in the original algorithm. The gradient of the floating image was added as the new internal
force to accelerate the convergence speed of iteration. The calculation offset is shown in the
following formulae, where Mx and My are the gradients in the x and y directions at point
(x, y) on the floating image, respectively.
 
S Mx
Ux = ∇ f ×    x +  , (27)
2 2 2
S x + Sy + α × ∇ f 2 M x + My + α 2 × ∇ f 2
2 2

 
Sy Sy
Uy = ∇ f ×    +  . (28)
S2x + Sy2 + α2 × ∇ f 2 M2x + My2 + α2 × ∇ f 2

The inertial demons algorithm was proposed on the basis of an active demons algo-
rithm. Here, the offset calculated by the previous iteration is added to the offset of the
current iteration to further improve the convergence speed and registration accuracy [52].
Its calculation is shown in the following formula, where k is the current number of iterations,
and the value of the coefficient β is between 0 and 1:
 
S Mx
Ux = β × Uxk−1 + ∇ f ×    x +  , (29)
2 2 2
S x + Sy + α × ∇ f 2 M x + My + α 2 × ∇ f 2
2 2

 
S y S y
Uy = β × Uxk−1 + ∇ f ×    +  . (30)
S2x + Sy2 + α2 × ∇ f 2 M2x + My2 + α2 × ∇ f 2
Drones 2024, 8, 329 15 of 28

4. Experiments and Discussion

This section introduces the datasets and evaluation criteria. Then, the speed, registration
rate, and robustness of 13 currently popular algorithms (ORB, KAZE, AKAZE, BRISK, optical
flow method demons, B-spline-based free-form deformation algorithm, FAST, ECC, SIFT-FSC,
SURF-GTM, SIFT, SURF, and VGG16) were compared in the registration of actual UAV remote
sensing spectral data to evaluate the performance of the proposed method. Finally, a real-time
processing experiment was conducted using UAVs for remote sensing flights.
Our algorithm runs on a 2.8–4.7 GHz Core i7-1165G7 quad-core 8-threaded processor
with 16 GB RAM, an Nvidia RTX2060 independent video card, and 6 GB GDDR6 video
memory. CUDA 11.4 was used under Microsoft Windows 10. The performance of the regis-
Drones 2024, 8, x FOR PEER REVIEW
tration process of all images on the GPU and CPU was compared. The GPU15implementation
of 28

was written in CUDA, and the CPU version was written in standard C.
memory. CUDA 11.4 was used under Microsoft Windows 10. The performance of the reg-
4.1. Datasets istration process of all images on the GPU and CPU was compared. The GPU implemen-
tation was written in CUDA, and the CPU version was written in standard C.
The AOTF spectral airborne datasets 1–6 were derived from AOTF unmanned aerial
remote sensing4.1.
forDatasets
different scenarios corresponding to roofs, towers, grassland, parking,
The AOTF
trees, and water. The drone spectral
flewairborne datasets 1–6of
at an altitude were derived
100 from AOTF unmanned
m, capturing imagesaerial
in two adjacent
remote sensing for different scenarios corresponding to roofs, towers, grassland, parking,
spectral wavelengths
trees, and of 580Theand
water. 620
drone flewnm.
at an The images
altitude of 100 m,have a size
capturing ofin2048
images × 680 pixels and
two adjacent
are shown in Figure
spectral 6.
wavelengths of 580 and 620 nm. The images have a size of 2048 × 680 pixels and
are shown in Figure 6.

Figure 6. Six pairs of original images. (a,b) Image pair 1; (c,d) image pair 2; (e,f) image pair 3;
(g,h) image pair 4; (i,j) image pair 5; (k,l) image pair 6. The left side of the image is a 580 nm image,
and the right side is a 620 nm image.
Drones 2024, 8, 329 16 of 28

4.2. Evaluation Criteria

SSIM [53], RMSE [54], MI [55], UIQI [56], and SAM [57] were used to evaluate the
effects of different algorithms and their execution in CPU and GPU.
1. SSIM
SSIM is an index to indicate the similarity between two images. It can measure the
difference between the enhanced and real image to guide the learning process. The formula
of SSIM is as follows:



2µ x µy + c1 2σxy + c2
SSI M (x, y) =   , (31)
µ2x + µ2y + c1 σx2 + σy2 + c2

where x and y are the two input images, µx and µy represent their respective averages, σx2 and
σy2 represent their respective covariances, and c1 and c2 are constants with a denominator of 0.
The values of SSIM range between 0 and 1, where 1 denotes two identical images. Thus, a
larger value indicates that more details from the original two images have been retained.
2. RMSE
A smaller RMSE value indicates better results, implying fewer differences between
the generated image and the original two images and the retention of more details.
v
1 M−1,N −1
u

M × N i=∑
[ I1 (i, j) − I2 (i, j)]2 ,
u
RMSE = t (32)
0,j=0

where M and N are the height and width of the image, respectively, I1 (i, j) is the pixel value
of the reference image at position (i, j), and I2 (i, j) is the pixel value of the registered image
at position (i, j).
3. MI
Mutual information (MI) is an important concept in information theory that describes
the correlation between two systems or the amount of information they contain. In image
registration, the mutual information between two images reflects the degree of mutual
inclusion of information between them through their entropy and joint entropy. For images
I1 and I2 , their mutual information is represented as:

MI ( I1 , I2 ) = H ( I1 ) + H ( I2 ) − H ( I1 , I2 ), (33)

where H ( I1 ) and H ( I2 ) are the entropy of images I1 and I2 , and H ( I1 , I2 ) is the joint entropy
of images I1 and I2 . When the similarity between two images is higher or the overlap
is greater, their correlation is stronger, and the joint entropy is smaller, which means the
mutual information is greater.
4. UIQI
Universal image quality index (UIQI) is a universal objective quality evaluation index
for images, for which the distortion of an image is determined by three factors: correlation
distortion, brightness distortion, and contrast distortion. Although this indicator is not
associated with the human visual system, its effectiveness is significantly higher than
the evaluation accuracy of traditional full reference image quality objective evaluation
indicators, such as the root mean square error and peak signal-to-noise ratio. Assuming X
is the original image, and Y is the image to be evaluated, then UIQI is expressed as:
  ! !
σXY 2µ X µY 2σX σY
U IQI = . (34)
σX σY µ2X + µY2 σX2 + σY2
Drones 2024, 8, 329 17 of 28

The range of UIQI in the formula is [−1, 1], where −1 is the worst effect, and 1 is
the best effect, indicating that the image to be evaluated has no distortion; µ X and σX2 are
the mean and variance of the original image pixel values, respectively; µY and σY2 are the
mean and variance of the pixel values of the image to be evaluated, respectively; σXY is the
covariance between the pixel values of the original image and the image to be evaluated.
5. SAM
The similarity between a test spectrum (pixel spectrum) and a reference spectrum can
be determined by calculating the “angle” between them. SAM can also be used to calculate
the similarity between two arrays, and its calculation result can be seen as the cosine angle
between the two arrays. The calculation formula is as follows:

dT x
SAM = cos−1 p p , (35)
(d T d) × ( x T x )

where d is the given target array, and x is the array to be tested. The smaller the output
value, the more similar the two arrays; the larger the distance between two arrays, the
lower the similarity.

4.3. Ground Experiment Results and Analysis

FSC [58], SURF-GTM [59], SIFT [60], SURF [61], and VGG16 [3] were compared in
terms of speed, registration rate, robustness, and other characteristics in the registration of
actual UAV remote sensing spectral data.
The proposed registration algorithm yielded SSIM parameters of the registered and
reference images that were closer to 1 for the six groups of image pairs than for the other
methods, and it produced the best registration effect (Table 2). The registration algorithms
based on optical flow theory (demons and FFD algorithms) showed advantages over other
algorithms, except in the Cars dataset. For the tested dataset, the VGG16 algorithm based
on artificial intelligence and machine learning did not show substantial advantages over
algorithms based only on features.

Table 2. Comparison of the structure similarity index measure (SSIM) of different registration algorithms.

Image Datasets
Algorithms
Roofs Tower Grassland Parking Trees Water
VGG16 [3] 0.1969 0.1058 0.1478 0.1925 0.0185 0.0967
ORB [58] 0.1740 0.1116 0.1093 0.1531 0.0550 0.0819
KAZE [59] 0.1741 0.0766 0.1787 0.1511 0.0064 0.0861
AKAZE [60] 0.1743 0.0772 0.1914 0.1505 0.0360 0.0851
BRISK [61] 0.1747 0.1053 0.1823 0.1502 0.0036 0.0797
Demons [62] 0.4500 0.4172 0.4868 0.4005 0.4006 0.3485
FFD [63] 0.1353 0.1319 0.3013 0.1279 0.1032 0.1510
FAST [64] 0.1742 0.0913 0.1825 0.1528 0.0453 0.0874
ECC [65] 0.5759 0.4210 0.5515 0.3294 0.3855 0.2110
SIFT-FSC [66] 0.1660 0.1556 0.3295 0.2227 0.1518 0.1921
SURF-GTM [67] 0.1662 0.1514 0.3343 0.2180 0.0716 0.1390
SIFT [68] 0.1724 0.0488 0.0620 0.1429 0.0637 0.0916
SURF [69] 0.1719 0.0635 0.1915 0.1507 0.0029 0.0869
Proposed 0.6661 0.7046 0.7281 0.6379 0.7557 0.5587
Drones 2024, 8, 329 18 of 28

A comparison of the RMSEs of different registration algorithms (Table 3) showed that

the proposed algorithm yielded the smallest error among the algorithms for all datasets.
VGG16 again showed no substantial advantages over other algorithms. The error of
the demons algorithm was smaller than that of the FFD algorithm. With the exception
of the water dataset, the feature-based ECC algorithm performed better than the other
feature-based algorithms.

Table 3. Comparison of root mean square error (RMSE) of different registration algorithms.

Image Datasets
Algorithms
Roofs Tower Grassland Parking Trees Water
VGG16 [3] 32.6225 22.0038 21.2094 26.4478 20.0595 20.3647
ORB [58] 39.8232 24.1672 21.2804 31.5082 18.5401 13.0100
KAZE [59] 39.7697 24.6450 17.0694 31.7053 18.5665 14.4373
AKAZE [60] 39.7677 24.5997 16.9560 31.6814 18.8502 13.6026
BRISK [61] 39.6348 24.2775 17.2148 31.4204 19.2785 14.5604
Demons [62] 24.3949 12.1627 8.2701 16.5976 11.6533 10.4814
FFD [63] 26.6226 19.9050 18.3493 26.7028 19.4471 19.8807
FAST [64] 39.6481 24.6021 17.0724 31.5625 16.9122 14.3712
ECC [65] 14.9568 18.9901 17.0026 25.2498 15.7109 20.8675
SIFT-FSC [66] 43.5612 23.7221 19.7786 33.0675 18.8597 22.7754
SURF-GTM [67] 43.8527 23.7213 20.0389 33.6706 20.7168 23.0848
SIFT [68] 39.3893 29.7595 22.6991 30.6268 17.0580 13.8813
SURF [69] 39.6625 25.3289 17.1490 31.5230 19.0615 13.7371
Proposed 15.6390 6.6643 4.4905 11.7514 5.2154 4.2349

A comparison of the MIs of different registration algorithms (Table 4) showed that the
algorithm proposed in this article has the highest mutual information value between the
registered image and reference image on the selected six datasets. The demons algorithm
based on the optical flow comb theory also has advantages in addition to the Cars dataset.
Except for the algorithm proposed in this article and the demons algorithm, the registration
results of other registration algorithms were relatively similar.

Table 4. Comparison of mutual information (MI) of different registration algorithms.

Image Datasets
Algorithms
Roofs Tower Grassland Parking Trees Water
VGG16 [3] 1.1146 1.1160 0.9283 1.2110 0.3776 1.4651
ORB [58] 0.9307 1.1555 1.2037 1.1815 0.2629 1.6509
KAZE [59] 0.9566 1.1506 1.2057 1.1816 0.4022 1.7132
AKAZE [60] 0.9561 1.1509 1.2234 1.1816 0.3923 1.7153
BRISK [61] 0.9555 1.1923 1.2131 1.1801 0.3954 1.6833
Demons [62] 2.2066 2.5214 2.3533 2.2706 1.9019 2.4924
FFD [63] 1.1860 1.1899 1.1016 1.1602 0.4298 1.5313
FAST [64] 0.9548 0.7090 1.1472 1.2116 0.4442 1.6514
ECC [65] 1.9540 1.6585 1.4887 1.4983 0.8823 1.6697
SIFT-FSC [66] 0.9963 1.2388 1.1978 1.2059 0.5592 1.6172
SURF-GTM [67] 0.9886 1.2687 1.1945 1.2030 0.4320 1.5586
SIFT [68] 0.9643 0.7810 0.7534 1.1097 0.4581 1.7272
SURF [69] 0.9581 1.1029 1.2240 1.1853 0.3679 1.7154
Proposed 2.2073 2.5468 2.4975 2.3240 2.0256 2.5974
 Drones 2024, 8, 329 19 of 28

Drones 2024, 8, x FOR PEER REVIEW 19 of 29

The registration effect of the proposed algorithm for eight image pairs is shown in
Figure 7 in the form of image overlays. The left column of the figure shows the images to be
be registered superimposed
superimposed on on the
the reference
reference images.
images. The
The large
large difference
differencebetween
betweenthe
theimages
im-
ages results in a small area of overlap; areas of difference are displayed in pseudo-color.left
results in a small area of overlap; areas of difference are displayed in pseudo-color. The
Thecolumn shows
left column the same
shows superposition
the same after after
superposition imageimage
registration. As the
registration. Asgenerated images
the generated
are now closer to the reference images, the area of overlap is large, and the pseudo-colored
images are now closer to the reference images, the area of overlap is large, and the pseudo-
area of
colored difference
area is small.
of difference is small.

Figure 7. Image registration results of the proposed algorithm: (a–l) pairs correspond to the regis-
Figure 7. Image registration results of the proposed algorithm: (a–l) pairs correspond to the registra-
tration results of image pairs 1–6, respectively, where the left image is an unregistered image overlay
tion results
display, and theofright
image pairsis1–6,
image respectively,
a registered imagewhere the display.
overlay left image is an unregistered image overlay
display, and the right image is a registered image overlay display.
In order to display the registered image and reference image, and the details after
registration, a chessboard is used to alternately display the reference image, the registered
image, and their details are displayed. The condition for selecting a detailed map is to
Drones 2024, 8, 329 20 of 28

Drones 2024, 8, x FOR PEER REVIEW In order to display the registered image and reference image, and the details after
20 of 28
registration, a chessboard is used to alternately display the reference image, the registered
image, and their details are displayed. The condition for selecting a detailed map is to
select a continuous feature on the ground. If the registration is correct, the features in the
select
detailamap
continuous feature on the
will be continuous. ground.
If the If the registration
registration is correct,
is not correct, the features
the ground in the
scenery will be
detail map will be continuous. If the
misaligned, as shown in Figure 8 below. registration is not correct, the ground scenery will be
misaligned, as shown in Figure 8 below.

Figure 8. Comparison of the details of different registration algorithms using checkerboard mosa-
Figure 8. Comparison of the details of different registration algorithms using checkerboard mosaicked
icked images (good registration results are displayed on a checkerboard without image misalign-
images (good registration results are displayed on a checkerboard without image misalignment).
ment).
The proposed algorithm performed well in all six datasets compared with the five
otherThe proposedalgorithms
registration algorithm (Figure
performed wellVGG16
8). The in all six datasets
algorithm compared
sourced fromwith the five
the literature
other registration algorithms (Figure 8). The VGG16 algorithm sourced from
did not demonstrate noticeable performance; the feature-based SIFT-FSC and SURF-GTM the literature
did not demonstrate
algorithms noticeable
yielded similar performance;
results and were notthe considered
feature-based SIFT-FSC
ideal. and SURF-GTM
The optical flow theory
algorithms yielded similar results and were not considered ideal. The optical
algorithms (FFD and demons) generally performed well, but the FFD algorithm deformed flow theory
algorithms (FFD and demons) generally performed well, but the FFD
images in the Cars and Roofs datasets to varying degrees after registration. algorithm deformed
images in the Cars and Roofs datasets to varying degrees after registration.
A CUDA architecture VS2023+CUDA+OPENCV development environment was
built to accelerate the processing algorithm and enable the comparison of runtimes be-
tween a CPU-only and a CPU+GPU processing architecture to meet the requirements of
Drones 2024, 8, 329 21 of 28

A CUDA architecture VS2023+CUDA+OPENCV development environment was built

to accelerate the processing algorithm and enable the comparison of runtimes between a
CPU-only and a CPU+GPU processing architecture to meet the requirements of real-time
remote sensing processing using the AOTF spectrometer. The images used in this test had
the same number of features (100) or the same registration accuracy. Two FFD cycles and
30 calculation cycles were used.
Even when using only two cycles, the FFD algorithm took the longest time (Table 5) due to
the time complexity of the B-spline-based FFD algorithm of m×n×16×[c+3]×[r+3]×2 ≈ O (n4).
Without CUDA, the computation load was substantial. The demons algorithm (based on
optical flow theory) achieved a large acceleration ratio. The feature-based FAST, ORB, and
SURF algorithms were relatively rapid on the CPU platform and thus did not achieve an
increase in CPU+GPU. The proposed CPU+GPU algorithm achieved an acceleration ratio
of ~30 times relative to that of the CPU alone. Furthermore, the absolute registration time of
the two spectral segments and the feature-based algorithm remained in the same order of
magnitude, and the average registration time was ~0.5 s. This indicates that the proposed
method can potentially enable real-time processing of remote sensing hyperspectral data.

Table 5. Comparison of registration time between CPU-only and CPU+GPU platforms for algorithms
suitable for GPU acceleration (unit: second).

Image Datasets
Algorithm Platform
Roofs Tower Grassland Parking Trees Water
CPU 2430.85 2442.26 2434.59 2439.99 2455.39 2447.01
FFD
CPU+GPU 133.12 132.93 132.93 132.81 132.70 133.03
CPU 19.84 21.31 21.79 20.55 21.80 21.52
Demons
CPU+GPU 0.43 0.43 0.44 0.43 0.44 0.44
CPU 1.59 2.45 7.03 3.64 2.73 3.01
FAST
CPU+GPU 0.45 0.48 0.43 0.58 0.50 0.43
CPU 0.41 0.39 0.42 0.41 0.39 0.41
ORB
CPU+GPU 0.34 0.34 0.34 0.35 0.34 0.34
CPU 2.02 2.70 3.37 2.02 4.27 1.57
SURF
CPU+GPU 0.22 0.26 0.29 0.23 0.77 0.22
CPU 15.53 16.47 16.89 15.95 16.24 16.54
Proposed
CPU+GPU 0.52 0.49 0.50 0.49 0.50 0.49

4.4. Real-Time Processing of the Remote Sensing Imaging Registration Experiment for Unmanned
Aerial Vehicles
The waypoint acquisition procedure was modified, and the registration procedure was
added to verify that the zoom lens-based AOTF spectrometer could register the collected
airborne spectral imaging data cube in real time. A UAV experiment was conducted
at the Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences
(34◦ 10′ 3.12′′ N, 108◦ 51′ 28.67′′ E) on 22 November 2023. Five flight waypoints were selected
during one flight to collect five spectral data cubes. The flight altitude was 100 m, the
trigger dwell time at a waypoint was 70 s, the wavelength switching step of the spectral
data cube was 5 nm from 400–1000 nm, and 121 bands of data acquisition and registration
were completed.
The proposed registration algorithm was adopted for image processing. Taking into
account the registration effect and computation time, through multiple experiments, it was
determined to set the number of extracted features to 100 and the registration iteration
number to 30. A micro airborne processing platform with GPU was used to verify the effect
of onboard real-time processing in terms of registration accuracy and registration time.
Drones 2024, 8, 329 22 of 28

The registration time between two spectral bands was approximately 0.5 s (Table 5);
thus, the alignment time required to complete all spectral bands was 120 × 0.5 = 60 s.
The frame rate was set to 2 Hz accordingly. Since the time between waypoints was 74 s
(waypoint stay of 70 s + flight time of 4 s [20 m at 5 m/s]), data acquisition of 121 spectral
bands of the AOTF spectrometer and the registration of the whole data cube could be
completed at each waypoint using the CPU+GPU processing mode.
The unregistered data cubes (1–5) in Figure 9 exhibited varying patterns for the three
spectral segments (R: 650 nm, G: 530 nm, and B: 480 nm) of synthetic pseudo-color, sug-
gesting that images of different spectral segments have varying fields of view for three
reasons: (1) image size deformation caused by zoom, (2) image drift caused by wave-
length switching, and (3) UAV platform jitter in remote sensing applications. Data cubes
6–10 in Figure 9 were registered using the proposed coarse-to-fine remote sensing image
registration framework based on features and optical flow theory. Parallel processing of the
CUDA architecture provided rapid registration, verifying the potential of this technique
Drones 2024, 8,for real-time
x FOR PEER REVIEW processing. 23 of 28

Figure 9. Real-time registration of remote sensing flight experiment and the waypoint data cube
Figure 9. Real-time registration
registration of remote
effect. (a) Experimental sensing for
environment flight
RGB experiment
camera shooting.and the waypoint
(b) Planned five way- data cube reg-
istration effect.points
(a) in the experiment. Cubes 1–5: without registration (different spectral images exhibit misa-
Experimental environment for RGB camera shooting. (b) Planned
lignment, with ghosting in the data cube display); cubes 6–10: with registration (the registered im-
five waypoints
in the experiment. Cubes
age displays 1–5: without
no misalignment, withregistration (different
no ghosting in the spectral images exhibit misalignment,
data cube display).

with ghosting in the data cube display); cubes 6–10: with registration (the registered image displays
no misalignment, with no ghosting in the data cube display).
Drones 2024, 8, 329 23 of 28

Drones 2024, 8, x FOR PEER REVIEW 24 of 29data

Figure 10 shows a quantitative comparison of the registration effect of waypoint
cubes and GPU registration acceleration.

Figure10.
Figure 10.Comparison
Comparison ofof the
the quantitative
quantitative effect
effect of
of waypoint
waypoint data cube
cube alignment
alignment and GPU alignment
align-
ment acceleration.
acceleration. (a–e) Quantitative
(a–e) Quantitative comparison
comparison of SSIM
of SSIM and RMSE
and RMSE between
between the unaligned
the unaligned and and
aligned
aligned data cubes adjacent to each other for the five waypoints. (f) Comparison of the average pro-
data cubes adjacent to each other for the five waypoints. (f) Comparison of the average processing
cessing time of data cube alignment using CPU and CPU+GPU for the five waypoints.
time of data cube alignment using CPU and CPU+GPU for the five waypoints.
Drones 2024, 8, 329 24 of 28

Figure 10a–e shows the SSIM and RMSE between adjacent wavelengths before and
after registration for the five selected waypoints in the form of curves. The SSIM between
adjacent bands of the data cube after registration using the proposed algorithm was consid-
erably greater than without registration. The RMSE of the data cube after registration was
smaller than without registration.
Figure 10f compares registration time between the CPU platform and the CPU+GPU
platform. Here, the registration algorithm of the five data cubes was reused offline after the
flight test was completed. It can be seen from the comparative bar chart that the average
processing time of adjacent spectral segments at five waypoints is approximately 15 s using
CPU and 0.5 s using CPU+GPU. Therefore, the CPU+GPU processing architecture achieves
approximately 30 times faster acceleration.
Finally, the hyperspectral data cube captured at the third waypoint of the experimental
data was registered, and the registration performance of the algorithm proposed in this
paper was compared with the algorithms in the reference literature. The comparison results
are shown in Table 6, which compares the performance of different algorithms in achieving
data cube registration.

Table 6. Performance comparison of different registration algorithms for data cube registration.

Flight Point 3 (Cube 3)

Algorithm
SSIM RMSE MI UIQI SAM
VGG16 [3] 0.0529 124.2055 0.3930 0.0580 0.6589
ORB [58] 0.2773 98.8223 1.3868 0.3415 0.5026
KAZE [59] 0.2146 108.3649 1.0637 0.2664 0.5507
AKAZE [60] 0.1898 110.5435 1.1167 0.2341 0.6031
BRISK [61] 0.1734 122.5900 0.6924 0.1950 0.5437
Demons [62] 0.3085 98.7734 1.4259 0.3802 0.5015
FFD [63] 0.0971 128.3170 0.3605 0.1071 0.5960
FAST [64] 0.1598 103.8626 1.0778 0.1955 0.5517
ECC [65] 0.1449 101.7744 1.2016 0.1806 0.5264
SIFT-FSC [66] 0.0581 119.1905 0.5133 0.0653 0.6890
SURF-GTM [67] 0.0376 124.9966 0.3564 0.0425 0.6706
SIFT [68] 0.1118 140.5192 0.1166 0.1154 0.6221
SURF [69] 0.0566 121.0245 0.4621 0.0632 0.6706
Proposed 0.3325 89.3380 1.5146 0.3998 0.4719

The registration results of different registration algorithms on data cube 3 are shown
in Table 6. This article proposes a new coarse-to-fine remote sensing image registration
method based on feature and optical flow theory. The registered data cubes demonstrate
strong performance in terms of the SSIM, RMSE, MI, UIQI, and SAM values. Therefore,
the method proposed in this article is highly appropriate for completing the registration of
unmanned aerial vehicle hyperspectral remote sensing image data cubes based on AOTF
spectral imager.
Currently, verification has been completed on five waypoints in a flight experiment.
However, it is necessary to complete all processing steps, including concatenation of all
waypoints, operation between spectral bands, and inversion of some application parame-
ters. The processing capacity of the platform is subject to certain limitations. The processing
platform with the selected GPU is currently the smallest processing platform available.
If the processing platform is too large, it will be limited by the maximum takeoff weight
and range of the drone. The AOTF spectral scanning imaging spectrometer has a unique
advantage in selecting spectral bands. By selecting the bands of interest in the later stage
Drones 2024, 8, 329 25 of 28

and reducing the number of bands for data processing, there is potential for completing
real-time hyperspectral remote sensing processing for a specific application.

5. Conclusions
A zoom lens-based AOTF spectral imager has the capability to address the issues
of image blurring and spectral segment drift in remote sensing. However, image size
deformation caused by zoom, image drift caused by wavelength switching, and UAV
platform jitter cause slight field-of-view differences in images of adjacent spectral segments.
Therefore, registration of each spectral segment is required to use these data.
A new coarse-to-fine remote sensing image registration framework based on feature
and optical flow theory was proposed in this study. The issues of scale transformation,
rotation, and other overall transformations were mainly addressed in the coarse registra-
tion stage, whereas issues with local image details were primarily addressed in the fine
registration stage.
The proposed method was compared with current algorithms to assess its relative
performance. We found the algorithm suitable for distributed and parallel processing in
the fine registration stage. The algorithm’s acceleration effect was verified in a CUDA
architecture (VS2023+CUDA+OPENCV) development environment.
AOTF spectral imager data were then registered in real-time during waypoint shooting,
following suitable modification of the data acquisition and processing program on the un-
manned aerial remote sensing platform. This proved that the proposed algorithm and the
CPU+GPU platform meet the requirements of real-time registration and processing on a UAV.
Certain potential issues and limitations should be considered regarding our findings.
First, it remains unknown whether increasing the number of iterations might impact the
spectral energy information. Second, although the proposed algorithm achieved a relative
acceleration effect of ~30 times, the absolute registration time of a group of images should
be further shortened to better implement this methodology.
Several future developments are planned for the developed system. First, the effects
on the spectral energy and the spectral information with an increased number of iterations
when using the coarse-to-fine registration framework of the feature + optical flow method
will be further investigated. Second, we will elucidate whether implementing OpenMP
multithreading technology for some serial programs by the CPU and using OpenGL can
improve real-time performance. Finally, research on data cube splicing will be carried out
based on data cube registration.

Author Contributions: Conceptualization: H.L., B.H., and X.H.; data curation: H.L.; formal analysis:
H.L. and Z.Z.; funding acquisition: T.Y. and X.H.; investigation: H.L. and T.Y.; methodology: H.L.
and T.Y.; project administration: T.Y.; resources: B.H. and T.Y.; software: X.L.; supervision: B.H. and
T.Y.; validation: J.L.; visualization: X.W.; writing—original draft: H.L. All authors have read and
agreed to the published version of the manuscript.
Funding: This research was funded by a Class A plan from a major strategic pilot project of the
Chinese Academy of Sciences, grant number XDA23040101; the National Natural Science Foundation
of China, grant numbers 61872286 and 62272376; the Key R&D Program of Shaanxi Province of China,
grant numbers 2020ZDLGY04-05 and S2021-YF-YBSF-0094; the Shaanxi Key Laboratory of Deep
Space Exploration Intelligent Information Technology, grant number 2021SYS-04; the National Key
R&D Program of China, grant number 2017YFC1403700; the “Light of the west” project of the Chinese
Academy of Sciences, grant number XAB2017B25; and the Photon project, grant number S24-030-III.
Data Availability Statement: No new data were created or analyzed in this study. Data sharing
is not applicable to this article. The code for the algorithm used in this study can be accessed at
https://github.com/6red/DOC.git on 16 July 2024.
Conflicts of Interest: The authors declare no conflicts of interest.
Drones 2024, 8, 329 26 of 28

References
1. Li, X.; Ai, W.; Feng, R.; Luo, S. Survey of remote sensing image registration based on deep learning. Natl. Remote Sens. Bull. 2023,
27, 267–284. [CrossRef]
2. Jiang, J.; Shi, X. A robust point-matching algorithm based on integrated spatial structure constraint for remote sensing image
registration. IEEE Geosci. Remote Sens. Lett. 2016, 13, 1716–1720. [CrossRef]
3. Yang, Z.; Dan, T.; Yang, Y. Multi-temporal remote sensing image registration using deep convolutional features. IEEE Access 2018,
6, 38544–38555. [CrossRef]
4. Mo, Y.; Kang, X.; Duan, P.; Li, S. A robust UAV hyperspectral image stitching method based on deep feature matching. IEEE
Trans. Geosci. Remote Sens. 2022, 60, 1–14. [CrossRef]
5. Ordóñez, A.; Acción, Á.; Argüello, F.; Heras, D.B. HSI-MSER: Hyperspectral image registration algorithm based on MSER and
SIFT. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 12061–12072. [CrossRef]
6. Chen, H.; Zhang, H.; Du, J.; Luo, B. Unified framework for the joint super-resolution and registration of multiangle
multi/hyperspectral remote sensing images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 13, 2369–2384. [CrossRef]
7. Lu, B.; Dao, P.D.; Liu, J.; He, Y.; Shang, J. Recent advances of hyperspectral imaging technology and applications in agriculture.
Remote Sens. 2020, 12, 2659. [CrossRef]
8. Syed Taimoor Hussain Shah, S.A.Q.; ul Rehman, A. Classification and Segmentation Models for Hyperspectral Imaging
—An Overview. In Intelligent Technologies and Applications; Springer: Cham, Switzerland, 2021; Volume 1382. [CrossRef]
9. Wang, J.; Ding, N.; Zheng, Y.; Zhao, Y.; Gao, F.; Li, J.; Wang, J.; Gao, M.; Wu, J. Overall design technology of hyperspectral imaging
system based on AOTF. In Proceedings of the International Symposium on Optoelectronic Technology and Application 2014:
Imaging Spectroscopy; and Telescopes and Large Optics, Beijing, China, 18 November 2014.
10. Nag, S.; Hewagama, T.; Georgiev, G.T.; Pasquale, B.; Aslam, S.; Gatebe, C.K. Multispectral snapshot imagers onboard small
satellite formations for multi-angular remote sensing. IEEE Sens. J. 2017, 17, 5252–5268. [CrossRef]
11. Jaiswal, B.; Singh, S.; Jain, A.; Sankarasubramanian, K.; Nandi, A. AOTF based spectro-polarimeter for observing Earth as an
Exoplanet. arXiv 2023, arXiv:2302.10712. [CrossRef]
12. Sharikova, M.O.; Balandin, I.A.; Batshev, V.I.; Kozlov, A.B. Spatial and spectral correction of an acousto-optical imaging
spectrometer. J. Opt. Technol. 2023, 90, 684–690. [CrossRef]
13. Zhang, H.; Zhao, H. Accurate design of a TeO2 noncollinear acousto-optic tunable filter with refractive index correction. Opt. Lett.
2023, 48, 3395–3398. [CrossRef]
14. Zhao, H.; Cheng, C.; Guo, Q.; Ma, R.; Yang, Y. Analysis of phase mismatch for mercurous bromide-based non-collinear AOTF
design in spectral imaging applications. Materials 2024, 17, 1703. [CrossRef]
15. Yu, K.; Guo, Q.; Li, N.; Cheng, C.; Zhao, H. Spectral calibration method for mid-infrared AOTF imagers. Infrared Laser Eng. 2023,
52, 20230291. [CrossRef]
16. Liu, H.; Hou, X.; Hu, B.; Yu, T.; Zhang, Z.; Liu, X.; Liu, J.; Wang, X.; Zhong, J.; Tan, Z. Image blurring and spectral drift in imaging
spectrometer system with an acousto-optic tunable filter and its application in UAV remote sensing. Int. J. Remote Sens. 2022, 43,
6957–6978. [CrossRef]
17. Tondewad, P.S.; Dale, M.P. Remote sensing image registration methodology: Review and discussion. Procedia Comput. Sci. 2020,
171, 2390–2399. [CrossRef]
18. Li, L.; Han, L.; Ding, M.; Cao, H.; Hu, H. A deep learning semantic template matching framework for remote sensing image
registration. ISPRS J. Photogramm. Remote Sens. 2021, 181, 205–217. [CrossRef]
19. Ruiqi, L.; Bowu, Y.; Dou, Q.; Yi, L.; Baorui, D.; Shuang, W.; Huarong, J.; Biao, H.; Licheng, J. Deep Global Feature-Based Template
Matching for Fast Multi-Modal Image Registration. In Proceedings of the 2021 IEEE International Geoscience and Remote Sensing
Symposium IGARSS, Brussels, Belgium, 11–16 July 2021; pp. 5457–5460.
20. Ye, Y.; Wang, M.; Hao, S.; Zhu, Q. A novel keypoint detector combining corners and blobs for remote sensing image registration.
IEEE Geosci. Remote Sens. Lett. 2021, 18, 451–455. [CrossRef]
21. Wu, G.-L.; Chang, H.-H. An accurate feature point matching algorithm for automatic remote sensing image registration.
In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney,
Australia, 29 November–1 December 2017. [CrossRef]
22. Zhang, T.; Zhao, R.; Chen, Z. Application of migration image registration algorithm based on improved SURF in remote sensing
image mosaic. IEEE Access 2020, 8, 163637–163645. [CrossRef]
23. Chen, S.; Zhong, S.; Xue, B.; Li, X.; Zhao, L.; Chang, C.-I. Iterative scale-invariant feature transform for remote sensing image
registration. IEEE Trans. Geosci. Remote Sens. 2021, 59, 3244–3265. [CrossRef]
24. Jhan, J.-P.; Rau, J.-Y. A generalized tool for accurate and efficient image registration of UAV multi-lens multispectral cameras by
N-SURF matching. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 6353–6362. [CrossRef]
25. Wu, Y.; Xiao, Z.; Liu, S.; Miao, Q.; Ma, W.; Gong, M.; Xie, F.; Zhang, Y. A two-step method for remote sensing images registration
based on local and global constraints. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 5194–5206. [CrossRef]
26. Ordóñez, Á.; Argüello, F.; Heras, D.B. GPU accelerated FFT-based registration of hyperspectral scenes. IEEE J. Sel. Top. Appl.
Earth Obs. Remote Sens. 2017, 10, 4869–4878. [CrossRef]
27. Liu, Y.; Cao, H.; Zhao, Y.; He, Q.; Yang, Y.; Wang, L.; Lin, G.; Zhou, J. A remote sensing image registration algorithm based on
multiple constraints and a variational Bayesian framework. Remote Sens. Lett. 2021, 12, 296–305. [CrossRef]
Drones 2024, 8, 329 27 of 28

28. Zhou, C.; Zhang, G.; Yang, Z.; Zhou, J. A novel image registration algorithm using wavelet transform and matrix-multiply
discrete Fourier transform. IEEE Geosci. Remote Sens. Lett. 2022, 19, 8002605. [CrossRef]
29. Lee, W.; Sim, D.; Oh, S.-J. A CNN-based high-accuracy registration for remote sensing images. Remote Sens. 2021, 13, 1482.
[CrossRef]
30. Zeng, Y.; Ning, Z.; Liu, P.; Luo, P.; Zhang, Y.; He, G. A mosaic method for multi-temporal data registration by using convolutional
neural networks for forestry remote sensing applications. Computing 2020, 102, 795–811. [CrossRef]
31. Ye, Y.; Tang, T.; Zhu, B.; Yang, C.; Li, B.; Hao, S. A multiscale framework with unsupervised learning for remote sensing image
registration. IEEE Trans. Geosci. Remote Sens. 2022, 60, 5622215. [CrossRef]
32. Ying, C.; Lei, C.; Qi, Z.; Wei, W.; Jiahao, W. Improved remote sensing image registration of residual block densely connected
network based on reinforcement learning. In Proceedings of the 6th International Conference on Intelligent Informatics and
Biomedical Sciences (ICIIBMS), Oita, Japan, 25–27 November 2021. [CrossRef]
33. Fernández-Fabeiro, J.; Gonzalez-Escribano, A.; Llanos, D.R. Distributed programming of a hyperspectral image registration
algorithm for heterogeneous GPU clusters. J. Parallel Distrib. Comput. 2021, 151, 86–93. [CrossRef]
34. Liu, Y.; Zhou, Y.; Zhou, Y.; Ma, L.; Wang, B.; Zhang, F. Accelerating SAR image registration using swarm-intelligent GPU
parallelization. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 13, 5694–5703. [CrossRef]
35. Ordóñez, Á.; Heras, D.B.; Argüello, F. Multi-GPU registration of high-resolution multispectral images using HSI-KAZE in a
cluster system. In Proceedings of the 2022 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Kuala
Lumpur, Malaysia, 17–22 July 2022. [CrossRef]
36. Zhang, X.; Zhao, X. High-precision registration algorithm and parallel design method for high-resolution optical remote sensing
images. Int. J. Pattern Recognit. Artif. Intell. 2021, 35, 2154020. [CrossRef]
37. Jiang, Q.; Qiu, Y.; Wen, Y.; Wang, H.; Xu, W. Design of data acquisition system for AOTF polarization spectral imaging instrument.
Infrared Laser Eng. 2012, 41, 218–222.
38. Liu, H.; Yu, T.; Hu, B.; Hou, X.; Zhang, Z.; Liu, X.; Liu, J.; Wang, X.; Zhong, J.; Tan, Z.; et al. UAV-borne hyperspectral imaging
remote sensing system based on acousto-optic tunable filter for water quality monitoring. Remote Sens. 2021, 13, 4069. [CrossRef]
39. Jiachun, W.; Dapeng, Z.; Xianghua, D.; Qichao, W.; Zhigang, L. Design and experiment of hyper-spectral polarization imaging
system based on AOTF. Infrared Laser Eng. 2017, 46, 1136002. [CrossRef]
40. Ryu, S.Y.; You, J.-W.; Kwak, Y.; Kim, S. Design of a prism to compensate the image-shifting error of the acousto-optic tunable filter.
Opt. Express 2008, 16, 17138–17147. [CrossRef]
41. Ren, Y.; Cai, H.; Tan, J.; Tan, Y.; Zhang, X.; Zheng, F.; Ma, W. Imaging drift of acousto-optic modulator spectral camera. Chin. Opt.
2013, 6, 179–186. [CrossRef]
42. Suhre, D.R.; Theodore, J.G. White-light imaging by use of a multiple passband acousto-optic tunable filter. Appl. Opt. 1996, 35,
4494–4501. [CrossRef]
43. Feng, R.; Du, Q.; Luo, H.; Shen, H.; Li, X.; Liu, B. A registration algorithm based on optical flow modification for multi-temporal
remote sensing images covering the complex-terrain region. Nation Remote Sens. Bull. 2021, 25, 630–640. [CrossRef]
44. Horn, B.K.P.; Schunck, B.G. Determining optical flow. Artif. Intell. 1981, 17, 185–203. [CrossRef]
45. Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011
International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011. [CrossRef]
46. Tang, M.; Liang, K.; Qiu, J. Small insulator target detection based on multi-feature fusion. IET Image Proc. 2023, 17, 1520–1533.
[CrossRef]
47. Lei, S.; Zhu, F. UAV remote sensing image registration algorithm based on ORB and improved RANSAC. J. Nat. Sci. Heilongjiang
Univ. 2020, 37, 8. [CrossRef]
48. Zhang, G.; Guo, L.; Xiong, B.; Chu, J. Active demons algorithm based on multi-resolution and adaptive fractional differential.
J. Comput. Res. Dev. 2018, 55, 2753–2763. [CrossRef]
49. Chen, Q.; Liu, J.; Tang, Z.-H.; Li, J.-Q.; Wu, M. Detection and extraction of image edge curves and detailed features using fractional
differentiation. Acta Electron. Sin. 2013, 41, 1873–1880. [CrossRef]
50. Thirion, J.P. Image matching as a diffusion process: An analogy with Maxwell’s demons. Med. Image Anal. 1998, 2, 243–260.
[CrossRef] [PubMed]
51. Wang, H.; Dong, L.; O’Daniel, J.; Mohan, R.; Garden, A.S.; Ang, K.K.; Kuban, D.A.; Bonnen, M.; Chang, J.Y.; Cheung, R. Validation
of an accelerated ‘demons’ algorithm for deformable image registration in radiation therapy. Phys. Med. Biol. 2005, 50, 2887.
[CrossRef] [PubMed]
52. Santos-Ribeiro, A.; Nutt, D.J.; McGonigle, J. Inertial demons: A momentum-based diffeomorphic registration framework. In
Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016; Ourselin, S., Joskowicz, L., Sabuncu, M., Unal, G.,
Wells, W., Eds.; Springer Nature: Cham, Switzerland, 2016; pp. 37–45.
53. Peng, M.; Li, G.; Zhou, X.; Ma, C.; Zhang, L.; Zhang, X.; Shang, K. A registration-error-resistant swath reconstruction method of
ZY1-02D satellite hyperspectral data using SRE-ResNet. Remote Sens. 2022, 14, 5890. [CrossRef]
54. Gu, Y.; Wang, C.; Li, X. An intensity-independent stereo registration method of push-broom hyperspectral scanner and LiDAR on
UAV platforms. IEEE Trans. Geosci. Remote Sens. 2022, 60, 5540014. [CrossRef]
55. Bin, J.; Zhang, H.; Bahrami, Z.; Zhang, R.; Liu, H.; Blasch, E.; Liu, Z. The registration of visible and thermal images through
multi-objective optimization. Inf. Fusion 2023, 95, 186–198. [CrossRef]
Drones 2024, 8, 329 28 of 28

56. Wang, S.; Fan, F. Thangka. Hyperspectral image super-resolution based on a spatial-spectral integration network. Remote Sens.
2023, 15, 3603. [CrossRef]
57. Guo, A.; Dian, R.; Li, S. A deep framework for hyperspectral image fusion between different satellites. IEEE Trans. Pattern Anal.
Mach. Intell. 2023, 45, 7939–7954. [CrossRef]
58. Fan, Y.G.; Chai, J.; Xu, M.; Wang, B.; Hou, Q. Improved fast Image registration algorithm based on ORB and RANSAC fusion.
Opt. Precis. Eng. 2019, 27, 702–717. [CrossRef]
59. Ordóñez, Á.; Argüello, F.; Heras, D.B.; Demir, B. GPU-accelerated registration of hyperspectral images using KAZE features.
J. Supercomput. 2020, 76, 9478–9492. [CrossRef]
60. Yan, Q.; Li, Q.; Zhang, T. Research on UAV Image Mosaic Based on Improved AKAZE Feature and VFC Algorithm, In Proceedings
of the 2021 6th International Conference on Multimedia and Image Processing, Zhuhai, China, 8–10 January 2021.
61. He, L.; Liu, J.; Li, G. Fast image registration approach based on improved BRISK. Infrared Laser Eng. 2014, 43, 2722–2727.
62. Zhang, D.; Huang, H.; Shang, Z. Nonrigid image registration algorithm based on mutual information active demons. Laser
Optoelectron. Prog. 2020, 57, 161009. [CrossRef]
63. Sun, W.; Niessen, W.J.; Klein, S. Randomly perturbed B-splines for nonrigid image registration. IEEE Trans. Pattern Anal. Mach.
Intell. 2017, 39, 1401–1413. [CrossRef] [PubMed]
64. Zhang, Y.; Liu, Y. Study of image registration system based on FAST feature. Comput. Eng. Appl. 2016, 52, 167–170. [CrossRef]
65. Hwooi, S.K.W.; Sabri, A.Q.M. Enhanced correlation coefficient as a refinement of image registration. In Proceedings of the IEEE
International Conference on Signal and Image Processing Applications (ICSIPA), Kuching, Malaysia, 12–14 September 2017.
[CrossRef]
66. Wu, Y.; Ma, W.; Gong, M.; Su, L.; Jiao, L. A novel point-matching algorithm based on fast sample consensus for image registration.
IEEE Geosci. Remote Sens. Lett. 2015, 12, 43–47. [CrossRef]
67. Sedaghat, A.; Mohammadi, N. High-resolution image registration based on improved SURF detector and localized GTM. Int. J.
Remote Sens. 2019, 40, 2576–2601. [CrossRef]
68. Chang, H.-H.; Wu, G.-L.; Chiang, M.-H. Remote sensing image registration based on modified SIFT and feature slope grouping.
IEEE Geosci. Remote Sens. Lett. 2019, 16, 1363–1367. [CrossRef]
69. Ordóñez, Á.; Heras, D.B.; Argüello, F. SURF-based registration for hyperspectral images. In Proceedings of the IGARSS 2019-2019
IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan, 28 July–2 August 2019. [CrossRef]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.