4420 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO.
8, AUGUST 2018
3-D Deep Learning Approach for Remote
Sensing Image Classification
Amina Ben Hamida, Alexandre Benoit , Patrick Lambert, and Chokri Ben Amar, Senior Member, IEEE
Abstract— Recently, a variety of approaches have been
enriching the field of remote sensing (RS) image processing and
analysis. Unfortunately, existing methods remain limited to the
rich spatiospectral content of today’s large data sets. It would
seem intriguing to resort to deep learning (DL)-based approaches
at this stage with regard to their ability to offer accurate semantic
interpretation of the data. However, the specificity introduced by
the coexistence of spectral and spatial content in the RS data
sets widens the scope of the challenges presented to adapt DL
methods to these contexts. Therefore, the aim of this paper is
first to explore the performance of DL architectures for the RS
hyperspectral data set classification and second to introduce a
new 3-D DL approach that enables a joint spectral and spatial
information process. A set of 3-D schemes is proposed and eval-
uated. Experimental results based on well-known hyperspectral
data sets demonstrate that the proposed method is able to achieve
a better classification rate than state-of-the-art methods with
lower computational costs.
Index Terms— Classification, deep learning (DL), hyperspec-
tral, pixel-based, remote sensing (RS).
I. I NTRODUCTION
T ODAY, remote sensing (RS) plays a fundamental role
in providing a rich source of information for a variety
of applications. It is now a major means for advanced and
numerous purposes, such as the long-term climate studies,
population evolution analysis, and sometimes even the pre-
cocious prevention of calamities. In fact, RS has opened
doors for not only a deeper understanding of the Earth
itself but also for delicate investigations into its population
and environmental behaviors. In reality, such advances are
mainly boosted by the collaboration of serious industrial and
academic research. The impressive breakthroughs witnessed
on a technical level, acquisition tools, and new open data
Manuscript received May 7, 2017; revised October 16, 2017 and
February 2, 2018; accepted March 4, 2018. Date of publication April 20,
2018; date of current version July 20, 2018. This work was supported
by the Research and Education France Ministry. (Corresponding author:
Alexandre Benoit.)
A. Ben Hamida is with the Informatics, Systems, Information and
Knowledge Processing Laboratory, Université Savoie Mont Blanc, F-74000
Annecy, France, and also with Research Groups in Intelligent Machines,
Ecole Nationale d’Ingénieurs de Sfax, Sfax 3038, Tunisia (e-mail:
amina.ben-hamida@univ-smb.fr).
A. Benoit and P. Lambert are with the Informatics, Systems, Informa-
tion and Knowledge Processing Laboratory, Université Savoie Mont Blanc,
F-74000 Annecy, France (e-mail: alexandre.benoit@univ-smb.fr; patrick.
lambert@univ-smb.fr).
C. Ben Amar is with Research Groups in Intelligent Machines,
Ecole Nationale d’Ingénieurs de Sfax, Sfax 3038, Tunisia (e-mail:
chokri.benamar@ieee.org).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TGRS.2018.2818945
0196-2892 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
models have dramatically impacted RS data products. This fast
progress has mainly led to the creation of not only over-
whelming quantities of data sets but also of very rich spatial
and spectral content. Faced with harder and more complex
images, the renovation of the classically used approaches has
been needed. In fact, the basic philosophy behind the first
RS classification methods relied on the so-called “shallow
structures.” Different techniques have been used, including
artificial neural networks [1], classification trees [2], and
support vector machine [3]. The bag of visual words was
introduced as a baseline for recent RS image classification [4],
since it enables a better understanding of the data content and
the interpixel dependencies. Although these tools were—until
recently—highly ranked in the classification field, they are
now incapable of coping with the abundance of today’s image
content. As a response to this lack of efficient methods, serious
efforts have been made, and several approaches have been set
up, such as the graphic-based one detailed in [5] or some
handcrafted feature tools that can effectively describe the
spatial and spectral content of the images [6]. Despite high
performance, these approaches remain limited because of their
lack of a generic aspect, adaptation to different contexts, and
an impassable need for expert knowledge in the parameter
setup phase. Therefore, there is an urgent need for more
convenient analysis methods and approaches that allow hier-
archical comprehension of the data and thorough learning
of its content. Certainly, when dealing with tricky learning
tasks, it is currently almost impossible not to acknowledge
the achievements of deep learning (DL). In fact, since the
impressive comeback of neural networks in 2006, the machine
learning community has become very popular, mainly thanks
to the emergence of DL-based approaches and their remark-
able performances. Early deep methods started in simpler ways
with digit classification to recently become a winning tool for
complex image classification tasks in the 2012 Large Scale
Visual Recognition Challenge (ILSVRC2012). Over the past
several years, DL has been growing as one of the most efficient
techniques for a wide range of applications and fields and has
managed to overcome different challenges when dealing with
Big Data issues. However, it likewise brings more precision
and accuracy into smaller scale applications that mainly focus
on today’s wealth of spatial and spectral content. Therefore,
one of the main focuses is now pointed at the ability of DL
approaches to solve RS data classification problems. Currently,
an important share of RS research is devoted to investigating
techniques that enable effective interpretation, analysis, and
extraction of relevant knowledge.
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION
In this paper, a general overview of the DL evolution
phases and the currently existing methods is presented. The
main challenges that are disrupting its progress are also
presented along with a special focus on DL techniques used
for RS image classification. Finally, a new deep network
structure is proposed and examined for an RS case study
on a small well-known hyperspectral data set which then
draws the basic guidelines of how to deal with the RS
data blast. Models and trained weights are then made avail-
able at https://github.com/AminaBh/3D_deepLearning_for_
hyperspectral_images.
II. D EEP L EARNING OVER THE Y EARS
Over the years, a lot of research has been dedicated to
machine learning and artificial intelligence. Obviously, talking
about neural networks is not a new subject as the field has
been around since the 1950s [7]. Starting from the mid-
1990s, machine learning has gone through several transition
periods paving the way for the impressive comeback of neural
networks. However, the proposed techniques relied so much
on human involvement in the process for system tuning
and data annotation along with high computational power
requirements that neural nets where surpassed by other less
constrained methods, such as support vector machine-based
approaches. Consequently, without sufficient resources, neural
network-based approaches went through a “winter period”
where a few advances appeared in state-of-the-art methods.
Thanks to Geoffrey E. Hinton and his team, shallow structures
with few layers have been replaced by deeper architectures
with many more layers with the introduction of deep belief
networks (DBNs) in 2006 [12]. This has revolutionized both
the academic and industrial worlds. This progress has been
greatly assisted by technical evolution on different levels: the
world has witnessed the birth of richer annotated databases
along with powerful computational tools, such as GPUs. Since
then, much literature has been focused on DL methods. The
main tool behind the success of DL is the introduction of more
processing layers which induces more representational levels
and therefore ensures progressive dissociation of the concepts
contained in the data. Consequently, not only does DL enhance
the learning process, these approaches have also managed to
overcome the classical ones by getting rid of the previously
used engineered features. Over the years, a rich repository has
been established to encompass numerous deep architectures.
These methods can be categorized into three main classes
regarding their architectures, aims, and techniques that are
detailed in Sections II-A–II-D.
A. Deep Networks For Unsupervised Learning:
Generative Approaches
The absence of target labels or knowledge information
during the learning phase limits the application to a feature
identification process. Most of today’s available data are unla-
beled data sets which raises the question: to what extent can
we learn meaningful representations? In this case, the lower
level abstractions are more tightly related to simple features,
and higher level abstractions are dedicated to high semantic
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
4421
level concepts and objects. Different approaches belong to
this specific category namely autoencoders [8]. Basically, they
can perform image hierarchical learning through two types
of modules: a first set of data encoding layers followed by
a decoding set of layers that tries to reconstruct back the
input. Another proposed architecture for unsupervised learning
is the restricted Boltzmann machine (RBMs) [9]. As first
introduced, this approach relied on connected neuronlike units
that make stochastic decisions about whether to be ON or OFF.
Therefore, it can be seen as a neural network model [10].
Although the main concept behind it seems very tempting for
image processing applications, its execution was difficult and
time-consuming. As a remedy for this problem, restrictions
were added to the network topology forbidding connections
between the variables within the same layer, and leading to
the one layer RBMs [11] as well as its deeper version the deep
Boltzmann machine. DBNs where then proposed in [12] and
rely on a clever combination of RBMs along with a classifier.
B. Deep Networks for Supervised Learning:
Discriminative Approaches
Target labels are expected to help learning and data clas-
sification, whether they are present in a direct or indirect
form. This category of approaches is intended to accom-
plish pattern classification tasks, often by characterizing the
posterior distributions of classes presented in the data. They
are also called discriminating deep networks. Deep stacking
networks (DSNs), convolutional neural networks (CNNs) [13],
and recurrent neural networks (RNNs) [14] are the main archi-
tectures used for supervised tasks. As first introduced, the main
idea behind the DSN design derives from the concept of
arranging series of simple classification modules, as proposed
and explored in [15] and [16]. At an early stage, classifiers
are set to then be stacked on top of each other ensuring
the learning of complex concepts. The DSN architecture was
originally presented in [17]. Although different varieties of
DSNs were created in order to diminish the computational
cost of the process, these architectures remain very expensive,
which challenges their users and deflects interest toward other
sets of approaches. One of the main alternatives is the CNN,
as first presented in [13]. These convolutional (Conv) layers
can be viewed as a series of trainable filters that slide all
over the input’s dimensions (width, height, and even depth).
Since these layers share many weights but significantly less
than classical fully connected (FC) neuron-based networks,
the stacking of the layers on top of each other allows a gradual
increase in the data representation semantic level without
exploding the computational cost. Usually, a series of FC lay-
ers followed by a classifier is inserted at the end of each phase.
This basically finalizes the representation learning process by
modeling the target concepts from a composition of their
already high semantic level input features. The CNNs benefit
from the bonus of introducing the subsampling property which
guarantees a decrease in feature map resolutions, thus reducing
the computing costs while enforcing robustness against trans-
lations. In fact, discriminative approaches generally follow
a similar philosophy that aims at first representing input
images as high-resolution low-level features representations,
 4422 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018
starting by oriented contours, that are gradually subsampled
and composed into more complex and more numerous patterns
while going through the network architecture. Depending on
the task, a final layer may be used to format output to the
required type. Luckily, all these basic tools have allowed the
creation of a rich benchmark of robust designs for different
CNN networks. These architectures have been found highly
effective and been commonly used in computer vision and
image recognition [18], [19].
C. Hybrid Deep Networks
The term hybrid for this third category refers to the deep
architecture that either comprises or makes use of both genera-
tive and discriminative model components. These architectures
often operate in a multistage learning process, where the
generators are trained using a specific strategy. The recent
generative adversarial networks [20] highlight the interest of
simultaneously training the two components where a generator
continuously improves and tries to fool a discriminator which
continuously tries to differentiate real and fake generated data.
Recently, it was shown in [21] that deep hybrid architec-
tures, or multilevel models that integrate discriminative and
generative learning objectives, offer a strong viable alternative
to multistage learners.
D. Evolution of Different CNNs
As one of the most successful deep architectures, CNNs
have progressively found their way into today’s applications.
By less than 20 years, we moved from the five-layered
leNet5 architecture [13] dedicated to digits recognition to
the more advanced residual network [22] variants that can
include hundreds of layers that are now able to recognize
thousands of visual concepts. Actually, the main architecture
that effectively familiarized the world with CNNs is the eight-
layer AlexNet. As detailed in [23], this technique has the
advantage of being deeper and more expressive than the other
approaches by stacking a series of convolution layers. The
AlexNet architecture was a clear winner at ILSVRC2012,
and since then, this challenge has been systematically won
by CNNs every year, always improving performance levels
by improving depth, width, and processing path strategies
while reducing the number of parameters [24], [25]. Then,
as the world has evolved toward more sequence-dependent
applications (video, text, and so on), RNNs whose output
depends on the input and the previous iteration states complete
CNN architectures. The long short-term memory cells [26]
are a flexible example of such family that enables long and
short range interactions by the use of trainable state gates.
Such tools enable impressive results on various application
use cases, such as image captioning [27] and video semantic
segmentation [28].
III. D EEP L EARNING FOR R EMOTE S ENSING
I MAGE C LASSIFICATION
The content of satellite images with high resolution in
both space and frequencies is remarkably complex, providing
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
details of objects, such as houses, trees, or even cars on
the parking lots. In order to be able to fully describe the
content of these images, a deep hierarchical representation
is highly recommended. The lowest level is represented by
primitive vectors describing the color, texture, and shape. At a
higher level, simple objects, such as roads, forests, or lakes,
are described by unique combinations of primitive vec-
tors. However, individually considered, these objects cannot
describe the scene or grasp the overall meaning of the image as
they give different interpretations according to their neighbor-
ing, as well as their spatial and spectral positions. Therefore,
in order to extract meaningful information from the image,
the spatial interactions to the next level of the representation
hierarchy must be taken into account. These models lead to
the discovery of semantic rules that define the final level of
abstraction: high-level semantic classes, such as residential
districts, commercial areas, or ports are of high semantic level.
This extremely rich repository of information has created the
need for DL as a key solution.
A. State-of-the-Art DL for RS
It is often possible to resort to ideas that come from
the multimedia field to proceed with RS data sets. Current
multimedia-inspired DL models have managed to provide a
baseline for the use of DL in RS key applications. Recent
challenges focus on semantic segmentation and ensure high
accuracy rates in the case of a traditional three spectral band
task [29]. However, the creation of more complex RS data
has catalyzed more research into a better understanding of data
with rich spectral content, such as hyperspectral and multispec-
tral images. As detailed in [30], first trials only relied on the
spectral information presented in the data itself. For instance,
the approaches as presented in [23], [31], and [32] suffered a
lack of spatial information and therefore probably disregarded
a very important element of the image content. The same
problem was seen when only processing the spatial content of
the data as detailed in [33]–[36]. Therefore, more complex
yet effective solutions have been presented in different forms
and models to take into account both spectral and spatial com-
ponents guaranteeing a maximum profit from the insights and
information restrained in the images. Early solutions resorted
to a marginal processing of spectral and spatial information
as presented in [37]. In this case, the spectral information
is processed apart from the spatial component which is
extracted later to be joined for feature extraction based on deep
architectures, such as stacked autoencoders (SAEs). Neural
network classifiers are then implemented in the final layer.
Autoencoders are also the basic concept behind the model
introduced in [38], presenting a spatiospectral framework that
merges spectral information from adjacent pixels to add spatial
information to the processed pixel. Hidden layers are then
inserted in order to learn the spectral features, and a supervised
learning is ensured by an output softmax layer. Incorporating
both spatial and spectral information improves the classifica-
tion performances as mentioned in all the above-mentioned
methods. However, the use of the SAE or more generally using
large layers of FC neurons explodes the number of parameters
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION
to train and demand a large number of training samples. In the
case of data sets with a few annotations, training systems
with a large number of parameters are not tractable or lead
to overfitted and suboptimal solutions. In the specific
case of hyperspectral image analysis, DL methods recently
opened wide doors into taking huge leaps. More specifically,
3-D (i.e., one spectral dimension plus two spatial ones) CNNs
were introduced as a solution to obtain an accurate and
computationally efficient architecture. As presented in [39],
a 3-D-like approach starts the process with a randomized
PCA applied on the spectral dimension of the image. There-
fore, the inputs for the first convolutional (Conv) layer (C1)
are 3-D patches of size s × s × Cr where s is the width
and height of the spatial patch while Cr is the number of
the retrained principle spectral components. A second Conv
layer is applied to the output of C1. Finally, a C2 element
vector is produced and fed as input to a multilayer percep-
tron classifier. However, with this approach, each retrained
spectral dimension is processed independently with stan-
dard 2-D convolutional filters. Another approach is developed
in [40], where a 3-D convolution is literally deployed by the
first layer followed by two 1-D Convs and ending with two FC
layers. This approach is close to the one presented in [41] but
adds spatial information. However, it still introduces a large
amount of parameters (60 000) that need to be trained with
only 1800 samples. The main concern in most of the above-
presented cases is then how to deal with the high number of
parameters to be trained with a few samples while improving
image analysis. Recently, architectures significantly reduce
this ratio such as [42] that maximizes the parameter reuse.
B. CNNs for RS Image Classification
In view of the wealth of recent RS image content, it is
crucial to find deep architectures that maintain the balance
between efficiently processing huge amounts of data and not
exploding the computational costs while also providing high
accuracy. The employment of deep classification techniques
for the RS field would seem to be a promising path of
applications. However, further investigation reveals different
challenges that must be overcome to reach accurate low-cost
data interpretation. The main challenge is to efficiently adapt
deep architecture to take into account not only the spatial
dimension of hyperspectral images but also their rich spectral
content. Out of all the current DL networks, CNNs are one
of the best available tools for machine vision. These models
have helped DL become one of today’s hottest topics. Thanks
to the variety of layers that one CNN can encompass, these
networks provide an efficient tool for data comprehension
and representation. The fact that they can be fitted to differ-
ent applications and are relatively low-cost architectures for
tremendous tasks makes them one of the most extensively
used DL approaches. In this paper, use cases, CNNs, are
a primordial choice that can be fitted to RS classification
tasks. Basically, these architectures must perform well without
overfitting or undertraining the system. However, the evolution
toward effective CNNs in such cases has been diminished by
the following challenges.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
4423
1) High-Dimensional Data: When dealing with high-
dimensional data, DL approaches become computation-
ally expensive. These high costs are mainly due to the
slow learning process that is needed to learn the data
abstractions and establish an effective representation
from low levels to the highest semantic interpreta-
tions. In fact, a high-dimensional data source contributes
greatly to the volume of the raw data, as well as
complicating learning from this data. The most effective
solution so far proposed is the use of CNNs, since the
neurons in the hidden layer units do not need to be
connected to all of the nodes in the previous layer, but to
a more localized receptive field. Moreover, the resolution
of the image data is also reduced when moving toward
higher layers in the network as depicted in [43] and [44]
thanks to pooling layers. However, this problem remains
very challenging and leads to more complex and harder
issues.
2) Large Heavy Models: DL models have so far accom-
plished remarkable results relying on deep and wide
models. Therefore, large numbers of parameters are
required to learn complicated features and representa-
tions from the data itself as explained in [45]. These
heavy models are hard to train, costly to fit, and com-
plicated to establish. Moreover, such heavy models are
greedy in terms of labeled data. This requirement is hard
to establish, since the field is suffering from a serious
lack of rich hyperspectral and multispectral annotated
data.
3) Architecture Optimization: The key point in favor of
using DL today is its ability to cope with a wealth
of applications. However, this results in hardening and
complicating the tasks of establishing deep models
that are inexpensive and effective in processing data.
Obviously, regarding the variety of fields that today’s
community is involved in, one can notice an urgent
need to optimize deep architectures in terms of com-
putational costs, accurate results, and required training
information. Recently various strategies have been pro-
posed to optimize pretrained architectures in terms of
inference speed and memory footprint reduction by layer
factorization and weights pruning [46]–[48]. However,
such posttraining optimization strategies cannot allow to
disregard architecture design optimization prior training,
since large architectures cannot be reliably trained on
small data sets. Therefore, a lot of wise choices must
be taken at early stages: selecting one specific type of
deep network, choosing whether to fine-tune a pretrained
architecture, or starting from scratch.
The CNN structure introduces an accurate solution for most
of the previously presented challenges. However, more recent
studies that tend to enhance the use of deeper architectures
have also been established. Residual modules as presented
in residual networks [22] have made it possible to design
extremely deep networks with more than a thousand layers.
More recently, dense networks were presented in [42] that
emphasize the interest of denser connections across layers. The
main purpose of such methods is generally to address complex
 4424
Fig. 1. Overview of the proposed 3-D deep architecture.
problems with high variability targets at multiple scales.
Recent works tried to adapt these concepts to hyperspectral
data use cases as detailed in [49]. The aim of this paper is to
prove the efficiency of noncomplex architectures.
IV. 3-D D EEP L EARNING A RCHITECTURE FOR
RS I MAGE S EMANTIC S EGMENTATION
As a solution for the challenges presented in the previous
sections, we introduce a new 3-D-based architecture that is
dedicated to hyperspectral images and tackles most of the DL
for RS aspects of difficulty.
A. General Overview of the Architecture
A joint spatiospectral model is needed to examine both
spectral and spatial information in hyperspectral data. The
advantage of such a framework is that both the compo-
nents are merged and joint in a nonseparable way from the
early stages of the process. This solution makes maximum
use of the information presented in the data and radically
lowers costs. This paper proposes to use a new 3-D CNN
architecture that, unlike the previously mentioned approaches,
simultaneously processes the spatial and spectral components
with real 3-D convolutions giving better investments of the
few samples available with fewer trainable parameters. This
proposal decomposes the problem as the processing of a series
of volumetric representations of the image. Therefore, each
pixel is associated with an n × n spatial neighborhood and
a number of f spectral bands. As a result, each pixel is
treated as an n × n × f volume. The main concept behind
this architecture is to combine the traditional CNN network
with a twist of applying 3-D convolution operations instead of
using 1-D convolution operators that only inspect the spectral
content of the data. An overview of the 3-D architecture is
presented in Fig. 1.
Different blocks of CNN layers are stacked on top of
each other in order to ensure deep efficient representations
of the image. First, a 3-D convolution-based set of layers
is introduced in order to cope with the 3-D input voxels.
Each and every one of these layers encompasses a number of
volumetric kernels that simultaneously execute convolutions
on the width, height, and depth axis of the input. Such
3-D convolutions stack is followed by a set of N1D 1 × 1
convolution (1-D) layers that discards the spatial neighbor-
hood and a series of NFC Fully Connected layers. Basically,
the proposed architecture considers 3-D voxels as input data
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018
Fig. 2. Example illustrating the evolution of feature shapes (SizeOut) of
each layer [output size is obtained according to (1)].
and first generates 3-D feature maps that are gradually reduced
into 1-D feature vectors all along the layers. This procedure
is ensured by the choice of specific configurations of the
convolution filter strides and paddings following (1), where
the stride is the distance between two consecutive positions
of the kernel expressed with a number of pixels. The padding
is used to manage boundary effects and is basically employed
as a number of zeros concatenated at the beginning and at the
end of an axis. Using padding enables to make the convolution
output the same size as the input while no padding reduces
the output data shape
 
SizeIn − Kernel Size + 2 × pad
SizeOut = + 1. (1)
stride
As previously said, the input is fed to the network as a
3-D volume (voxel) of size n × n × f . The first phase consists
of using a series of N3D 3-D convolutional layers. Each
layer i is characterized by a number of ki filters. The kernels
of the filters are of size (mi × mi × f li ) where mi<=n
and fli<=f. In this case, convolution layers are deployed
for two purposes. First, they are introduced as conventional
spatiospectral convolution layers with a stride equal to 1. Then,
they play the role of pooling layers thanks to the choice of
larger strides to downsample the data as suggested in [50].
The duality between Conv and Pool layers in a sequential
way progressively learns and reduces the data components’
dimensions. The f li > mi rule along with the removal of
padding on the spatial dimension for some layers make the
transition toward the creation of a first 1-D output vector.
This output is then fed to a series of N1D 1-D Conv layers that
each encompasses pi filters. In the end, the network introduces
a set of NFC FC layers that ends with a Softmax classifier
where the softmax activation of the i th output unit is detailed
in (2). Since the final layer’s size is chosen to be equal to the
input number of targeted classes (nclass), this FC guarantees
a probabilistic representation for the different classes
e xi
P(x i ) = nclass (2)
c e xc
where x i denotes the 1-D output vector of the final Conv layer.
An example of a 3-D Conv layer network is illustrated
in Fig. 2 where, f = 103, n = 3, f li = 3, and mli can
take 3 or 1 as values. Padding is used on the spectral scale
only. The strides are alternated between one and two in order
to create a pooling effect after every convolutional operation.
So, as a summary and as detailed in Algorithm 1, the pro-
posed process starts by dividing the hyperspectral image into
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION
Algorithm 1: Classification of Hyperspectral Images at
the Pixel Level. Each Pixel is Processed with respect to
Its Neighborhood Only as a Voxel of Shape n × n × f
Input : S: number of samples per batch;
epoch I T _tr ai n: number of iterations required
to parse all the train data set with S samples per batch
(one epoch);
epoch I T _test: number of iterations required to
parse all the test data set with T samples per batch (one
epoch);
Max E poch: Maximum number of epochs to
train the network;
X train : Input training voxels;
X t est : Input testing voxels
Output: mAccuracy: the average accuracy on the test
data set;
All trained weights
for epoch in range Max E poch do
Testing: evaluate accuracy for each voxel from X t est
for it in range epoch I T _tr ai n do
1) Randomly sample S voxels from X train not
already been considered in the current epoch;
2) Forward propagation of the S input voxels
through the network in order to calculate the
average loss; 3) Backward propagation to update
network weights with respect to the average loss
value;
end
Testing: evaluate accuracy for each voxel from X t est
for it in range epoch I T _test do
1) Sample T voxels from X t est not already been
considered in the current epoch; 2) Run a forward
propagation on the batch samples. 3) Retrieve and
cumulate the obtained accuracy values
end
compute m Accur acy, the average accuracy values on
the entire test data set obtained in the current epoch.
end
two different sets of pixels: a training set of pixels and a testing
one. In fact, each pixel is taken into account as a 3-D n ×n × f
voxel in the paper’s context. A training period is considered
to parse one time all the pixels of the training data set (one
epoch) to learn the network weights. Follow a test period
that evaluates the performance level reached on the whole test
data set. Those two periods are applied Max E poch times to
get the final result. In this way, the evolution of the network
performance is monitored all along the training.
B. Main Common Parameters
When establishing a DL architecture, the most crucial phase
is to make wise choices for the different parameters to be set.
Although different models can be obtained from one basic
DL architecture, one can easily notice that there are common
parameters that can be fulfilled from the early stages of the
network creation.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
4425
1) Solver Method: It plays a major role in improving loss
during the learning process and contributes in both the
forward and backward propagation phases and there-
fore ensures the update of the network’s parameters.
Different solver methods have been introduced so far,
such as the Stochastic Gradient Descent (type: “SGD”),
Adam (type: “Adam”), Nesterov’s Accelerated Gradient
(type: “Nesterov”), and RMSprop (type: “RMSProp”) as
detailed in [51] and [52]. The SGD with momentum set
to 0.9 was selected for this use case. Several tests with
different solver types were the reason behind this choice
which appears to be a simple and robust method.
2) Weights Regularization Method: It is basically used to
introduce lateral regularization to the network. The main
types of normalization methods are L1 and L2 weights
regularization; they both provide the choice to either
normalize within the same channel or even normalize
across channels. L1 normalization proved to be more
efficient in the paper’s context. One recent regularization
technique is the dropout method [53] which randomly
disables some inputs thus reducing neuron dependencies.
This also complements L1 and L2 regularization by pre-
venting the network from overfitting. Therefore, we have
combined the use of both L1 and a 0.5 probability
dropout on the FC layer only.
3) Nonlinearity: Most of the attention today is dedicated
to the use of ReLU nonlinearities, since they enable
faster training convergence. Recent nonlinearities, such
as ELU [54], have also been experimented but did not
lead to improved results.
4) Weight Initialization: It is a crucial preprocessing phase
for DL network training, since it sets the state of differ-
ent parameters at the starting point. Different methods
can be used including the recent Xavier and He et al.
initialization method [55] methods. The latter, often
called MSRA, was chosen since it is adapted to the
ReLU nonlinearities that we used.
5) Learning Rate: It coordinates to what extent each step of
the process influences the weight updates. We consider
an initial learning rate of 0.001 to explore rapidly the
search space and find a good local minimum. Then,
for each (Max E poch/3) iteration, the learning rate is
divided by 10 in order to converge to a lower local
loss value and then increase accuracy (MaxEpoch is the
number of epochs needed to train the network).
6) Batch Size: Instead of optimizing the network from a
single sample at a time, which can lead to suboptimal
solutions, averaging errors over a set of samples were
proved to be more efficient. Therefore, the batch size
influences the loss convergence efficacy and in turn the
system’s update phase, which makes it a very critical
parameter and basically data-dependent. Values ranging
from 10 to 1 were tested on the small considered data
set, and the optimal choice is equal to 3.
7) Bias or Batch Normalization: Batch normalization has
recently been proposed to normalize neuron activation
across layers and replace neuron bias variables. Despite
being efficient, batch normalization requires consistent
 4426 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018

hyperspectral_images. The strategy is to start from a sim-

ple state-of-the-art architecture and then gradually extend it
by understanding its bottlenecks. Fig. 4 presents an overall
view of the considered deep networks where the Conv lay-
ers with a stride equal to 1 are referred to as Conv and
those with a stride equal to 2 as ConvPool. The number
of filters per layer is introduced as [number of filters].
So, in Fig. 4, each layer is presented as follows: “Layer-
Name[number of filters].” The a–e schemes, respectively, rep-
resent the 3-, 4-, 6-, 8-, and 10-layer networks as detailed in the
following.
Fig. 3. Captured images. (a) Pavia University. (b) Pavia Center. (c) KSC. 1) Three Layers Versus Four Layers (a Versus b): As first
inspired from the network in [41], the presented archi-
and stable batch statistics in the training and test data- tecture created a three-layer 3-D network that gathers
base. This is often helped by the use of a large batch size. two 3-D layers, one single 1-D layer along with two FC
This constraint is difficult to fulfill on tiny databases layers and a Softmax. However, the important number
so each time we restricted our experiments to neuron of neurons included in the first FC layer increases the
layers equipped with bias parameters initialized to 0 at number of parameters without really improving the accu-
the beginning of the training. racy rates. This layer is basically dedicated to increase
the capacity of the final classifier while taking care of
V. E XPERIMENTS AND A NALYSIS the spatial arrangement in the feature maps. However,
In this section, the experiments conducted on RS images the increased number of parameters conflicts with the
using DL architectures are presented and compared with other low number of training samples. Therefore, in a new
state-of-the-art approaches. four-layer network, only one FC, with the number of
neurons equal to the number of targeted classes, is left
A. Data Sets and a new Conv layer with a stride equal to two is added
to play the role of a pooling layer. The network is then
The results are obtained from experiments applied on the
able to keep up with the same performance level with a
Pavia University, Pavia Center, and Kennedy Space Cen-
lower training cost.
ter (KSC) data sets, all collected by the AVIRIS sensor and
2) Six-Layer Network (c): The noticeable drop in the
shown in RGB colors in Fig. 3. The first employed data
number of trained parameters in the four-layer network
were a capture of an area over Pavia University, northern
provided an interesting opportunity to develop deeper
Italy, with a spatial resolution of 1.3 m. The image comprises
models with more Conv layers. Therefore, one six-layer
610 × 340 pixels with 103 bands. Next, the Pavia Center
3-D architecture was created relying on two 3-D Conv
data set is a 102-band data set that presents one image of
layers followed by four 1-D layers, where the duality
size 1096 × 1096 pixels and of 1.3-m geometric resolution.
Conv/ConvPool is sequentially applied to the network.
Finally, the KSC data set was acquired over the KSC, FL,
Finally, one FC layer with a Softmax was kept. The
USA, on March 23, 1996. This image has 224 bands from
number of filters for each layer was set to 35 except
400 to 2500 nm and the spatial resolution is 18 m. After
for the first Conv that only gathered 20 filters. In fact,
removing water absorption and low signal-to-noise bands,
a wide spectrum of filter numbers with values ranging
it has 512 × 453 pixels with 176 bands. The first two data
from 5 to 50 filters per layer was tested. The optimal
sets include some challenging scenes among the nine classes
combination is then presented in this paper. This choice
which are, respectively, water, trees, asphalt, self-blocking
goes with the standard state-of-the-art tendency that
bricks, bitumen, tiles, shadows, meadows, and bare soil for
shows that fewer filters are required at the beginning
Pavia University and asphalt, meadows, gravel, trees, painted
of the architecture.
metal sheets, bare soil, bitumen, self-blocking bricks, and
3) Squeezing the Net Toward Deeper Architectures
shadows for the Pavia Center data set. The KSC data set
(d and e): Since different deep models can guarantee
includes 13 classes which are scrub, willow swamp, cabbage
the same accuracy level, the choice among them is
palm hammock, cabbage palm/oak hammock, slash pine,
then based on the cost and number of parameters
oak/broadleaf hammock, hardwood swamp, graminoid marsh,
each network can take. Therefore, as inspired from the
spartina marsh, cattail marsh, salt marsh, mud flats, and water.
SqueezeNet presented in [56], the trials to compress
the model led to the creation of lighter models. Thanks
B. Different 3-D Architectures to the use of smaller numbers of filters along with
Extensive sets of experiments were conducted. The main 1 × 1 × 3 filters in the pooling phase, the network
approaches that best summarize the performance level of the can reach the same accuracy levels with a smaller
3-D architectures are reported in the following. The most number of parameters. Indeed, 1 × 1 Conv layers
performing models and trained weights are then made avail- with a number of neurons lower than the one in the
able at https://github.com/AminaBh/3D_deepLearning_for_ previous standard convolutional layer (two neurons

Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION
Fig. 4. Overview of our 3-D architectures.
for ConvPool versus 35 neurons for Conv) allow a
significant reduction of the number of parameters. As a
result, deeper architectures were created for a better
learning.
4) Eight- and Ten-Layer Networks (d and e): A deeper light
architecture has been created with fewer parameters.
An eight-layer 3-D network was created first introducing
the duality between three 3-D Conv layers and three
1-D ConvPool layers, followed by two 1-D layers and
a single FC layer with a softmax. The same goes for
a 10-layer network with the advantage of adding one
more sequence of 3-D layers along with pooling ones.
5) Networks Width Versus Depth: Talking about the
depth of the deep networks is a very rich debate that
generates a lot of questions. However, it has been
recently proved [45] that one of the keys for better
performances is to find the right balance between the
network’s depth and width. In other words, fixing the
number of layers per network is a very crucial step that
has a huge impact on the efficiency of the architecture.
But, knowing that the network has a variant number
of filters per layer catalyzes the concerns about which
number to choose. Therefore, estimating the appropriate
number of filters for each layer (the width of the
network) according to its depth is an important decision
to be taken in order to harmonize the cost and accuracy
of a deep network. In this paper, the first layer is
characterized with 20 filters, whereas the rest of the
layers have 35 filters, except for the pooling layers that
have 2, 4, or 8 filters according to their position in the
network as discussed above in this section.
C. Experiments and Results
Experiments and tests were executed on the 3, 4, 6, 8,
and 10 Conv layered architectures as detailed in the previous
section. For the sake of an accurate comparison with state-of-
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
4427
the-art methods, two main data splitting strategies were used.
When proceeding with the three- and four-layer networks,
only 200 randomly chosen pixels per class were used for
training (almost 4%), whereas the rest of the data pixels
were kept for the testing phase. Then, in the case of deeper
architectures, 5% of the images were deployed for training
using the same class balance strategy. Each model is trained
and evaluated three times using different nonoverlapping train
and test random splits. Accuracy levels are averaged to report
a synthetic performance measure. We measured a redundant
0.2% precision error and do not report this value within the
tables to lighten the presentation. All the tests were executed
on a four-core intel i7-6600U laptop CPU with no GPU
included. The presented results were obtained using the caffe
library [57].
1) Three Layers Versus Four Layers (a Versus b): First,
as detailed in [4], the introduction of the 3-D architecture
is inspired from [41]. Technically, two main differ-
ences distinguish the proposed architecture from the one
detailed in [41]. First, trainable convolution layers with
strides greater than one are used instead of the classically
used max pooling filters. Then, the 100 neuron FC
introduced in [41] is replaced by a 50 neuron FC.
However, the accuracy difference cannot be explained
only by these factors. In fact, as detailed in Table I,
if the number of neurons in the FC layer is divided by 2
(50 neurons versus 100 in [41]), the parameter number
decreases by almost 18% while the performance level
remains stable. This shows that the initial system has
actually too many degrees of freedom which misleads
the image representation and limits the system efficiency.
Although these primitive results do not enhance the
level of the accuracy rate, they provide us with hints
toward establishing better models. The first observation
to be taken into account is that the choice of the spatial
neighborhood is very crucial and data-dependent. Here,
 4428 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018

TABLE I
A CCURACY L EVEL OF THE T HREE -L AYER CNN M ODELS (N ETWORK a) U SING 4.4% OF THE D ATA FOR T RAINING

TABLE II
A CCURACY L EVEL OF THE F OUR -L AYER CNN M ODELS (N ETWORK b) U SING 4.4% OF THE D ATA FOR T RAINING

the 5 × 5 input spatial neighborhood seems to be
the optimum choice for the Pavia University data set,
whereas the 3 × 3 one performs better in the cases of
the Pavia Center and KSC data sets. However, the 7 × 7
spatial neighborhood seems to be very extended com-
pared with the spatial components of the data sets which
makes it a low performer. Furthermore, the decrease in
the number of neurons in the first FC layer results in a
decrease in the number of the overall trained parameters
without influencing the accuracy rates.
As detailed in Table II, the removal of the first FC
layer along with the introduction of the spatiospectral
concept in the network enables better results compared
with [41]. In fact, in this case, the data representa-
tion is more relevant when going from 3-D voxels to
1-D vectors followed by a single nc-neuron FC layer
(nc = number of the data set classes) combined with a
Softmax. Not only does this model benefits from better
accuracy rates, it also witnesses an important decrease
on the computational cost level. Going down from over
60 000 parameters trained in [41] to 28 749 parameters,
the proposed four-layer network ensures both a better
learning of the data content and a lower training cost
process. Here again, the results prove that the choice
of the spatial neighborhood is 5 × 5 for the Pavia
University data set and 3 × 3 for the Pavia Center data
set. The reliance on the 3-D Conv layers for combined
spatiospectral classification of the data is therefore a
basic key for better results when compared with [41]
that only resort to the spectral signature of each pixel in
the classification process.
2) Six-Layer Network (c): Establishing a six-layer network
was inspired by the important decrease in the number of
parameters in the three- to four-layer transition case. The
introduction of a sequence of Conv and Pooling duets in
the network gather both the benefits of going deeper and
involving fewer parameters. In other words, more Conv
layers ensure higher semantic level representation of the
data, whereas the Pooling ones guarantee a dimension
reduction of the representation. This way, the dimension
of the vectors at the entry of the FC layer is remark-
ably reduced thus significantly reducing the number
of parameters. As shown in Table III, an important
decrease in the number of parameters (from a tenth
of the previous 60 000) is witnessed, along with an
increase in the accuracy rate. These tests also prove
that the choice of the spatial neighborhood is highly
dependent on the data content. The same model can
outperform the results in [5] in the case of the Pavia
Center data set with a 3 × 3 neighborhood while it
does not reach the state-of-the-art approach results in
the case of Pavia University even when using 5 × 5
neighborhood.
3) Squeezing the Net, 8- or 10-Layer Networks (d or e):
According to the results presented in Tables III, IV
and V, the main key behind a successfully performing
deep network is to create a balance between a deep
yet light architecture. When examining the models,
it can be seen that the number of layers in the network
(the depth) and the number of neurons per layer (the
width) manipulate the most important share of the
network performance. Therefore, in Tables VI and VII,
the tests were executed for different combinations of a
number of neurons per layer, a number of 3-D layers
per architecture, and the overall number of layers in the
network. Although a wider range of values was tested

Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION 4429

TABLE III
A CCURACY L EVEL OF THE S IX -L AYER CNN M ODELS (N ETWORK c) U SING 5% OF THE D ATA FOR T RAINING

TABLE IV
A CCURACY L EVEL OF THE E IGHT-L AYER CNN M ODELS (N ETWORK d) U SING 5% OF THE D ATA FOR T RAINING

TABLE V
A CCURACY L EVEL ON KSC D ATA S ET U SING 5% OF THE D ATA FOR T RAINING

TABLE VI TABLE VII

S QUEEZING THE N ET ON PAVIA U NIVERSITY (N ETWORKS c–e) S QUEEZING THE N ET ON PAVIA C ENTER (N ETWORKS c–e)
U SING 5% OF THE D ATA FOR T RAINING U SING 5% OF THE D ATA FOR T RAINING

provides more opportunities to create deeper mod-

for each parameter presented in the following, only the
most effective ones are mentioned at this stage.
As detailed in Tables VI and VII, the decrease in the
number of neurons per pooling layer enables lighter
models that maintain the same accuracy rate ranges.
Basically, the decrease in the width of the network
els. Therefore, the six-layer network performs less
than the 8- and 10-layer ones. However, the eight-layer
architecture seems to be the best choice especially when
relying on three 3-D layers, along with a small number
of neurons in the pooling layers. Not only does it
reduce the number of parameters but it also enhances
the accuracy rate.
The best derived combination in our case is an eight-
layer network with three 3-D blocks. The number of

Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
 4430 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018

Fig. 5. Pavia University data set. (a) Image, (b) ground truth, and (c) classification output of six-layer network.

Fig. 6. Confusion matrix using the eight-layered network. (a) Pavia University. (b) Pavia Center. (c) KSC.

filters is equal to 20 for the first Conv layer and
increased to 35 for the following Conv layers. However,
the numbers of neurons for each Pooling layer are
equal to 2, 2, 2, and 4. Here, we proceed with an
input voxel of size [5,5,103]. Table IV highlights the
best performing architectures in the case of eight-layer
networks.
D. Architecture Selection
The introduction of different architectures proposed in
Section V-B enables different performance levels. At this
stage, the choice of an optimal network is made. As previously
detailed, many factors interfere with the selection process of
a best performing network. In fact, the progressive evolution
toward a deeper, yet lighter network, draws the guideline
toward an easier choice.
The first presented architectures (a and b) managed to
establish a solid baseline for the creation of deeper 6-, 8-,
and 10-layer networks. Since, the c, d, and e models enable
high accuracy rates, the computational cost plays then a major
role in the choice making process. As detailed in Tables I–VII,
the eight-layer network enable the highest accuracy rates for
all of the three data sets. Besides, it ensures low computational
costs with lower training parameter numbers. Therefore, for
what comes next, the eight-layered architecture will be used.
As shown in Fig. 5, the display of the output classification map
does not give much information about the system’s precision,
since the accuracy rates overpass the 90% in most of the
six- and eight-layer cases of study. Therefore, the confusion
matrices are drawn and presented in Fig. 6 in order to
demonstrate the performance level of the networks. In this
figure, the first two data sets represent classes ranging from 0
to 8 and the KSC confusion matrix represents classes ranging
from 0 to 12 according to their enumeration order used in
Section V-A. Although these matrices show highly accurate
classification rates, they also demonstrate the confusions made
by the trained network as shown in Fig. 6. In the case of
the Pavia University data set, the Bitumen pixels are mixed
up with the shadow pixels with almost 1/10 mistaken pixels
among all the classified ones. The lowest accuracy rate (almost
60%) is witnessed in the case of the KSC image, where
the cabbage palm/oak hammock class is confused with four
other classes. The mistaken pixels are classified as 20%

Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION
Fig. 7. Processing time required to reach 95% of the final accuracy on the
Pavia University data set using the eight-layer 3 × 3 network.
oak/broadleaf hammock, 10% cabbage palm hammock, and
less than 10% divided between slash pine and water. The
system behavior toward such classes is basically driven by
the high spectral and spatial similarity between the two class
characteristics.
In order to get better insights on the choice of a given
architecture, taking a look at the accuracy and loss values
along the training phase is critical. Actually, these indicators
can validate the choice of hyperparameters, such as the learn-
ing rate, but it also shows the training stability and how an
architecture reaches a stable performance level. Plotting those
curves for several architectures can, however, be confusing,
added to the fact that they are not perfectly reproducible
due to random weights initialization. Then, for readability,
we report the monitored values obtained at a specific iteration
close to the maximum accuracy value but before the steady
state. It somehow represents a form of early stopping in the
training process where system performances are estimated.
More specifically, we select the iteration index that reports
95% of the final accuracy value. Note that averaging those
values along multiple experiments enhances related values
confidence. Fig. 7 shows an example of such monitoring report
on a single experiment. One can also observe a step at iteration
22 000 for both the loss and accuracy curves. This actually
corresponds to the iteration where a decrease in the learning
rate occurs. This change suddenly improves accuracy, since
a good local minimum has been found previously, and then
convergence to a lower loss value is made possible. In the
proposed network setup, this step happens a few iterations
before the proposed architectures report 95% of their final
accuracy value. Finally, Table VIII makes the synthesis of the
obtained results on the most performing architectures. As a
conclusion, the eight-layer architectures enable high accuracy
rates within a reasonable period of time in this paper. One
can also notice the impressive convergence speed of the eight-
layer architecture relying on 3 × 3 pixels neighborhood. Such
architecture enables access to high-accuracy values at very low
cost and is then the best compromise to obtain rapidly a good
classifier in environment with low energy and processing time
limitations.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
4431
TABLE VIII
P ROCESSING T IME R EQUIRED TO R EACH 95% OF THE F INAL
A CCURACY ON THE PAVIA U NIVERSITY D ATA S ET
E. Computational Cost and Training Data Requirements
Obviously, the developed deep architecture overperforms the
existing methods in the case of the Pavia Center data set.
Unlike different approaches, such as [58]–[60], the 3-D pro-
posed architectures enable from scratch-training with no prior
data preprocessing. Not only does it simplify the processing
task, the 3-D architecture enables the same performance levels
as the previously mentioned approaches. Besides, the time
taken is a main factor when evaluating DL architectures. Here,
the classification process takes almost 3 h at the most with
a single Intel i7-6600U laptop CPU (no GPU used) in the
case of heavy architectures along with relatively large data
set images. However, it only takes less than 2 min in the
case of light networks with a few training samples. Another
key factor that normally influences the performances of these
classification processes is the amount of data involved for the
training phase. For example, when using only 5% of the image
pixels for training, the 3-D architecture cannot overpass the
98% accuracy rate in the case of the Pavia University data set
(even when using the most performing architecture). However,
when dedicating 10% of the data for training as experimented
in [58], the accuracy rates in that case are more than 99.4%. As
detailed in Fig. 8, the 3-D architecture is capable of reaching a
99% accuracy rate while only using 9% of the image pixel to
train the network. The state-of-the-art method detailed in [37]
also deploys 9% of the data set for training. However, this
approach underperforms the proposed 3-D architecture with a
98% accuracy level for about 20 000 trained parameters against
our 99% accuracy level for less than 7000 trained parameters.
F. Model Transferability
Sections V-B and V-C demonstrate the possibilities of
establishing a new light deep neural network that takes into
account both the spectral and the spatial raw data. It has been
proven so far that this architecture performs well in the case
of hyperspectral data. However, regarding the lack of richly
annotated hyperspectral images, this paper and more generally
state-of-the-art methods only examine and review training and
testing in the same context. First, the training is executed using
a subset of a single image that is specific to a given context.
Then, testing and inference are based on the remaining pixels
of the same image, i.e., we generalize on the same context.
However, many questions can arise from this strategy, since the
 4432 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018
Fig. 8. Accuracy (%) versus amount of training data (%) using the eight-layer CNN model (network d).
TABLE IX
A CCURACY L EVEL FOR F INE -T UNING U SING 5%
OF THE D ATA FOR T RAINING
DL models are highly exposed to the possibility of overfitting
and so did the model overfit on each specific data context?
Can the learned features be transferred from one data set to
the other?
Therefore, in what follows, the learned feature transferabil-
ity between different contexts is proposed, trained using one
specific image, and evaluated based on a second image. Since
the target classes are the same, transfer learning between Pavia
University and Pavia Center is proposed. However, the number
of spectral bands differs (103 bands versus 102 bands) so that
the output dimensions of the convolution layers will differ
from one data set to the other. In this context, it is necessary
to resort to architecture fine-tuning: given a neural network
architecture trained on a given data set, all its architecture
components (except the last FC layers) are kept and their
weights are made constant. Finally, the FC layers are replaced
with new ones whose number of connections (weights) are
compatible with the new data set and the shape of the
convolution layer output. A rapid training of the new network
component is performed on the new data set on its own training
set, and performances are evaluated on the test set.
As detailed in Table IX, the use of a pretrained eight-
layer model in the case of the Pavia University and Pavia
Center data sets provide an accurate pixelwise classification
of the data. In fact, the deep neural networks are able to
maintain nearly the same precision level when fine-tuned
and trained from scratch (98.4% versus 98.9% and 90.4%
versus 92.9%). Basically, the pretrained architectures proposed
in this paper demonstrate a strong ability to generalize to other
context images. As in the case of the Pavia University and
Pavia Center data sets, the reuse of pretrained networks saves
the huge effort required to recreate a specific architecture to
be trained from scratch from each and every use case.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
VI. C ONCLUSION
The processing of hyperspectral data in general is a very
delicate procedure that demands the effective use of both
spatial and spectral components. The benefit of the 3-D archi-
tecture introduced in this paper is not only to accurately
classify the hyperspectral data but also to establish deep
comprehension of the images at low cost. One of the most
valuable consequence is the ability to efficiently optimize deep
networks on small sized annotated data sets which then also
reduce the cost of the data. The main concern now is to
investigate ways to innovate and enhance the created models
in order to process with larger and heavier data sets. As a
remedy for such a challenge, both residual and dense networks
enable the fusion of different representation levels. Therefore,
they would seem like an appealing solution to enhance the
existing CNN architectures. Furthermore, hyperspectral data
calibration is still an open issue that currently confines its use
to limited areas. An interesting path would therefore to create
architectures that are able to cope with this issue.
R EFERENCES
[1] T. Kavzoglu and P. M. Mather, “The use of backpropagating artificial
neural networks in land cover classification,” Int. J. Remote Sens.,
vol. 24, no. 23, pp. 4907–4938, 2003.
[2] D. K. McIver and M. A. Friedl, “Using prior probabilities in decision-
tree classification of remotely sensed data,” Remote Sens. Environ.,
vol. 81, nos. 2–3, pp. 253–261, Aug. 2002.
[3] M. Fauvel, J. Chanussot, and J. A. Benediktsson, “Evaluation of kernels
for multiclass classification of hyperspectral remote sensing data,” in
Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), vol. 2.
May 2006, p. 2.
[4] A. Ben Hamida, A. Benoit, P. Lambert, and C. Ben Amar, “Deep
learning approach for remote sensing image analysis,” in Proc. Big Data
Space (BiDS), 2016, p. 133.
[5] S. Lefèvre, L. Chapel, and F. Merciol, “Hyperspectral image classi-
fication from multiscale description with constrained connectivity and
metric learning,” in Proc. IEEE Workshop Hyperspectral Image Signal
Process., Evol. Remote Sens. (WHISPERS), Jun. 2014, pp. 1–4.
[6] G. Camps-Valls, D. Tuia, L. Bruzzone, and J. A. Benediktsson,
“Advances in hyperspectral image classification: Earth monitoring with
statistical learning methods,” IEEE Signal Process. Mag., vol. 31, no. 1,
pp. 45–54, Jan. 2014.
[7] F. Rosenblatt, “The perceptron: A probabilistic model for information
storage and organization in the brain,” Psychol. Rev., vol. 65, no. 6,
p. 386, 1958.
[8] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of
data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507,
2006.
 BEN HAMIDA et al.: 3-D DL APPROACH FOR RS IMAGE CLASSIFICATION
[9] R. Salakhutdinov and G. Hinton, “Deep Boltzmann machines,” in Proc.
Artif. Intell. Statist., 2009, pp. 448–455.
[10] D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A learning algorithm
for Boltzmann machines,” Cognit. Sci., vol. 9, no. 1, pp. 147–169, 1985.
[11] A. Fischer and C. Igel, “An introduction to restricted Boltzmann
machines,” in Progress in Pattern Recognition, Image Analysis, Com-
puter Vision, and Applications. 2012, pp. 14–36.
[12] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm
for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554,
2006.
[13] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based
learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11,
pp. 2278–2324, Nov. 1998.
[14] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[15] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-
wise training of deep networks,” in Proc. Adv. Neural Inf. Process. Syst.,
2007, pp. 153–160.
[16] L. Breiman, “Stacked regressions,” Mach. Learn., vol. 24, no. 1,
pp. 49–64, 1996.
[17] L. Deng and D. Yu, “Deep learning: Methods and applications,”
Found. Trends Signal Process., vol. 7, nos. 3–4, pp. 197–387,
Jun. 2014.
[18] D. C. Cireşan, U. Meier, L. M. Gambardella, and J. Schmidhuber,
“Deep, big, simple neural nets for handwritten digit recognition,” Neural
Comput., vol. 22, no. 12, pp. 3207–3220, 2010.
[19] D. Ciregan, U. Meier, and J. Schmidhuber, “Multi-column deep neural
networks for image classification,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jun. 2012, pp. 3642–3649.
[20] I. J. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv.
Neural Inf. Process. Syst., 2014, pp. 2672–2680.
[21] A. G. Ororbia, II, C. L. Giles, and D. Kifer. (2016). “Unifying adversarial
training algorithms with flexible deep data gradient regularization.”
[Online]. Available: https://arxiv.org/abs/1601.07213
[22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016,
pp. 770–778.
[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
Process. Syst., 2012, pp. 1097–1105.
[24] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolu-
tional networks,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 818–833.
[25] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 1–9.
[26] M. M. Botvinick and D. C. Plaut, “Short-term memory for serial order:
A recurrent neural network model,” Psychol. Rev., vol. 113, no. 2,
pp. 201–233, 2006.
[27] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural
image caption generator,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., Jun. 2015, pp. 3156–3164.
[28] M. Fayyaz, M. H. Saffar, M. Sabokrou, M. Fathy, R. Klette, and
F. Huang. (2016). “STFCN: Spatio-temporal FCN for semantic video
segmentation.” [Online]. Available: https://arxiv.org/abs/1608.05971
[29] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., Jun. 2015, pp. 3431–3440.
[30] L. Zhang, L. Zhang, and B. Du, “Deep learning for remote sensing data:
A technical tutorial on the state of the art,” IEEE Geosci. Remote Sens.
Mag., vol. 4, no. 2, pp. 22–40, Jun. 2016.
[31] J. Zabalza et al., “Novel segmented stacked autoencoder for effective
dimensionality reduction and feature extraction in hyperspectral imag-
ing,” Neurocomputing, vol. 185, pp. 1–10, Apr. 2016.
[32] K. Karalas, G. Tsagkatakis, M. Zervakis, and P. Tsakalides, “Deep
learning for multi-label land cover classification,” Proc. SPIE,
vol. 9643, pp. 9643-1–9643-14, Oct. 2015. [Online]. Available:
https://doi.org/10.1117/12.2195082, doi: 10.1117/12.2195082.
[33] M. Fauvel, Y. Tarabalka, J. A. Benediktsson, J. Chanussot, and
J. C. Tilton, “Advances in spectral-spatial classification of hyperspectral
images,” Proc. IEEE, vol. 101, no. 3, pp. 652–675, Mar. 2013.
[34] K. Huang, S. Li, X. Kang, and L. Fang, “Spectral–spatial hyperspectral
image classification based on KNN,” Sens. Imag., vol. 17, no. 1, p. 1,
Dec. 2016.
[35] W. Zhao and S. Du, “Learning multiscale and deep representations for
classifying remotely sensed imagery,” ISPRS J. Photogramm. Remote
Sens., vol. 113, pp. 155–165, Mar. 2016.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
4433
[36] F. Zhang, B. Du, L. Zhang, and L. Zhang, “Hierarchical feature
learning with dropout k-means for hyperspectral image classification,”
Neurocomputing, vol. 187, pp. 75–82, Apr. 2016.
[37] X. Ma, J. Geng, and H. Wang, “Hyperspectral image classification via
contextual deep learning,” EURASIP J. Image Video Process., vol. 2015,
no. 1, p. 20, Dec. 2015.
[38] A. Romero, C. Gatta, and G. Camps-Valls, “Unsupervised deep feature
extraction for remote sensing image classification,” IEEE Trans. Geosci.
Remote Sens., vol. 54, no. 3, pp. 1349–1362, Mar. 2016.
[39] K. Makantasis, K. Karantzalos, A. Doulamis, and N. Doulamis, “Deep
supervised learning for hyperspectral data classification through con-
volutional neural networks,” in Proc. IEEE Int. Geosci. Remote Sens.
Symp. (IGARSS), Jul. 2015, pp. 4959–4962.
[40] V. Slavkovikj, S. Verstockt, W. De Neve, S. Van Hoecke, and
R. Van de Walle, “Hyperspectral image classification with convolutional
neural networks,” in Proc. 23rd ACM Int. Conf. Multimedia, Oct. 2015,
pp. 1159–1162.
[41] W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, “Deep convolutional
neural networks for hyperspectral image classification,” J. Sensors,
vol. 2015, Jan. 2015, Art. no. 258619.
[42] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten.
(2016). “Densely connected convolutional networks.” [Online]. Avail-
able: https://arxiv.org/abs/1608.06993
[43] Y. Bengio, “Artificial neural networks and their application to sequence
recognition,” Ph.D. dissertation, School Comput. Sci., McGill Univ.,
Montreal, QC, Canada, 1991.
[44] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, “A neural proba-
bilistic language model,” J. Mach. Learn. Res., vol. 3, pp. 1137–1155,
Feb. 2003.
[45] S. Zagoruyko and N. Komodakis. (2016). “Wide residual networks.”
[Online]. Available: https://arxiv.org/abs/1605.07146
[46] S. Han, H. Mao, and W. J. Dally. (2015). “Deep compression: Com-
pressing deep neural networks with pruning, trained quantization and
Huffman coding.” [Online]. Available: https://arxiv.org/abs/1510.00149
[47] S. Han et al., “EIE: Efficient inference engine on compressed deep
neural network,” in Proc. 43rd Int. Symp. Comput. Archit., Jun. 2016,
pp. 243–254.
[48] A. Parashar et al., “SCNN: An accelerator for compressed-sparse
convolutional neural networks,” in Proc. 44th Annu. Int. Symp. Comput.
Archit., 2017, pp. 27–40.
[49] Z. Zhong, J. Li, L. Ma, H. Jiang, and H. Zhao, Deep Residual Networks
for Hyperspectral Image Classification. Piscataway, NJ, USA: Institute
of Electrical and Electronics Engineers, 2017.
[50] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. (2014).
“Striving for simplicity: The all convolutional net.” [Online]. Available:
https://arxiv.org/abs/1412.6806
[51] L. Bottou, “Large-scale machine learning with stochastic gradient
descent,” in Proc. COMPSTAT, 2010, pp. 177–186.
[52] D. P. Kingma and J. Ba. (2014). “Adam: A method for stochastic
optimization.” [Online]. Available: https://arxiv.org/abs/1412.6980
[53] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: A simple way to prevent neural networks
from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958,
2014.
[54] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. (2015). “Fast and
accurate deep network learning by exponential linear units (ELUs).”
[Online]. Available: https://arxiv.org/abs/1511.07289
[55] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:
Surpassing human-level performance on ImageNet classification,” in
Proc. IEEE Int. Conf. Comput. Vis., Dec. 2015, pp. 1026–1034.
[56] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally,
and K. Keutzer. (2016). “SqueezeNet: AlexNet-level accuracy with
50x fewer parameters and <0.5 MB model size.” [Online]. Available:
https://arxiv.org/abs/1602.07360
[57] Y. Jia et al., “Caffe: Convolutional architecture for fast feature embed-
ding,” in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 675–678.
[58] Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extrac-
tion and classification of hyperspectral images based on convolutional
neural networks,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10,
pp. 6232–6251, Oct. 2016.
[59] Y. Yuan, J. Lin, and Q. Wang, “Dual-clustering-based hyperspectral band
selection by contextual analysis,” IEEE Trans. Geosci. Remote Sens.,
vol. 54, no. 3, pp. 1431–1445, Mar. 2016.
[60] Q. Wang, J. Lin, and Y. Yuan, “Salient band selection for
hyperspectral image classification via manifold ranking,” IEEE
Trans. Neural Netw. Learn. Syst., vol. 27, no. 6, pp. 1279–1289,
Jun. 2016.
 4434 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 56, NO. 8, AUGUST 2018
Amina Ben Hamida received the National Diploma
degree in electrical engineering from the Ecole
Nationale d’Ingénieurs de Sfax, Sfax, Tunisia. She is
currently pursuing the Ph.D. degree with Université
Savoie Mont Blanc, Annecy, France, with a focus on
novel approaches that deploy deep learning networks
for remote sensing image classification tasks, specif-
ically within the hyperspectral and multispectral
data communities. One of her biggest ventures for
her research project will be the establishment of
balance between high accuracy and low cost for deep
learning architectures.
Alexandre Benoit received the Ph.D. degree in
electronics and computer science from the Grenoble
Institute of Technology, University of Grenoble,
Grenoble, France, in 2007.
Since 2008, he has been an Associate Pro-
fessor with the Informatics, Systems, Information
and Knowledge Processing Laboratory, Université
Savoie Mont Blanc, Annecy, France. His research
interests include image and video understanding.
He actively participates to multimedia indexation
challenges such as TRECVid. He develops pattern
extraction and analysis methods and data fusion processes for high semantic
level image and video indexing and segmentation. He contributes to the
open source OpenCV library by providing a specific spatio-temporal filtering
module, bioinspired.
Authorized licensed use limited to: NUST School of Electrical Engineering and Computer Science (SEECS). Downloaded on November 02,2023 at 07:44:58 UTC from IEEE Xplore. Restrictions apply.
Patrick Lambert received the Ph.D. degree in
signal processing from the National Polytechnic
Institute of Grenoble, Grenoble, France, in 1983.
He is currently a Full Professor with the School of
Engineering, Université Savoie Mont Blanc, Annecy,
France, and a member of the Informatics, Sys-
tems, Information and Knowledge Processing Labo-
ratory, Annecy, France. His research interests include
image and video analysis, and actually dedicated
to non-linear color filtering and automatic image
understanding.
Chokri Ben Amar (SM’08) received the B.S.
degree in electrical engineering from the Ecole
Nationale d’Ingénieurs de Sfax (ENIS), Sfax,
Tunisia, in 1989, and the M.S. and Ph.D. degrees in
computer engineering from the National Institute of
Applied Sciences of Lyon, Lyon, France, in 1990 and
1994, respectively.
He spent one year at the University of Haute
Savoie, Annecy, France, as a Teaching Assistant
and a Researcher before joining the higher School
of Sciences and Techniques of Tunis, Tunis, as an
Assistant Professor, in 1995. In 1999, he joined the University of Sfax, Sfax,
where he is currently a Full Professor with the Department of Computer
Engineering and Applied Mathematics, Ecole Nationale d’Ingénieurs de Sfax.
His research interests include computer vision and image and video analysis.
These research activities are centered on wavelets and wavelet networks and
their applications to data classification and approximation, pattern recognition,
and image and video indexing and securing.
Dr. Ben Amar founded the IEEE Signal Processing Society (SPS) Tunisia
Chapter in 2009, and he is actually the Chair of this Chapter. During this
period, the chapter organized the five IEEE Distinguished Lectures and other
technical and professional activities. He has been the Advisor of the IEEE
SPS Student Chapter, ENIS, since 2010.