Journal Pre-proof

Deep attention based optimized Bi-LSTM for improving geospatial data

ontology

Palaniappan Sambandam, D. Yuvaraj, P. Padmakumari,

Subbiah Swaminathan

PII: S0169-023X(22)00114-8
DOI: https://doi.org/10.1016/j.datak.2022.102123
Reference: DATAK 102123

To appear in: Data & Knowledge Engineering

Received date : 3 August 2022

Revised date : 2 November 2022
Accepted date : 25 November 2022

Please cite this article as: P. Sambandam, D. Yuvaraj, P. Padmakumari et al., Deep attention based
optimized Bi-LSTM for improving geospatial data ontology, Data & Knowledge Engineering
(2022), doi: https://doi.org/10.1016/j.datak.2022.102123.

This is a PDF file of an article that has undergone enhancements after acceptance, such as the
addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive
version of record. This version will undergo additional copyediting, typesetting and review before it
is published in its final form, but we are providing this version to give early visibility of the article.
Please note that, during the production process, errors may be discovered which could affect the
content, and all legal disclaimers that apply to the journal pertain.

© 2022 Published by Elsevier B.V.

 Journal Pre-proof
Manuscript File Click here to view linked References

Deep Attention Based Optimized Bi-LSTM for Improving Geospatial Data

Ontology
*1Dr.Palaniappan Sambandam, 2Dr. D. Yuvaraj, 3Dr. P. Padmakumari,
4Dr. Subbiah Swaminathan

of
*1
Associate Professor, Department of Artificial Intelligence and Data Science,
KCG College of Technology, Anna University, Chennai, India.
2
Professor, Department of Computer science
Cihan University – Duhok, Kuridsitan Region, Iraq.

pro
3
School of Computing, Computer Science and Engineering
SASTRA Deemed to be University, Thanjavur, India.
4
Professor, Department of Information Technology,
Rajalakshmi Engineering College Thandalam, Tamil Nadu, India.
*
Email: palaniyappand12@gmail.com

Abstract: Recently, the geospatial semantic information of remote sensing (RS) has attracted
attention due to its various applications. This paper introduces a model for ontology based
geospatial data integration using novel deep learning techniques. Here, we use a semantic web
re-
technology to establish the spatial ontology of risk knowledge with deep learning (DL), namely
deep attention based bidirectional search and rescue LSTM for analysis. This approach takes
into consideration of the study which presents the technique driven by the spatial ontology
which minimizes the cost of modelling. The classification results from DL model enhances the
performance of the ontology module. In this paper, ontological reasoning and DL model are
jointly used for increase the module efficiency. The implementation of the proposed scheme is
implemented on MATLAB 2020a.The performance of the implemented scheme is compared
lP
against the existing models like U-Net, Semantic referee and collaboratively boosting
framework (CBF). The Overall accuracy (OA) of the system is found to be 0.923 on UCM
dataset. Thus, the developed spatial ontologies provide the semantic foundation to achieve a
semantic knowledge of geospatial data understandings.

Keywords: Ontology, Geospatial Data, Deep Attention Based Bidirectional Search and Rescue
LSTM
rna

1. Introduction
The analysis of the Earth by the huge amount of geoscience data obtained by model simulations
and sensor observation. It is called as “big data” since they satisfy velocity, veracity, variety
and volume [1]. The detection of geospatial object from RS has the objects like air, bridges and
planes on the surface of the earth [2]. The detection of geospatial object has the limitations like
heavy distribution, high arbitrary orientations and they are huge in size when compared to
object detection from natural images [3]. The geospatial data is the major access to data driven
geography in the big earth data, which need effective geospatial data sharing and integration
Jou

[4]. But this data sharing and integration has many problems. The major problems is semantic
heterogeneity occur due to the properties of several types, forms and source of geospatial data.
According to the metadata standard, many efforts are taken for this challenge like International
Organization for Standardization (ISO) and Federal Geographic Data Committee (FGDC). But
these efforts partially overcome the challenges. Therefore to overcome the heterogeneity issue
ontology framework are developed [5]. Ontology is a formal and exposed statements of
concepts that are shared in machine readable manner. It provides a semantic explanation for
 Journal Pre-proof

geospatial data and makes the system to find out the semantic meaning which are indirectly
present in the geospatial data content [6].
Further Ontology can be used for describing the relationship among semantic features [7].
As a basic process in the RS, the semantic segmentation of RS focus to annotate every pixel of
the RS imagery (RSI) using land-use/land-cover (LULC), and it plays a major role on huge

of
fields like ecological assessment and intelligent agriculture. RSI semantic segmentation is same
as natural image segmentation in the artificial intelligence (AI) [8]. But RSI provides complex
architecture and has many types of classification with arbitrary orientation [9]. The semantic
web and deep learning techniques are largely used in geographic information systems [10].
Initially researchers used geo-data for performing computer vision tasks like LULC, object

pro
recognition and semantic segmentation. Then, large number of applications like map query and
navigation are used.
Further, Machine learning (ML) models like Decision Tree (DT) [11], Maximum likelihood
estimate (MLE) [12] and Extreme Learning Machine (ELM) [13] are used for geospatial
semantic problems. Because, the performance is based on the handcrafted features, these
classifiers are not widely used [14]. Therefore with rapid progress of DL models like deep
recognition networks, deep hashing network and deep detection network. Many geosciences
research have an attention on using DL models for classification of images, state prediction
and feature extraction [15, 16]. The DL model convolutional neural network (CNN) was
re-
utilized for LULC on two radar image and 3 hyper spectral images. DL models can enhance
performance using different kinds of knowledge on the learning method from various sources
that explain the facts in various methods [17-20].
Recently, the research works have been used ontology to model measurements, observation
and geospatial web service access. But ontology reasoners can use only ontology samples like
Web Ontology Language (OWL) and Resource Description Framework (RDF). The process of
converting to ontology from geospatial data is ineffective, takes more time and sometime leads
lP
to error. Further the ontology devices like Protégé software is not able to effectively manipulate
ontology samples in huge quantity because of the memory consumption. Further, it is not an
expensive for ontology devices to include the processes assured by geo databases. Hence to
support ontology reasoning on geospatial data DL models are developed. DL have shown much
improvements in various applications of Remote sensing (RS).
The existing research works utilizes the data driven learning strategy and it doesn’t utilize
the high level domain expert’s knowledge. Hence, these models are sensitive to noise attacks.
rna

Further, the ontology with knowledge’s approaches have more benefits in illustrating and
providing knowledge. Some of the problems in existing models are: the ontology reasoning is
not directly used in the process of classification and doesn’t guide directly the classification.
Some existing method does not provide detailed information about geospatial data, it provides
the information only about data sharing and integration. Finally, many semantic barriers were
developed in the multisource data application. Hence, to tackle these challenges deep attention
based bidirectional search and rescue LSTM is introduced. In this system, the weight of the
bidirectional LSTM (Bi-LSTM) is updated by the Search and Rescue (SAR) optimization
technique.
Jou

1.1 Contribution
The contributions of the implemented model are:
 This paper introduces a model deep attention based bidirectional search and rescue
LSTM for geospatial data integration.
 The ontology reason is used to improve the semantic work of Bi-LSTM and the optimal
weight of SAR is used for updating the weight of Bi-LSTM.
 Journal Pre-proof

 Then the RSontology is used for establishing the objects attributes and relation among
the objects.
 The implemented model’s performance is compared with the recent published papers
with metrics Overall accuracy (OA) and Mean Intersection over Union (mIOU) on
UCM database.

of
The remaining structure of the research article is arranged as follows: section 2 is the most
recent related research models; section 3 explained the developed scheme; section 4 gives the
discussion of implemented results and at last, the overall conclusion of work in section 5.

2. Related works

pro
Kyzirakos et al. [21] developed a GeoTriples model which was used for transforming
geospatial data saved in raw files and spatially-enabled RDBMS to RDF graphs by
GeoSPARQL and stPARQL. This model works in 3 phases. Initially, it develops R2RML
mapping for transforming the input to RDF. Then there mapping were revised and at last
GeoTriples was used to perform these mapping and develops an RDF. The performance was
evaluated on geospatial datasets and proved that GeoTriples was very effective and scalable,
particularly when the mapping process was evaluated by Apache Hadoop.
Sun et al. [22] developed an ontology model called GeoDataOnt to ensure semantic
geospatial data sharing and integration. Initially, the geospatial data characteristics was defined
re-
and verified the semantic challenges in geospatial data sharing and integration. Then a
GeoDataOnt was modelled to explain about information in the ontology. Then the dimensions
like logic dimension, content dimension and ontological category dimensions were developed.
Based on the compound level characteristics, GeoDataOnt was categorized into morphology,
provenance and essential ontology. The implementation was done on Protégé software and
combined to develop ontology base.
Li et al. [23] presented a GeoDIP which was modelled on the basis of HPC cluster of
lP
neurons for pre-processing and big science analysis. Here GeoDIP was applied on precipitation
nowcasting by 3 datasets that ensured various information for precipitation prediction. The
experimental outcomes proved the integrated data from 3 datasets enhanced precipitation
prediction against time. This proved that the integration of data for DL for geoscience
application promotes the use of various datasets to verify natural characteristics by HPC cluster
parallel process. Finally it was proved that the GeoDIP model was encouraging model for DL
applications.
rna

Li et al. [24] proposed collaboratively boosting framework (CBF) model for combining
knowledge guided and data driven DL in an iterative manner. This model used a deep semantic
segmentation network (DSSN) and collected the original image as input of DSSN. Further, the
module of ontology was comprises of intra and extra taxonomy functions. Intra taxonomy was
used for classification results and extra taxonomy was used for generating the inferred
channels. The combination of referred channel and original image were obtained better results
and also improved the interpretability using knowledge reasoning.
Wang et al. [25] presented an ontology model for integrating RS imagery, in-situ
observations and image products. It was created using W3C Semantic Sensor Network (SNN)
Jou

with spatial, thematic and temporal rule. The brief explanation about the ontology construction
process and rule establishment were evaluated. The actual RS images, in-situ observations,
image products and sematic queries like SPARQL and DL were evaluated. Furthermore, many
rules were done using the Semantic Web Rule Language (SWRL) on the basis of existing
knowledge and experience. This model was beneficial for the professional knowledge sharing
and achieved the better performance. Table 1 presents the advantages and challenges of the
existing models.
 Journal Pre-proof

Table 1: Advantages and challenges of the existing models

Authors Methods Advantages Performance Drawbacks
and Achieved
citation
Kyzirakos GeoTriples
More effective on The processing This model does not

of
et al. [21] small datasets and time was produce better results
the time required decreased by on large datasets.
for the entire 86%
process is less
Sun et al. GeoDataOnt Extracted and - Does not provide

pro
[22] annotated sematic detailed information
information of the about geospatial data,
geospatial data it provides the
information only about
data sharing and
integration.
Li et al. GeoDIP Model was simple Achieved better produce poor results
[23] and reliable precision and F- when all the GOES-17
re- core of 0.795 variables were used
and 0.741
Li et al. CBF Enhanced the Achieved better failed to achieve
[24] interpretability mIOU value of autonomous learning
using knowledge 0.7098
reasoning
Wang et al. SNN Established the - Semantic barriers were
[25] feasibility and developed in the
lP
rationality of the multisource data
ontology application.
framwwork

Though, the DL models achieve better results in ontology, these models takes more time
for training and completing the process. Recently, the optimization has played a role in solving
engineering problems [31-32]. These algorithms improve the convergence speed and are easy
rna

to implement. Further, it reduces the processing time by optimizing the layers of the network.
Hence, in this work, the metaheuristic algorithm Search and Rescue (SAR) optimization is used
for optimizing the structure and parameters like batch size and unit. Hence, the performance of
the entire system is improved and achieves better results.

3. Proposed Methodology
The aim of the proposed scheme is to provide an ontology based RS imaging representation in
geospatial data. Ontology is a machine readable format and it is formal and used for expressing
the types of attributes, entities and relationship in a domain. In our work we use OWL since it
Jou

is built on RDF. The proposed model consists of three segments like ontology reasoning
attention Bi-LSTM and weight selection by SAR. In this ontological reasoning plays a major
role in classification to objectify its error that has misclassified region conceptualization on the
basis of physical properties and to arrange them with suitable ontology to provide the best
solution.
 Journal Pre-proof

3.1 Formalization of Semantic Information

The Semantic formalization is a general research problem for several techniques like semantic
integration, mapping, knowledge discovery and semantic search. Though the geographic space
is regarded as a general platform for interconnecting various conceptualizations and vagueness
affect semantic heterogeneities that need to be determined. Ontology models have been

of
considered as important to develop geospatial semantics, determine semantics heterogeneity,
ground conceptualization and integrate various semantic descriptions. Ontology is categorized
based on the formality level and generality level.
In formality level, the ranges of ontologies varies from informal to semiformal and formal.
In generality, there are four kind of ontologies are used. They are top-level, task, applications

pro
and domain technologies. The top-level ontology is states the basic concepts like entity,
relation, time, space and property. The domain describes the concept relevant to the
oceanography, natural disasters and meteorology. The task and domain ontologies may perform
the facts described by top-level ontology. Finally the application ontologies describe
conceptualization based on task and domain.
The major benefits provided by ontology on RS applications are based on the following
logics.
 Symbol Grounding. The relationship of the appropriate concept to the appropriate
sensor data and the relationship between the concepts themselves. The description
re-
logics (DLs) ontologies define a formalism to provide high level indications of low
level data.
 Knowledge sharing. The general conceptualization like semantics and vocabulary and
the selection of better ontology language offers a model to publish representation of RS
images therefore they can be reused and shared between intelligent agents.
 Reasoning. The use of DLs ontology indication permits the use of DLs reasoners that
can gather new information from clear descriptions. This provides flexibility and
lP
freedom when using new concepts since new information can be classified
automatically.
To be more coherent, the geospatial ontology must have the following characteristics:
 It should be grounded using proper and meaningful primitives.
 It should be process oriented and not like static architecture.
 It should consider time and space for application and foundational features of
ontologies due to correlation.
rna

 It should combine both realistic semantics and cognitive semantics

 Permits personalization because user’s data required to personal and situational factors.
 It should consider conceptual mapping hence the model can use the semantics thereby
improves the human computer interaction.

3.2 Bi-LSTM based classification module

Here, the ontological reasoning play a major role and is developed to improve the DL
performance. Figure 1 shows the framework of the implemented model. The model of Bi-
LSTM has input, embedding, Bi-LSTM, attention and output layer. Here the attention layer is
Jou

used for avoiding overfitting problems. Finally in output, classification is performed by

softmax function.
Bi-LSTM has two independent LSTMs, which obtain words annotations by adding the
information from both direction and hybrid the sentimental data in the annotation. At every
time, the forward LSTM is the hidden state by the previous hidden state and input vector, and
backward LSTM is the hidden state by opposite hidden state and input vector. The structure of
Bi-LSTM is explained as follows:
 Journal Pre-proof

Ontological
reasoning

Embedding layer
Bi-
LSTM

of
Input layer Output layer
Attention
layer

Weight
selection by
SAR

pro
……………..

……………..
……………..

…….
+

…….

……………..
……………..
…….

Figure 1: Framework of the implemented model

Input layer has the input sequence is represented as Ci  c1 , c2 , ....., cn  where n is the
re-
length. Then the input sequence is transformed into the input vector and it is represented as
si  u1 , u 2 , ....., u z  .
For si  u1 , u 2 , ....., u z  , the embedding vector we obtained is V  v1 , v2 , ....., v z . The
representation of vector vk  R of every word u k  R (k  1,2...........z ) and it is expressed as
n 1
lP
vk  Re LU (u c u k  bc ) (1)
Where n is an embedding dimension size, uc and bc are the weight and bias vector.
Bi-LSTM process the data both from previous and future for improving the prediction
performance. Bi-LSTM has forward and backward LSTM. For Embedding V  v1 , v2 , ....., v z 

of request Re , the LSTM in forward direction f o read the input from v1 to vz and computes
rna

   
the forward hidden sequence (h1 , h2 ,...... hz )( hi  R  ) . Similarly the LSTM in backward

direction f o read the input from v1 to vz and computes the backward hidden sequence
   
(h1 , h2 , ...... hz )( hi  R  ) . Therefore the final vector is stated as

 
h j  [h j ; h j ]Q ( h j  R 2  ) (2)
 
Where p is the hidden states dimensionality, hi and hi are forward and backward hidden
Jou

state.
Attention Layer is used for capturing the relevant information, because location based
attention process does not consider the relationship among the past and present hidden states,
it capture only information about individual hidden state. To use the information from both
past and present hidden states, attention mechanism is introduced and it is expressed as:
 zj  vS tanh (W [hz ; hi ]) (3)
 Journal Pre-proof

 z  soft max ([ z1 ,  z .......... z ( z 1) ]) (4)

S
Where v and W are the parameter to be learned, tanh is a activation function and  z is the
weight of attention

of
In Output layer, the dropout regularization damages some part of attention randomly so
that dropout h  is added to softmax after the attention vector
  soft max (u s h   bs ) (5)

x  arg max ( ) (6)

pro

Where us and bs are the parameters to be learned and x is the label which is predicted
using attention mechanism. Further the weight of Bi-LSTM is updated by SAR.
SAR [26] algorithm is used in the deep attention based Bi-LSTM for the computation of
weight. The major aim of this work is to improve the system’s accuracy. The SAR process is
influenced by the explorations evaluated by humans. Many animals and human being use
various methods for searching. Rescue is a method for retrieving people in distress and send
them to safe place. Humans search in group have group members and every searching group
manages their operations. The steps in weight updation for Bi-LSTM are given as follows:
re-
In the arbitrary way, the Bi-LSTM weights are assigned and it is given as:
 
X  X 1 , X 2 , ....., X g , ....., X  ; 1  u   (7)
Where,  is the entire weights.
The weight X and the selected features T are fed to the model for the estimation of output.
For network training, the outcome of training label and the quantity of squares of the current
outcome of the system are used to calculate the output error. It is expressed as,
 Ex 
DS
lP
1
Error e1  e
z  X ze (8)
DS z 1
e e
Where, X z is the predicted output, DS is the total number of data sample sand Ex z is the
expected output at current position.
According to the social and individual stages, the members of the group will search for all
'
iteration. When the objective function value in position ( f ( Z i ) is higher than prior one
rna

( f (Z i )) , then the prior position (Z i ) will be saved in the memory matrix (M) random position
by equation (9) and this position is considered as new position by Equation (10). The position
will be left otherwise the memory will not be updated.
Z i , if f ( Z i' )  f ( Z i ) (9)
M n 
M n , otherwise
Z , if f ( Z i' )  f ( Z i )
'
Xi   i (10)
Z i , otherwise
'
Here, Z i and Z i are the old and new position of i th human, M n is the n th position saved clue
Jou

in a memory matrix and the random value is indicated as n and it is between 1 to N. This
memory updation improves an algorithm diversity and algorithm ability for finding optimal
solution.
In SAR, time is regarded as necessary parameter because those people who may be affected
and delaying the SAR may leads to deaths. Hence, these processes should be done so that the
longest space is round in less time. Hence, if member in group can’t identify more remarkable
clues, when some number of searches around their present position and they leaves the present
 Journal Pre-proof

position and moves to a new location. For modelling this characteristics, the USN (unsuccessful
search number) is considered to be zero.
If a human identify most important clues in the initial or next stage of the search, the USN
is considered to be zero for that human or else, it will be increased by one and it is expressed
as:

of
USNi  1, if f ( Z i' )  f ( Z i )
USNi   (11)
0, otherwise
th
In this, USNi is the number of times, the i human does not have the ability to find the
most important clues. In the search space, the person moves to an arbitrary location by next

pro
process if the USN is greater than the maximum unsuccessful search number (MU) for a human
and USNi is set to 0 for that human being:
Z i , j  Z min
j 
 r4  Z max
j  Z min
j , j  1,...., DS (12)

max min
Where Z j and Z j are the maximum and minimum threshold values for j dimension and
the random number is denoted as r4 which ranges from [0, 1]. The random number will be
changed for all dimensions. By Equation (12), the weight is given to the system and the optimal
re-
solution is achieved by less error. Finally, the weight value is updated using Equation (12) in
Bi-LSTM structure on the basis of SAR optimization. The optimal weights are computed in an
iterative manner, until the maximum iteration is determined.

3.3 RSOntology (RSO)

RSOntology is utilized for describing the objects attributes and relationship among them. To
evaluate ontology, OWL (Ontology Web Language) is used for expressing RSO and it is a
formal language.
lP
rna
Jou

Figure 2: Illustration of RSO

Figure 2 represents the illustration of RSO. The major part of ontology is (oc: Geobject)
which has segment (oc: segment) vegetation (oc: vegetation), nonvegetation (oc:
nonvegetation), road (oc: pavement), Airplane (oc: Airplane), ship (oc: ship), building (oc:
Building), bare land (oc: ground), water (oc: water) and vehicle (oc: car). The segment
comprises of classified data like segment class (oc: classifiedsegment) and misclassified class
 Journal Pre-proof

as (oc: misclassifiedsegment). The object class is defined as (oc: geoclass). In this, oc: denotes
the attribute of subordinate. The spatial attribute relation has adjacent (geo: adjacentTo),
direction (geo: hasDirectionOf) and surround (geo: surroundeBy). The statistical characteristics
is given by geo: MaxClass.

of
3.4 Ontology Reasoning
Ontology Reasoning utilizes Descriptive Logic (DL) for symbolizing the rule in ontological
reasoning. The instance like oe: entity and oe: entity1 are classified correctly and it satisfy Eqn.
(13) and (14), then oe: misEntity satisfy Eqn. (15).
oc: Classifiedsegment(oe: entity)

pro
oc: Classifiedsegment(oe: entity1)
oc: MisClassifiedSegment (oe: misEntity)

3.5 Feedback from the reasoning to the classification

In this research, the reasoning will give the feedback to the classification by subsidiary data
that is added to the training set. The subsidiary data is indicated as the input image that
represents the image with the similar size as the input image. In our work, we selected the
shadow and elevation.
Shadow re-
The first channel shows the shadow occurrence, which is one of the major reasons for
misclassification. To describe the idea of shadow to the classifier, initially we have to localize
on RSontology. The values in the channel are -1 for no shadow, 1 for shadow and 0 for no
opinion.
Elevation
One more model that affect the classification and leads to misclassification is elevation. The
elevation variation of region is one of the major parameters in the shadow. The value of
lP
elevation with spatial and type relation of region, the reasoning can localize the shadow as
misclassified region with least elevation value based on the regions values intersect with
misclassification part. The value used in the channel are -1 for uncertain, 0 for low height, 1
for medium height and 2 for high height.

4. Results and discussion

The performance analysis of the developed ontology module is discussed in this section. The
rna

entire implementations have been processed in a system with 8 GB RAM and Intel Core i5
CPU with 3.0 GHz speed. To implement the proposed scheme, MATLAB 2020a is utilized.
The developed model’s performance is compared with the methods like CBF [24], Semantic
referee [27] and U-Net [28].

4.1 Dataset Details

The dataset used in this work is the UCM dataset [29] and it is a benchmark dataset. The UCM
database has 21 classes and consists of 100 images for every class of size 256x256. The image
ground resolution is about 0.3m. The sample set utilizes the heavily labelled DLRSD database
Jou

[30] from UCM and has seventeen classes. To minimize the similarity among classes, this
research work uses eight classes like vegetables (trees and grass), Pavement (pavement and
dock), water (water and sea), Ship (ship), Car (car), Building (building, tank and mobile home)
and Ground (sand, chaparral and bare soil) and eliminates images have fields and tennis court.
All groups are the original group combination in parentheses. The images are grouped into
training (80%), validation (10%) and testing (10%) sets randomly.
 Journal Pre-proof

4.2 Performance evaluation

To calculate the efficiency of the developed scheme quantitatively, there are some metrics
commonly employed. In this work, two metrics OA and mIOU are considered for evaluating
the performance. The developed system classification and effectiveness is evaluated using the
performance measures based on four key results for testing the classifier like True positive (T p )

of
, false positive ( Fp ) , True negative (Tn ) and False negative ( Fn ) .The description of all metrics
with the formula is described below.
Accuracy
Overall system’s efficiency is determined by the accuracy measure. For the proposed

pro
technique, accuracy is the ratio of exactly identified pixels
T p  Tn
OA  (13)
T p  Tn  F p  Fn
mIOU
It is a basic calculation metric used for segmentation of semantic image, which initially finds
the IOU for every semantic class and then finds the mean over classes.
1 T p  Tn
mIOU  1
N
(14)
re- N T p  Tn  Fp  Fn
Where N is the number of samples.

FWmIOU
It is also used for segmentation of semantic image, which is product of OA and Frequency
weighted mIOU (FWmIOU) for every semantic class and then finds the mean over classes.
1 n T p  Tn
FWmIOU  1 mIOU  (15)
lP
n T p  Tn  F p  Fn

4.3 Performance of the proposed model

The qualitative performance of the proposed model on UCM dataset is given in this section. In
this figure, the shadow and elevation estimation are presented.
rna
Jou
 Journal Pre-proof

of
(a) (b) (c) (d) (e)

pro
Figure 3: Classification results on UCM dataset (a) input image (b) ground truth image (c)
iteration (d) shadow (e) elevation

Figure 3 delineates the classification results on UCM dataset. It has input image, ground
truth image, iteration, shadow and elevation. Green colour shows the vegetation, yellow colour
is the car, pink colour shows the ship, blue shows the water, black colour represents the
building, mild grey shows the airplane and dark grey shows the pavement. Further light grey
colour is elevation, white colour represents the shadow and black colour represents the not
shadow (White, grey, black denoted high height, medium height, low height). This figure
re-
shows that the elevation and shadow results are accurate. When the number of iteration
improves, the results of classification is also accurate for shadow and elevation. Thus the hybrid
classifier and reasoned achieves the better results.

4.4 Comparison of various approaches

To prove the proposed model’s effectiveness, the performance of the proposed model is
compared with U-Net [28] and Semantic referee [27] and it is depicted in the Figure 3 to Figure
lP
6.
rna
Jou
 Journal Pre-proof

of
pro
(a) (b) (c) (d) (e)

Figure 4: Classification outcome of existing and developed model on UCM dataset (a) input
image (b) ground truth image (c) U-Net (d) Semantic referee (e) proposed

Figure 4 delineates the Classification outcome of existing and developed model on UCM
dataset. It includes input image, ground truth image and the results of U-Net, Semantic referee
and proposed model. Here the classification of the developed model is better than the other two
existing models. Thus the proposed model have shown its betterment in the visual
re-
classification.
lP
rna

Figure 5: Accuracy comparison of various models

Figure 5 shows the accuracy comparison of various models like CBF, semantic referee and
the proposed model. Here, the iteration is taken from 1 to 5. As shown in figure the performance
varies for every iteration. For iteration 1, the OA is about 0.814 for semantic referee, 0.832 for
CBF and 0.865 for the proposed DL model. Similarly for iteration 2, the OA is about 0.814 for
semantic referee, 0.847 for CBF and 0.871 for the proposed DL model. Like this for all iteration
Jou

the implemented model achieves higher OA due to the optimal weight selection by SAR.
 Journal Pre-proof

of
pro
Figure 6: Classification comparison of various models on OA

Figure 6 delineates the classification comparison of various models on OA evaluated on the

UCM database. In this comparison, the OA values achieved by the existing U-Net, Semantic
referee, CBF and the proposed DL are 0.802, 0.82, 0.85 and 0.923 respectively. Here the DL
re-
based model CBF and the proposed model achieve higher OA. Therefore, it is proved that the
ontological reasoning will be useful for improving the performance of classification.
lP
rna

(a) (b)
Figure 7: Classification comparison of various model on (a) mIOU (b) FWmIOU

Figure 7 delineates Classification comparison of various model on mIOU and FWmIOU

evaluated on the UCM dataset. In this, the proposed model. The mIOU of the U-Net is 0.66,
Semantic referee is 0.67, CBF is 0.709 and the proposed model is 0.825. Here, the developed
model achieves higher mIOU due to the optimal value selection. Further, FWmIOU value of
Jou

the U-Net is 0.68, Semantic referee is 0.710, CBF is 0.75 and the proposed model is 0.857.
From the figure it is observed that proposed model attained better mIOU and FWmIOU values
which show the effectiveness of the system. Hence it is proved that the ontological reasoning
with optimal value by SAR can be useful for increasing the performance of classification. Thus
the proposed model have proved its superiority than the other models.
 Journal Pre-proof

Table 2: Confusion matrix on UCM dataset

% Predicted value
Vegeta Non- Pavem Buildi water Airplane car Ship
tion Vegetatio ent ng
A n

of
c 80.1(- 7.6 3.1(+0. 6.1 0.5 (-0.4) 0.0 (+0.5) 1.7 (- 0.0
t 4.3) (++4.1) 0) (+1.2) 1.2) (+0.1
u )
a 15.6 (- 79.1 5.1 4.5 (- 0.3 (-0.0) 0.0 (+0.1) 0.1 (- 0.2
l 5.2) (+6.2) (+0.2) 1.1) 0.4) (+0.1

pro
)
v 4.2 (- 4.1 (+3.4) 78.6 (- 4.1 0.3 (-0.0) 0.3 (+3.1) 4.13 (- 1.3 (-
a 1.8) 0.4) (+0.9) 4.1) 1.6)
l 5.2(- 1.6 (+2.6) 5.4 85.1 0.3 (+0.4) 0.1 (+0.8) 1.1 (- 0.0
u 2.1) (+2.1) (3.1) 1.4) (0.0)
e 5.6 (- 3.7 (+1.5) 0.6 (- 0.1 89. 4 (- 0.0 (+7.1) 0.0 2.1 (-
0.5) 0.5) (+0.3) 6.2) (+0.1) 1.2)
0.0 0.4 (+0.3) 9.2 4.2 (- 0.0 (+0.6) 75.1 (- 9.1 (- 0.0
(+0.0 (+12.7) 1.2) re- 8.4) 6.1) (+0.2
)
1.1(+0. 0.2(+0.3) 2.1 1.7 0.0 (+0.1) 0.2 92.4 (- 0.0(
5) (+0.5) (+1.2) (+21.3) 14.8) +2.1)
0.0(+0. 0.0(+0.0) 0.1 0.0 1.0 (-0.4) 0.1 (+7.2) 0.0 98.3
0) (+0.2) (+0.2) (+0.2) (-8.1)

Table 2 delineates the Confusion matrix on UCM dataset. The overall accuracy in
lP
classification of initial and last rounds are given. The accuracy is improved for all the
parameters considered. The highest confusion without the use of reasoning is among vegetation
and non-vegetation; pavement and building; water is misclassified as airplanes; pavement is
misclassified as cars and airplanes. By using the reasoning there is a high improvements in the
minimization of confusion for vehicles like ship, car and airplane. The confusion on building
and pavement is lightly reduced. Similarly for all the parameters, the confusion is small due to
the ontology reasoning.
rna

5. Conclusion
This paper introduces a methodology for ontology based geospatial data integration by a DL
techniques. By using RSontology reasoning and deep attention based bidirectional search and
rescue LSTM, the model minimizes the quantity of misclassification. This spatial ontology
which reduces the cost of modelling. The overall implementation of the developed scheme is
evaluated on UCM dataset. The performance of the implemented scheme is compared against
the existing models by OA and mIOU. Finally OA and mIOU of the system is found to be
0.923 and 0.825 respectively. The visual classification of elevation and shows the
Jou

improvements in classification. Thus, the implemented DL based ontology will provide the
platform to achieve a geospatial data semantic information. In future, this DL model and
optimization model can be evaluated on the other datasets like Potsdam dataset to verify the
accuracy of the system. Further, for improving the performance of ontological reasoning, the
boundary constraints will be designed.
 Journal Pre-proof

Funding: No funding is provided for the preparation of manuscript.

Conflict of Interest: Authors declare that they have no conflict of interest.
Authors Contributions: All authors read and approved the final manuscript.
Data Availability Statement: Data sharing is not applicable to this article.

of
References
[1] Reichstein M., Camps-Valls G., Stevens B., Jung M., Denzler J., Carvalhais N., et al. Deep
learning and process understanding for data-driven Earth systemscience. Nature 566(7743)
(2019), pp. 195–204.
[2] Li K., Wan G., Cheng G., Meng L., and Han J. Object detection in optical remote sensing

pro
images: A survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote
Sensing 159 (2020), pp. 296-307.
[3] Marcos D., Volpi M., Kellenberger B., and Tuia D. Land cover mapping at very high
resolution with rotation equivariant CNNs: Towards small yet accurate models. ISPRS
journal of photogrammetry and remote sensing 145 (2018), pp. 96-107.
[4] Maass W., Parsons J., Purao S., Storey V.C., and Woo C. Data-driven meets theory-driven
research in the era of big data: opportunities and challenges for information systems
research. Journal of the Association for Information Systems 19(12) (2018), pp. 1.
[5] Bajaj G., Agarwal R., Singh P., Georgantas N., and Issarny V. A study of existing
re-
Ontologies in the IoT-domain (2017) arXiv preprint arXiv: 1707.00112
[6] Him J.S.S. A brokering approach to federating spatial data in a semantic web environment
(Doctoral dissertation, Curtin University) (2018).
[7] Haller A., Janowicz K., Cox S.J., Lefrançois M., Taylor K., Le Phuoc D., Lieberman J.,
García-Castro R., Atkinson R., and Stadler C. The modular SSN ontology: A joint W3C
and OGC standard specifying the semantics of sensors, observations, sampling, and
actuation, Semantic Web 10(1) (2019), pp. 9-32.
lP
[8] Li Y., Chao T., Yihua T., Ke S., Jinwen T. Unsupervised multilayer feature learning for
satellite image scene classification. IEEE Geoscience and Remote Sensing Letters 13(2)
(2016), pp. 157-161.
[9] Zhang Y., Ye M., Gan Y., and Zhang W. Knowledge based domain adaptation for semantic
segmentation. Knowledge-Based Systems 193, (2020), pp. 105444.
[10] Zhu X.X., Tuia D., Mou L., Xia G., Zhang L., Xu F., and Fraundorfer F. Deep learning in
remote sensing: a review (2017) CoRR abs/1710.03959.
rna

[11] Haoyuan H., Junzhi L., A-Xing Z. Modeling landslide susceptibility using Log it Boost
alternating decision trees and forest by penalizing attributes with the bagging ensemble,
Science of the Total Environment. (2020), pp. 718.
[12] Wang S., Zhang S., Wu T., Duan Y., Zhou L., and Lei H. FMDBN: A first-order Markov
dynamic Bayesian network classifier with continuous attributes. Knowledge-Based
Systems 195(11) (2020), pp. 105638.
[13] Shijin L., Hao W., Dingsheng W., Jiali Z. An effective feature selection method for
hyperspectral image classification based on genetic algorithm and support vector machine.
Knowledge-Based Systems 24(1) (2011), pp. 40-48.
Jou

[14] Li Y., Zhang Y., Huang X., Zhu H., and Ma J. Large-scale remote sensing image retrieval
by deep hashing neural networks. IEEE Transactions on Geoscience and Remote Sensing.
56(2) (2018), pp. 950-965.
[15] Li Y., Zhang Y., Huang X., and Ma J. Learning source-invariant deep hashing
convolutional neural networks for cross-source remote sensing image retrieval. IEEE
Transactions on Geoscience and Remote Sensing 56(11) (2018), pp. 6521-6536.
[16] Li Y., Zhang Y., and Zhu Z. Error-tolerant deep learning for remote sensing image scene
classification. IEEE Transactions on Cybernetics In press (2020).
 Journal Pre-proof

[17] Carranza-García M., García-Gutiérrez J., Riquelme J.C. A framework for evaluating land
use and land cover classification using convolutional neural networks. Remote Sens 11(3)
(2019), pp. 274.
[18] Rajendran G.B., Kumarasamy U.M., Zarro C., Divakarachari P.B., and Ullo S.L. Land-
use and land-cover classification using a human group-based particle swarm optimization

of
algorithm with an LSTM Classifier on hybrid pre-processing remote-sensing images.
Remote Sensing 12(24) (2020), pp. 4135.
[19] Lin Z., and Zhang G. Genetic algorithm-based parameter optimization for EO-1 Hyperion
remote sensing image classification. European Journal of Remote Sensing 53(1) (2020), pp.
124-131.

pro
[20] Neagoe V.E., and Neghina C.E. An artificial bee colony approach for classification of
remote sensing imagery. In 2018 10th International Conference on Electronics, Computers
and Artificial Intelligence (ECAI) IEEE (2018, June), pp. 1-4.
[21] Kyzirakos K., Savva D., Vlachopoulos I., Vasileiou A., Karalis N., Koubarakis M., and
Manegold S. GeoTriples: Transforming geospatial data into RDF graphs using R2RML and
RML mappings. Journal of Web Semantics 52 (2018), pp. 16-32.
[22] Sun K., Zhu Y., Pan P., Hou Z., Wang D., Li W., and Song J. Geospatial data ontology:
the semantic foundation of geospatial data integration and sharing. Big Earth Data. 3(3)
(2019), pp. 269-296. re-
[23] Li G., and Choi Y. HPC cluster-based user-defined data integration platform for deep
learning in geoscience applications. Computers & Geosciences (2021), pp. 104868.
[24] Li Y., Ouyang S., and Zhang Y. Collaboratively boosting data-driven deep learning and
knowledge-guided ontological reasoning for semantic segmentation of remote sensing
imagery, (2020) arXiv preprint arXiv: 2010.02451.
[25] Wang C., Zhuo X., Li P., Chen N., Wang W., and Chen Z. An Ontology-Based Framework
for Integrating Remote Sensing Imagery, Image Products, and In Situ Observations. Journal
lP
of Sensors (2020).
[26] Shabani A., Asgarian B., Salido M., and Gharebaghi S.A. Search and rescue optimization
algorithm: A new optimization method for solving constrained engineering optimization
problems. Expert Systems with Applications 161 (2020), pp. 113698.
[27] Alirezaie M., Längkvist M., and Sioutis M. Semantic referee: A neural-symbolic
framework for enhancing geospatial semantic segmentation. Semantic Web 10(5) (2020),
pp: 863-880.
rna

[28] Olaf R., Philipp F., and Thomas B. U-Net: Convolutional Networks for Biomedical Image
Segmentation. In: International Conference on Medical Image Computing and Computer-
Assisted Intervention, Springer International Publishing (2015).
[29] Yang Y., and Newsam S. Bag-of-visual-words and spatial extensions for land-use
classification, in: The 18thSIGSPATIAL International Conference on Advances in
Geographic Information Systems (2010).
[30] Shao Z., Yang K., and Zhou W. Performance evaluation of single-label and multi-label
remote sensing image retrieval using a dense labeling dataset. Remote Sensing 10(6)
(2018), pp. 964.
Jou

[31] Banchhor C., and Srinivasu N. Integrating Cuckoo search-Grey wolf optimization and
Correlative Naive Bayes classifier with Map Reduce model for big data classification. Data
& Knowledge Engineering, 127 (2020), pp. 101788.
[32] Xiao Z., and Ming Z. A state based energy optimization framework for dynamic virtual
machine placement. Data & Knowledge Engineering, 120 (2019), pp. 83-99.
 Journal Pre-proof

of
pro
re-
lP
rna
Jou
 Journal Pre-proof

Author's Biography

of
Corresponding author: Dr. S Palaniappan, PhD, working as an Associate
Professor in the Department of Artificial Intelligence and Data Science at
KCG College of Technology, Chennai. Graduated as Engineer in the year
2000 and done my masters in the year 2002 and completed my doctorate from
JNTU Hyderabad in the year 2017. Has a total of 19 years of teaching experience at various

pro
institutes across India and 1 year of research experience at NUS, Singapore. Authored or co-
authored over 21 International Journal publications in the field of Data Science, Network
Security and Software Engineering.

Co-author 1: Dr. D. Yuvaraj is a Doctorate degree holder in the field of

Computer Science and Engineering and he has completed his Bachelor
re-
degree (B.E) in Computer Science and Engineering from Bharathidasan
University, Trichy, and Tamilnadu, India in 2001. He was awarded Master
degree in Computer and Information Technology (M.Tech) from Manonmanium Sundaranar
University, Tirunelveli, and Tamilnadu, India in 2004. He has completed a PhD in
Information and Communication Engineering from Anna University, Chennai, India in 2017.
He has been serving for 21 years of experience in the field of teaching at the university level
lP
in India and abroad. Presently, he is working as an Assistant Professor in the Department of
Computer Science at Cihan University- Duhok, Kurdistan Region, Iraq. He is specialised in
the field of Image Processing, Data Mining, Information Retrieval and Artificial Intelligence.
He has published 60 papers in international journals and conference indexed by SCI, Web of
Science and Scopus.
rna

Co-author 2: P.Padmakumari is a Assistant Professor of Computer

Science and Engineering at School of Computing, SASTRA Deemed
University, Thanjavur, Tamilnadu, India. She received her Bachelor’s
degree in Science from Madurai Kamaraj University, Madurai, Tamil
Nadu, India. She obtained her Master’s degree in Computer Applications
from Anna University, Chennai, Tamil Nadu, India. She also received her Master’s degree in
Jou

Computer Science and Engineering from SASTRA Deemed University, Thanjavur, Tamil
Nadu, India. She received her Ph.D degree from SASTRA Deemed University, Thanjavur,
Tamil Nadu, India. She has 13 years of teaching experience and her research focuses on
scientific workflows on heterogeneous distributed systems, Machine learning, cloud
computing, fault tolerance. She has been contributing papers for many high-quality
technology
 Journal Pre-proof

of
Co-author 3: Dr.Subbiah Swaminathan currently working as a professor,
department of computer science and engineering,Saveetha school of
engineering, Saveetha institute of medical and technical sciences, Chennai.
He Completed doctorate in computer science and engineering in the area of cloud computing
security issues. Previously worked as principal in various affiliated engineering colleges.

pro
His area of research includes cloud computing, data security, computer network and data
mining. He is a life member of ISTE and CSI

re-
lP
rna
Jou
 Journal Pre-proof

Author Statement

We declare that this manuscript is original, has not been published before and is not currently

of
being considered for publication elsewhere. We confirm that the manuscript has been read
and approved by all named authors.

pro
re-
lP
rna
Jou
 Journal Pre-proof

Conflict of Interest:
Authors *1Dr.Palaniappan Sambandam, 2Dr.D.Yuvaraj, 3Dr.P.Padmakumari, 4Dr. Subbiah
Swaminathan, declares that they have no conflict of interest. Patients’ rights and animal
protection statements: This research article does not contain any studies with human or
animal subjects.

of
pro
re-
lP
rna
Jou