This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception of pagination.
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING
FusionNet: Edge Aware Deep Convolutional
Networks for Semantic Segmentation of Remote
Sensing Harbor Images
Dongcai Cheng, Gaofeng Meng, Member, IEEE, Shiming Xiang, Member, IEEE, and Chunhong Pan
Abstract—Sea–land segmentation and ship detection are two
prevalent research domains for optical remote sensing harbor im-
ages and can find many applications in harbor supervision and
management. As the spatial resolution of imaging technology im-
proves, traditional methods struggle to perform well due to the
complicated appearance and background distributions. In this pa-
per, we unify the above two tasks into a single framework and apply
the deep convolutional neural networks to predict pixelwise label
for an input. Specifically, an edge aware convolutional network is
proposed to parse a remote sensing harbor image into three typical
objects, e.g., sea, land, and ship. Two innovations are made on top
of the deep structure. First, we design a multitask model by simul-
taneously training the segmentation and edge detection networks.
Hierarchical semantic features from the segmentation network are
extracted to learn the edge network. Second, the outputs of edge
pipeline are further employed to refine entire model by adding an
edge aware regularization, which helps our method to yield very
desirable results that are spatially consistent and well boundary
located. It also benefits the segmentation of docked ships that are
quite challenging for many previous methods. Experimental results
on two datasets collected from Google Earth have demonstrated the
effectiveness of our approach both in quantitative and qualitative
performance compared with state-of-the-art methods.
Index Terms—Edge aware regularization, harbor images,
multitask learning, semantic segmentation.
I. INTRODUCTION
EA–LAND segmentation and ship detection are two funda-
S mental and vital topics in the research field of remote sens-
ing images. As for sea–land segmentation, it aims to separate the
sea area from land exactly. The accuracy of segmentation results
is of great importance to coastline navigation, resource manage-
ment, and protection [1]–[4]. It has also been commonly used
as a preprocessing step for ship detection [5], [6]. The boundary
Manuscript received May 12, 2017; revised July 24, 2017; accepted August
22, 2017. This work was supported in part by the National 863 projects under
Grant 2015AA042307, in part by the National Natural Science Foundation of
China under Grant 91338202, Grant 91438105, Grant 91646207, and Grant
61370039, and in part by the Beijing Natural Science Foundation under Grant
4162064. (Corresponding author: Shiming Xiang.)
The authors are with the National Laboratory of Pattern Recognition, Institute
of Automation, Chinese Academy of Sciences, Beijing 100190, China (e-mail:
dccheng@nlpr.ia.ac.cn; gfmeng@nlpr.ia.ac.cn; smxiang@nlpr.ia.ac.cn; chpan
@nlpr.ia.ac.cn).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/JSTARS.2017.2747599
1939-1404 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
1
accuracy of segmentation results will directly influence the lo-
cation of region of interest (ROI) for docked ships. As for ship
detection and classification, it plays an important role in mar-
itime security and other applications such as traffic surveillance
and sea pollution monitoring [7]. Further work on top of ship
detection is the ship classification. However, limited literature
has been found on it due to the lack of adequate remote sensing
images for model learning.
The difficulties of harbor image processing mainly lie in three
aspects. First, the complicated texture distribution in land re-
gions will affect the results of both sea–land segmentation and
ship detection. For sea–land segmentation, traditional meth-
ods [6], [8] often confront the problem of misclassification in
land. Also, the ship detection methods [5], [9] will produce
false alarms within land region, which is a key problem to be
solved. Second, the intricate boundary between sea and land
may cause under segmentation. The situation is apparent for
thin and fine-structured objects such as wharfs [10], which will
also bring negative effect on inshore ship detection. Finally, the
disturbance of cloud, wave, and shadow factors will influence
results. Specifically, clouds and waves are often mistaken for
land part in sea–land segmentation and the shadow will affect
to segment a spatially consistent ship body.
The methods for sea–land segmentation differ depending on
the type of imagery used. For multispectral images, the nor-
malized difference water index (NDWI) [11] is an important
metric, which takes advantage of the fact that the reflectance of
water area is near to zero in near-infrared band and high in green
band. Most works [12]–[14] on sea–land segmentation of multi-
spectral images are based on NDWI map. The processing steps
are relative simple and can obtain accurate results. While for
panchromatic images, most methods [6], [8], [15] are based on
threshold selection, together with morphological operations to
eliminate errors in the results. With the improvement of spatial
resolution of remote sensing images, threshold-based methods
face problems when dealing with scenes with complicated tex-
ture and intensity distribution. The results may contain lots of
misclassifications both in land and sea regions.
Previous works for ship detection in optical remote sensing
images can be coarsely categorized as the methods for offshore
ship detection and for inshore ship detection. Traditional pro-
cessing flow of offshore ship detection includes ROI extraction,
feature computation, and classification [6], [7], [16]–[20]. In
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING
the stage of ROI extraction, image segmentation approaches
[16], [20] and wavelet transformation [7], [17] are frequently
used. Since most extraction steps in those methods are unsu-
pervised, which only employ low-level spectral information,
sometimes there is a high false positive rate. In the stage of fea-
ture extraction and classification, the shape feature of the ship
(such as ratio of length and width) together with some texture
features (e.g., local binary patterns [21], local multiple patterns
(LMPs) [6], and circle frequency (CF) feature [22]) is calcu-
lated and concatenated. The descriptors are then fed into some
machine learning models such as support vector machines [6],
AdaBoost [22], and extreme learning machine [7] to get final
predictions.
For inshore ship detection, the ship head detection is critical.
Many existing methods [5], [9], [23] employ the “V” shape fea-
ture of the head to identify head point. Afterward, the ship body
is located by line detection and some morphological operations.
The processing steps of above methods are usually complicated.
Moreover, the parameters are often empirically chosen and make
the results sensitive to them.
During last several years, the deep convolutional neural net-
works (DCNNs) have experienced rapid advance in many vision
tasks such as object recognition [24]–[29], image classifica-
tion [30], [31], and semantic segmentation [32]–[39]. Recently,
many works [7], [40]–[42] have attempted to apply DCNNs to
remote sensing images. However, there is still limited appli-
cation on harbor images, which is mainly due to the lack of
adequate images to train the model well. The harbor images
have their own characteristic compared with natural images.
For instance, the direct use of region proposal network in Faster
RCNN [27] to generate ship proposals may cause problem since
the angle of ship should also be considered.
To overcome the above shortcomings in existing sea–land
segmentation and ship detection methods, we propose a novel
deep model, i.e., FusionNet, to unify those two tasks into a single
framework by tackling a three-class (land, sea, and ship) seman-
tic segmentation problem. For an input image, the prediction of
pixel-level label can simultaneously accomplish segmentation
and detection. Our network consists of two components: the
segmentation network and the edge detection network. Specif-
ically, the segmentation network produces class label for each
pixel, whereas the edge network outputs boundaries between
different classes. The edge network is constructed by extracting
hierarchical semantic features from the segmentation network
and forms multitask with it. To further achieve fusion of both
networks, we propose an edge aware regularization to encour-
age probability propagation among pixels within the same class.
The segmentation results are spatially coherent by employing
semantic affinity between pixels.
This paper is distinguished by the following contributions.
1) We introduce a deep model, i.e., FusionNet, for remote
sensing image segmentation. In this network, the tasks of
both sea–land segmentation and ship detection are unified
into a single framework. Our model can be employed to
solve both three-class (sea, land, and ship) and two-class
(sea and land) problems. To the best of our knowledge, it
is the first attempt to use semantic segmentation model to
tackle ship detection problem.
2) We design a segmentation and edge detection network to
form multitask learning. The edge network encodes each
position by extracting semantic feature hierarchies from
the segmentation network, which provides complemen-
tary cues to adjust the gradients of entire model.
3) Based on the multitask design, we further propose an edge
aware regularization to fuse the segmentation and edge
networks. The outputs of the edge network are explicitly
employed to refine prediction maps of the segmentation
network. The local smoothness will propagate among pix-
els within the same class and is controlled by semantic
edges, which helps to obtain segmentation results that are
spatially consistent and well boundary located.
The remainder of this paper is organized as follows. In
Section II, we make a brief review on previous works. The
structure of the proposed model will be described in detail in
Section III. In Section IV, we will introduce the datasets used
in this paper and conduct extensive comparative experiments
with state-of-the-art methods. Conclusions will be drawn in
Section V.
II. PREVIOUS WORKS
A. Sea–Land Segmentation
Many early works on sea–land segmentation for optical re-
mote sensing images are based on threshold selection [6], [8],
[15]. For instance, Liu and Jezek [15] proposed an approach
to determine the thresholds for local regions by fitting a bi-
model Gaussian curve. The method is suitable for images with
coarse spatial resolution, while the convergence of the fitting
process to a global optimum is not guaranteed. You and Li [8]
built the statistical model for sea area to determine the thresh-
old based on OTSU [43] results. Afterward, they rectified the
connected mislabeled regions in land results by evaluating their
variances. However, the misclassification in sea area is unable
to be solved. Recently, supervised method [10] for sea–land
segmentation has been proposed by integrating seeds learning
and graph cut [44]. Since it employs superpixels to build graph
model, the results may contain under segmentation for some
thin and fine-structured objects, such as wharfs.
B. Ship Detection
For offshore ship detection, one of the hottest research issue in
traditional methods is how to select valuable features. LMP [6]
and CF [22] are proved to be effective in distinguishing ships
from false alarms. Recently, DCNNs-based models have been
applied in ship detection. Tang et al. [7] extracted ship candi-
dates using wavelet coefficients and learned high-level features
by training two deep networks. Zou and Shi [40] trained two
sets of deep filter banks to obtain ship candidates and semantic
features respectively, and constructed the low-level filter banks
by singular value decomposition.
For inshore ship detection, the method of ship head detec-
tion is frequently used. Li et al. [9] identified the ship head by
transforming pixels within a local window into the polar coor-
dinate system, and searched the ship body from head according
to the direction of line segments. Liu et al. [5] selected ship
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING 3

Fig. 1. Architecture of our proposed model. There are two outputs: segmentation results and edge detection results. We employ the encoder–decoder structure [36]
as our segmentation network. Our edge network is built by extracting hierarchical semantic features from layers of segmentation network. During back propagation,
the edge output is used to further refine entire model from the end. The output size of layer is labeled on the top, whereas the channel numbers are labeled at the
bottom. Detailed configurations of the above two networks are listed in Tables I and II.

head by corner detection and parabola fitting, and located ship
body through shape analysis. However, both the above meth-
ods focus on detection of ships with “V” shape head. Recently,
Liu et al. [45] proposed an supervised approach to generate ship
proposals in the rotated bounding box (RBB) space. The learned
model can work with various ships and achieves high recall.
C. Semantic Segmentation
The applications of DCNNs in semantic segmentation
[32]–[34], [36], [37] have been prevalent since the appearance of
fully convolutional networks (FCNs) [32], which converts the
fully connected layers in detection structure to convolutional
layers, and implement upsample operation to obtain original-
sized prediction maps. Several models promote performance on
the basis of FCNs to obtain finer spatial layout in the results.
For instance, Chen et al. [34] increased the size of dense maps
by removing last several subsample layers and smooth the edge
results by linking a conditional random field (CRF). Studies in
[33] and [36] refined semantic features by stacking a symmetri-
cal deconvolutional and upsampling structure on top of FCNs.
Recently, advances in instance segmentation [38], [46]–[49]
have been driven by combining R-CNN-based detection model
and FCN-based segmentation model. The basic idea is to si-
multaneously address object classes, boxes, and masks. For in-
stance, DeepMask [48] learns to produce segment proposals,
which are classified by Fast R-CNN. Dai et al. [49] designed
model to obtain instance-level segment candidates by computing
position-sensitive maps. Mask R-CNN [46] achieves state-of-
the-art results by adding a branch on Faster R-CNN to generate
segment masks. Both results of detection and segmentation are
improved by multitask learning.
D. Multitask Learning
There are previous research works on training edge network
by expanding a pipeline from segmentation network. For in-
stance, semantic boundaries are defined as the linear combi-
nation of feature maps of FCNs in [50], and are employed to
construct the pairwise term in the final optimization function.
Chen et al. [51] computed edge predictions from intermediate
layers of the segmentation network, and proposed a domain
transform approach to align segmentation scores with object
boundaries. However, our method is different in two aspects.
First, we use the encoder–decoder model as our basic segmen-
tation network, and extract rich hierarchical features to build
the edge network. Compared with FCNs, the decoder network
can generate semantic meaningful feature maps with high spa-
tial resolution to maintain more finer details, which is effective
for predicting boundaries. Second, our entire model is learned
end to end. Both of the two networks are trained in parallel,
which will improve the robustness of the segmentation network
by providing with multiple regularizations.
III. STRUCTURE OF PROPOSED MODEL
In this section, we will give a description of the configuration
of our proposed model, i.e., FusionNet. First, we will provide
the structure of our segmentation network. Second, we will
describe the configuration of our edge network and how to train
it in parallel with the segmentation network. Finally, a fusion
strategy of how to employ outputs of the edge network to refine
the segmentation results is proposed.
A. Encoder–Decoder Network for Segmentation
Fig. 1 illustrates the overall architecture of our proposed
model. There are two networks in our structure, i.e., the seg-
mentation network and edge network, which will run in parallel
in the training process. The segmentation network outputs prob-
ability maps to indicate how likely the pixel belongs to each
class, and the edge network outputs probability that the pixel
belongs to boundaries between different classes.
We employ SegNet [36] as the basic structure of our segmen-
tation network for its ability of producing segments with finer
details. As presented in Fig. 1, it consists of two components:
the encoder network and decoder network. The encoder net-
work is configured in the same way as FCNs, which contains a
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING

TABLE I four times of max-pooling operation in the encoder network such

CONFIGURATION OF OUR SEGMENTATION NETWORK
that the size of feature map at the end is 19 × 19. In the decoder
network, we perform symmetrical operations, which consists
Segmentation network of four unpooling layers and corresponding convolution, BN
Encoder and ReLU layers. After the last 32-channel convolutional layer,
we add another convolutional layer with three-channel output
Name Kernel size Stride Pad Output size
to fit the deep model into our three-class problem. Finally, a
Input – – – 300 × 300 × 3 three-channel softmax layer is attached to obtain the prediction
Conv1-1 3×3 1 1 300 × 300 × 32 maps.
Conv1-2 3×3 1 1 300 × 300 × 32
Pool1 2×2 2 0 150 × 150 × 32 We employ cross entropy loss to train our segmentation net-
Conv2-1 3×3 1 1 150 × 150 × 64 work, which is defined as
Conv2-2 3×3 1 1 150 × 150 × 64
2×2 75 × 75 × 64
1 
Pool2 2 0 N K
Conv3-1 3×3 1 1 75 × 75 × 128 Lossseg (l, p, θseg ) = −σ(li = k) log pk ,i (1)
Conv3-2 3×3 1 1 75 × 75 × 128 N i=1
Conv3-3 3×3 1 1 75 × 75 × 128 k =1
Pool3 2×2 2 0 38 × 38 × 128
Conv4-1 3×3 1 1 38 × 38 × 256 where li is the ground truth label of point i, and pk ,i is the
Conv4-2 3×3 1 1 38 × 38 × 256 output probability that i belongs to kth class. K is the number
Conv4-3 3×3 1 1 38 × 38 × 256
Pool4 2×2 2 0 19 × 19 × 256
of classes and equals 3 in our segmentation problem. σ(·) is
an indicator function. It takes 1 when li = k and 0 otherwise.
Decoder
θseg is the parameter set of the segmentation network. N is the
Unpool4 2×2 2 0 38 × 38 × 256
Conv4-1-D 3×3 1 1 38 × 38 × 256 number of all pixels in the batch. Normalization over the batch
Conv4-2-D 3×3 1 1 38 × 38 × 256 is performed after total loss is calculated.
Conv4-3-D 3×3 1 1 38 × 38 × 128
Unpool3 2×2 2 0 75 × 75 × 128
Conv3-1-D 3×3 1 1 75 × 75 × 128 B. Deep Edge Network
Conv3-2-D 3×3 1 1 75 × 75 × 128
Conv3-3-D 3×3 1 1 75 × 75 × 64 The results of semantic segmentation may contain mislabel
Unpool2 2×2 2 0 150 × 150 × 64 for pixels around boundaries. Typically, the ship results often
Conv2-1-D 3×3 1 1 150 × 150 × 64
Conv2-2-D 3×3 1 1 150 × 150 × 32 have a mixture with their adjacent wharf due to the loss of
Unpool1 2×2 2 0 300 × 300 × 32 semantic details caught by pooling operations. The phenomenon
Conv1-1-D 3×3 1 1 300 × 300 × 32 is notable for objects with unclear boundaries. To this end, we
Conv1-2-D 3×3 1 1 300 × 300 × 32
Output 3×3 1 1 300 × 300 × 3 propose a multitask framework by expanding a branch from
Softmax – – – 300 × 300 × 3 the segmentation network and learn both segmentation and edge
network simultaneously. The basic ideas behind this are: first,
We use encoder–decoder architecture for its fine ability of recovering
details. By adding “-D,” we refer to layers in the decoder network. The
the training of the edge network provides complementary cues
BN and ReLU layers are omitted in the table for brevity. for better regularizing the boundary of adjacent objects; and
second, we can further employ outputs of semantic edges to
series of convolution and pooling operations to achieve feature refine segmentation results.
extraction. As the layer goes deeper, we can obtain high-level The structure of our edge network is illustrated in Fig. 1, and
features with more semantic information. However, the spatial detailed configuration is presented in Table II. We expand the
information of feature maps is lost and cannot be recovered by edge network by extracting five groups of feature maps from the
a simple interpolation to the original size of input image. The segmentation network. The lower level features contain local de-
SegNet model overcomes this problem by stacking a decoder tails, and are extracted from the encoder network. The high-level
network on top of the encoder part, which is a mirrored version features provide semantic information, and are extracted from
of the encode network and contains a series of convolutional the decoder network. Specifically, for each of the five feature
and unpooling layers to restore finer spatial layout of objects. map groups, we fist link it with an 1 × 1 convolutional layer,
Table I presents the detailed configuration of our segmen- and then upsample them to the original size of the input. The
tation network. We adopt the strategy in VGG network [31] 1 × 1 convolutional layer is employed in two purposes. First, it
that the convolution channels are augmented with the increase serves as a connection between the segmentation network and
of net depth. The input image is three-channel with size fixed edge network, which makes the entire model flexible to train
at 300 × 300. There are four groups of convolutional layers task specific while close related subnetworks. Second, it helps
(10 convolutional layers altogether) in the encoder network. For to lower the dimension of feature maps, which will reduce mem-
each convolutional layer, the kernel size is 3 × 3 and stride is ory consumption and shorten the training and inference time. In
1. Between two successive convolutional layers, batch normal- Table II, for each feature, the first layer indicates from which the
ization (BN) [52] and rectified linear units (ReLU) [53] are branch is expanded. After convolution and unpooling, the final
applied sequentially. Both BN and ReLU layers are employed features are concatenated and fed into the classifier. To achieve
to accelerate the learning process. After each group of convolu- this, we add a convolutional layer with two-channel output and
tional layers, a max-pooling layer with 2 × 2 kernel and stride a softmax layer at the end of the edge network to obtain final
2 is added to achieve translation invariance. Therefore, there are prediction maps.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING 5

TABLE II
CONFIGURATION OF OUR EDGE NETWORK

Edge network

Feature 1

Name Kernel size Stride Pad Output size

Conv2-2 3×3 1 1 150 × 150 × 64

Conv1-E 1×1 1 0 150 × 150 × 32
Unpool1-E 2×2 2 0 300 × 300 × 32
Feature 2
Name Kernel size Stride Pad Output size
Conv3-3 3×3 1 1 75 × 75 × 128
Conv2-E 1×1 1 0 75 × 75 × 32
Unpool2-1-E 2×2 2 0 150 × 150 × 32
Fig. 2. Semantic segmentation results of SegNet and our method. (a) Orig-
Unpool2-2-E 2×2 2 0 300 × 300 × 32
inal image. (b) Result of SegNet. (c) Result of FusionNet (ours). (d) Ground
Feature 3 truth. Improved regions are marked by yellow rectangles for better comparison.
Name Kernel size Stride Pad Output size Compared with SegNet, our model can obtain segmentation results with higher
spatial consistency and boundary accuracy.
Conv3-3-D 3×3 1 1 75 × 75 × 64
Conv3-E 1×1 1 0 75 × 75 × 32
Unpool3-1-E 2×2 2 0 150 × 150 × 32
Unpool3-2-E 2×2 2 0 300 × 300 × 32
end, losses of both networks are calculated and the gradients are
back propagated. In the inference phase, we can simultaneously
Feature 4
obtain segmentation results and semantic boundaries for an input
Name Kernel size Stride Pad Output size
image.
Conv2-2-D 3×3 1 1 150 × 150 × 32
Conv4-E 1×1 1 0 150 × 150 × 32
Unpool4-E 2×2 2 0 300 × 300 × 32 C. Edge Aware Network Regularization
Feature 5
The spatial resolution loss of prediction maps caused by pool-
Name Kernel size Stride Pad Output size
ing operation will lead to blobby segments that lack fine object
Conv1-2-D 3×3 1 1 300 × 300 × 32 details [50]. It results in two common problems in semantic
Conv5-E 1×1 1 0 300 × 300 × 32
Concat – – – 300 × 300 × 160 segmentation of harbor images. One is the coarse segmented
Output 3×3 1 1 300 × 300 × 2 boundaries between ship and its neighboring wharf. As pre-
Softmax – – – 300 × 300 × 2 sented in Fig. 2(b), the results often contain mixture around the
Each feature group is expanded from a convolutional layer of segmentation
boundaries due to low contrast, which will directly influence
network. The final features are concatenated and fed into a convolutional layer the ship detection accuracy. The other is the misclassification
with two-class output. By adding “-E,” we refer to layers in the edge network. in land area. It will bring difficulty in coastline extraction and
increase the ratio of false alarms of ships.
We also utilize cross entropy loss to train the edge network
In Section III-B, we have introduced a multitask learning
1  framework. The incorporation of the edge network has two ad-
N C
Lossedg (y, pe , θedg ) = −σ(yi = c) log pec,i (2) vantages: on one hand, the multitask formulation helps to opti-
N i=1 c=1
mize segmentation model; and on the other hand, we can further
where yi is the true edge label of point i, and pec,i denotes the improve performance of the whole model with outputs of the
probability that i belongs to cth class. The upper mark e indicates edge network. To this end, a regularization method to achieve
probability for the edge network. C is the number of classes and fusion of two networks is introduced in this section.
equals 2 in the edge detection problem. θedg is the parameter set Specifically, we employ outputs of the edge network to reg-
of the edge network. ularize prediction maps of the segmentation network. For each
The entire model is learned end to end. During training, these point, a local smooth loss with respect to its neighbors is defined
two networks are trained in parallel. Parameters of the segmenta- as
tion network are shared by the two networks and can be updated −p e
1  
K 1,j
1 −p e
jointly, whereas parameters of the last classification layers of Lossre,i = (pk ,i − pk ,j ) e 2 1,j (3)
the two networks as well as the expanding branch in the edge 8K
j ∈ Nb(i) k =1
network are adjusted separately.
We input the network with the original image and its ground where j is the neighbor pixel of i, Nb(i) is the eight-neighbor
truth segmentation map. To obtain the reference map for the system of i, k denotes the channel of prediction maps of the
edge network, we add another layer to automatically calculate it segmentation network and K is the number of channels, pk ,i
according to the ground truth map of the segmentation network. is the segmentation probability of i on channel k, pe1,j depicts
Specifically, for each point, if it has different label with one the probability that j belongs to edge, and the upper mark e
of its eight neighbors, we define it as true edge point. At the indicates probability from the edge network.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

6 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING

We define the above pixelwise loss in a similar way as the
pairwise term used in semisupervised methods. Actually, the
total smooth loss within an image can be simplified as
1 classes will be degraded. We set λ at 10 in this paper and find
Lossre = Tr(P T LP )
4K
where P = (p0 , p1 , p2 ) ∈ RN×3 , N is the number of total pix-
els within the image, and 3 denotes the number of classes in our
segmentation problem. Each vector pi , i = 0, 1, 2, corresponds
to the ith probability map of the segmentation network. The
probability map is converted to the vector by row. Tr(·) denotes
the trace of matrix. L is the Laplacian matrix, which is con-
structed by L = D − W , where W ∈ RN×N is the adjacency
matrix and each element of it is constructed by the edge aware
−p e
1,j
1 −p e
pairwise term wij = e . The W matrix is sparse and each
1,j
row has only eight nonzero elements, which corresponds  to the
eight neighbors of i. D is a diagonal matrix, dii = N j =1 wij .
In (3), the smaller the probability that neighbor pixel belongs
to edge, the more similar segmentation probabilities should the
two pixels have. However, the smoothing operation is faded with
the increase of the likelihood that the neighbor pixel belongs to
edge. In this way, the segmentation maps will be aligned with
boundaries between different classes, which helps to obtain finer
results with more spatial consistency.
The gradients of lossre,i with respect to pk ,i , pk ,j , and pe1,j
are calculated as
−p e
∂lossre,i 1  1,j
Parameter λ in (8) is employed to adjust the relative weight
of regularization term. If λ is too large, the segmentation results
will be over smoothed and the boundary accuracy of neighboring
(4) remarkable performance improvement brought by incorporating
the edge aware loss.
In the training process, parameters of both segmentation net-
work and edge network are updated according to their respective
cross entropy loss as well as the edge aware loss. In practice, we
first train the multitask model in parallel, then fine-tune it using
(8). Implementation details can be found in Section IV-B.
The proposed edge aware regularization has some similarity
with CRF. Actually, we can treat Lossseg in (8) as the unary
term in CRF, whereas Lossre as the pairwise term in CRF. In the
process of fine-tuning, local consistency among neighboring
pixels is encouraged and controlled by some regularization,
which also resembles the CRF. In spite of that, our method is
different with CRF in some aspects. First, the proposed model is
end to end, which makes our method flexible and efficient, while
CRF is often employed as a postprocessing step after training
in CNNs-based models. Second, the regularization term in our
method is built by semantic edges, which is more effective in
measuring semantic similarity between pixels than low-level
color-based pixel affinity function that is used in CRF.
IV. EXPERIMENTS
In this section, we will first introduce the two datasets used

= (pk ,i − pk ,j )e
1 −p e
1,j (5) in this paper, and then present both quantitative and qualitative
∂pk ,i 4K comparisons with the state-of-the-art methods.
j ∈N b(i)
−p e
1,j
∂lossre,i 1 1 −p e A. Dataset Description and Augmentation
= (pk ,j − pk ,i )e 1 , j (6)
∂pk ,j 4K We evaluate the performance of our proposed model on two

K −p e datasets. One is the dataset of harbor images, which contains
∂lossre,i −1 1,j
1 −p e
= (pk ,i − pk ,j )2 e 1,j . (7) land, sea, and ship and is utilized for the segmentation of three-
e
∂p1,j 8K(1 − pe1,j )2 class problem. The other is the sea–land dataset and is used for
k =1
the segmentation of two-class problem. Images in both datasets
From (5), we can see that when the neighbor pixel has smaller
are RGB-channel and collected from Google Earth, which is a
edge probability and when the two pixels have more different
∂ loss suitable platform for providing adequate remote sensing images
segmentation probabilities, the larger is ∂ p kr,ei , i . That means the
to train the deep model.
gradient is controlled by both outputs of semantic edges and 1) Harbor Image Dataset: It contains 380 images in this
segmentation results. When the neighbor pixel has large edge dataset with spatial resolution of 1.0 m. Among them, 300 im-
probability, the gradient is dominated by the small value of expo- ages are used for training, ten images are used for validation,
nential term such that the influence of edge aware regularization and 70 images are for testing. Most of the images have size
in back propagation is negligible. larger than 800 × 800. The numbers of ships for training, val-
We define the final loss function of our network by combining idation, and testing are 1654, 69, and 397, respectively. Most
above three loss terms of them are military ships. The segmentation ground truth of
LossFusionNet = Lossseg + Lossedg + Lossre all the images are labeled by hands, and ground truth maps of
K the edge network are automatically calculated during training.
1   
N C
Note that we have not labeled ship targets with too small size
= −σ(li = k) log pk ,i + as ship since they present unclear visual textures. Actually, we
N i=1 c=1
k =1
treat them as part of land due to their high intensities compared
− σ(yi = c) log pec,i with sea area. In this paper, we only consider ships that have
⎫ apparent ship feature and abandon small targets like boats.
λ  
K −p e
1,j ⎬ The training and validation dataset are augmented by crop-
1 −p e
+ (pk ,i − pk ,j )2 e 1,j .
8K ⎭ ping and rotating. Since the ship head can provide valuable
j ∈ Nb(i) k =1
semantic features for ship detection, we extract training patches
(8) that are centered at the head, tail, and middle of the ship,
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING
respectively. Moreover, we scan the boundary of sea and land,
and sample patch centers with a distance of 30. The patch cen-
ters are then screened to avoid getting too close to each other.
We further rotate each patch at step of 90◦ , and then flip rotated
patches in horizontal and vertical reflections as in [54]. There
are totally 78 296 training patches with each size of 300 × 300
obtained in this way. For validation dataset, we only crop it
to reduce the testing time in training process. The number of
cropped patches is 432.
2) Sea–Land Image Dataset: It contains 140 training im-
ages, six validation images, and 50 testing images. Most have
size larger than 800 × 800. We collect those images with spa-
tial resolution of 3.0–5.0 m to enlarge the imaging scope. The
sea backgrounds in those images are diverse and complicated,
which makes the segmentation task challenging. There are very
few ships with small sizes in these sea–land scenes. They are
labeled as land in the ground truth.
We augment the training set by first cropping four corners
and center of the original image with size 300 × 300. Then, we
further crop the original image randomly to generate another 20
samples. After that, each of the 25 patches is rotated and flipped
as mentioned above to further enlarge it eight times. There are
totally 28 000 training samples obtained in this way. For images
in validation set, we also just crop them and obtain 150 samples.
B. Implementation Details
The network is implemented under Caffe [55] library. The
training phase includes two stages. First, we train the segmen-
tation network and edge network in parallel. Then, the overall
network is fine-tuned according to (8). Satisfactory results can
be obtained in this way. We set the batch size at 4, initial learning
rate (lr) at 0.001, and decrease lr to one-tenth when the accuracy
on validation set stops increasing. The network is converged af-
ter 80 000 iterations during training and 30 000 iterations during
fine-tuning. The value of λ in (8) is selected at 10 in this paper,
at which we can obtain satisfactory results. Detailed analysis of
the influence of λ can be drawn in Section IV-G.
At testing phase, the input size of the network is adjusted
to 800 × 800 to increase the segmentation accuracy. Input size
for each layer of the network is also adjusted accordingly. We
process each of the testing image with bilinear interpolation,
and reinterpolate outputs to the original size after forward pass.
The negative effect brought by interpolation is negligible since
our testing images have size similar to 800 × 800.
C. Evaluation Criteria
To evaluate segmentation results of different methods, we
employ three metrics, i.e., Precision, Recall, and F-score, which
are defined as
TP
Precision = (9)
TP + FP
TP
Recall = (10)
TP + FN
2 · Precision · Recall
F-score = (11)
Precision + Recall
7
where TP, FP, and FN are true positive, false positive, and false
negative, respectively. Precision measures accuracy of the pre-
dicted points and Recall corresponds to the fraction of true
points that are correctly predicted in the class. F-score is an
overall metric that combines Precision and Recall.
To assess the boundary results of different methods, we first
define Boundary Precision (BP) and Boundary Recall (BR) in
a same way as (9) and (10)
 of correct boundary points on segmentation boundary
BP =
 of boundary points on segmentation boundary
(12)
 of correct boundary points on segmentation boundary
BR = .
 of true boundary points
(13)
One boundary point in segmentation results is considered as
correct if the distance between it and its closet ground truth
boundary is smaller than N . N will be set with different values.
The F-score of boundary is then defined as
2 · BP · BR
F-score-b = . (14)
BP + BR
In the experiments, we assess boundary accuracy of both
segmentation results and outputs of the edge network.
D. Comparative Results on Harbor Image Dataset
We will discuss performance of the proposed FusionNet on
semantic segmentation of land, sea, and ship, and present qual-
itative and quantitative comparisons with state-of-the-art meth-
ods. Five CNNs-based models are compared and details of them
are summarized as follows.
1) FCNs [32]: We build FCNs in the same way as the encoder
network in Fig. 1. The last pooling layer is removed.
Therefore, there are totally three pooling operations. After
that, we unpool outputs of conv4 3 and concatenate them
with outputs of conv3 3. Then, a four times unpooling
layer and a convolutional layer with three-channel output
are sequentially added. Finally, the softmax layer is used
to obtain the prediction maps.
2) Deeplab [34]: We set stride of the last two pooling layers
in the encoder network at 1 and adopt the “holes” strategy
in the fourth group of convolutional layers. The outputs
are four times upsampled to the original size.
3) SegNet [36]: The SegNet shares configuration with our
segmentation network, i.e., the encoder–decoder model.
4) SegNet+EdgeNet: The parallel model of segmentation and
edge network. Both networks are trained simultaneously
to generate the segmentation results and edge detection
results.
5) FusionNet: Final architecture of our proposed network.
Compared with SegNet+EdgeNet, we employ (8) to fur-
ther fine-tune it to achieve fusion of two networks.
We present quantitative comparisons of different methods on
the testing set in Table III. Visual illustration of segmentation
results of military ships is exhibited in Fig. 3. Fig. 4 illustrates
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING

TABLE III
AVERAGE SEGMENTATION RESULTS OF DIFFERENT METHODS ON TESTING IMAGES OF THE HARBOR IMAGE DATASET

Values displayed in bold and blue are the best and second best among five original methods, whereas values displayed in red are the best among five methods that employ
CRF as the postprocessing step.

Fig. 3. Visual comparison of segmentation results with military ships. (a) Original image. (b) Result of FCNs. (c) Result of SegNet. (d) Result of FusionNet
(ours). (e) Ground truth. (f) Output of the edge network. In (b), (c) and (d), improved regions of our method compared with FCNs and SegNet are marked by red
rectangles. FusionNet achieves more consistent segmentation with finer boundary results by incorporating semantic edges.

visual results of several nonmilitary ships. We will discuss the
quantitative and qualitative results separately.
Table III presents the quantitative performance of different
comparing methods on semantic segmentation of land, sea, and
ship. We calculate the Precision, Recall, and F-score of all the
three classes, and give the average F-score of three classes in
the last column. In each column in Table III, the best value is
displayed in bold, and the second best is displayed in blue. We
also evaluate the performance of comparing methods with ad-
ditional postprocessing step by adding a CRF [34] layer, which
can help to reduce misclassifications in land area and obtain
smoother results.
From Table III, we can see that FusionNet achieves the best
results on F-score of all the three classes. Especially, the F-
score value on ship class of our method is more than 6% higher
than that of SegNet, and the average F-score of our method is
more than 2% higher than that of SegNet. Compared with three
traditional networks, our parallel model (SegNet+EdgeNet) also
achieves better performance. Specifically, the ship F-score and
average F-score of SegNet+EdgeNet are more than 3% and 1%
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING 9

Fig. 4. Visual comparison of segmentation results with nonmilitary ships. (a) Original image. (b) Result of FCNs. (c) Result of SegNet. (d) Result of FusionNet
(ours). (e) Ground truth. (f) Output of the edge network. We mark improved regions of our method compared with FCNs and SegNet by red rectangles. Results
within yellow rectangles in the first row illustrate an example of ship false alarm that exists in all three methods.

TABLE IV significantly improved due to the smooth operation within land

F1-SCORE-B (%) OF DIFFERENT METHODS ON HARBOR IMAGE DATASET
Best and second best results of segmentation methods are displayed in bold and
blue, respectively. Best values of the outputs of edge network are displayed in red.
higher than those of SegNet, which has best values on these two
metrics among three traditional methods.
It is worth noting that the three traditional methods also per-
form well on segmentation of sea and land due to the rich
hierarchical feature learning ability of CNNs. The accuracy im-
provement of our method on sea and land is not as remarkable
as that on ship. One of the reason is that compared with sea and
land, the ship regions are relative small, which makes segmen-
tation results have a bigger fluctuation for different methods. In
spite of that, the visual improvement of our method on sea–land
is still obvious, such as results in the fourth row of Fig. 3 and
comparisons in Fig. 7. Moreover, we can obtain a remarkable
improvement of our method by comparing boundary accuracy
of different methods in Table IV.
The F-score metrics of all the comparing methods are en-
hanced by adding CRF layer. Note that the ship Precision is
region, whereas the ship Recall is degraded because the pixel-
level smooth constraint in CRF blurs the segmented boundaries
between the ship and its adjacent class. Examples of CRF ef-
fect are presented in Fig. 5. The first two rows exhibit that the
misclassifications in land area are reduced with CRF. The last
two rows show zoomed-in results of the regions within blue
rectangles of the first two rows, from which we can see that the
boundary accuracy of ship is decreased due to the low contrast
around it.
It is worth noting that the average F-score of FusionNet is
more than 1% higher than that of SegNet+EdgeNet_CRF, which
demonstrates the effectiveness of employing semantic edges to
measure semantic similarity between pixels than spectral-based
affinity function.
Figs. 3 and present visual comparisons of different methods
on military ships and nonmilitary ships, respectively. Outputs of
the edge network are illustrated in the last column. We can see
that our method also performs well on nonmilitary ships in spite
of rather few training samples since they share many features
with military ships. The results of FusionNet contain fewer false
alarms of ship, such as comparisons in the red rectangles of the
fourth row in Fig. 3 and first row in Fig. 4. Results in the
first and third row of Fig. 3 demonstrate that our method can
better reduce misclassification for neighboring ship and wharf.
In addition, our method is more robust to shadows around the
ship body, as ship results in the second row of Fig. 3 show.
Boundary accuracy of segmentation results of different meth-
ods are further calculated. We also evaluate the outputs of our
edge network. Qualitative results are presented in Fig. 6. De-
tailed values are listed in Table IV. The distance tolerance N
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING

Fig. 5. Visual presentation of CRF effect. (a) Original image. (b) Result of FusionNet. (c) Result of FusionNet_CRF. (d) Ground Truth. The first two rows
illustrate comparisons by adding the CRF layer. Improved regions are marked by red rectangles. The last two rows show zoomed-in results of the regions within
blue rectangles in (a) of the first two rows. The CRF operation can smooth results in land regions, while it may also blur the ship boundaries.

Fig. 6. (a) Boundary accuracy of segmentation results on harbor image dataset. (b) Accuracy of the outputs of edge network on harbor image dataset.

is set from 1 to 5. Compared with the parallel model (Seg-
Net+EdgeNet), FusionNet achieves significant improvement
both in boundary accuracy of the segmentation results and
outputs of the edge network, which has demonstrated the ef-
fectiveness of edge aware regularization in refining the entire
model. Note that for both parallel model and FusionNet, the
edge network achieves higher F-score-b value than that of its
corresponding segmentation results. The intuitive explanation
is that our edge network is expanded from the segmentation net-
work by adding additional convolutional and unpooling layers,
which improves its capacity to learn more task-specific model.
E. Comparative Results on Sea–Land Image Dataset
Experiments are conducted to evaluate our model on the sea–
land dataset, which is a two-class segmentation problem. We
adjust output channel of the last convolutional layer and soft-
max layer in the segmentation network to 2. The following five
methods are compared to assess their performance.
1) LATM [15]: Locally adaptive thresholding method. It au-
tomatically set thresholds for each local region by fitting
a bimodal Gaussian curve. The final results are processed
by morphological operations.
2) SMS [8]: Global thresholding method, which builds a sta-
tistical model for the sea area based on the OTSU results.
The “holes” in land areas of the results are filled accord-
ing to the variance difference between sea model and
themselves.
3) SegNet [36]: The SegNet shares configuration with our
segmentation network.
4) SegNet+EdgeNet: The parallel model of segmentation and
edge network.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING 11

Fig. 7. Visual comparison of sea–land segmentation with different methods. (a) Original image. (b) Result of LATM. (c) Result of SMS. (d) Result of SegNet.
(e) Result of FusionNet. (f) Ground Truth. Improved regions in FusionNet results compared with SegNet are marked by red rectangles.

TABLE V
AVERAGE SEGMENTATION RESULTS OF DIFFERENT METHODS ON TESTING IMAGES OF SEA–LAND IMAGE DATASET

Values displayed in bold and blue are the best and second best among five comparing methods.

Fig. 8. (a) Boundary accuracy of segmentation results on sea–land image dataset. (b) Accuracy of the outputs of edge network on sea–land image dataset.

5) FusionNet: Final architecture of our proposed network,
which is fine-tuned on the basis of SegNet+EdgeNet.
We implement LATM and SMS by ourselves using C++ pro-
gramming, and fill the connected “hole” regions whose size are
smaller than 200 in their land results to get better comparative
effect.
Fig. 7 illustrates visual comparison of five segmentation meth-
ods. In can be seen that both LATM and SMS are prone to
misclassify land pixels due to their complicated intensity and
texture distribution. The misclassifications cannot be simply
resolved by morphological operations. CNNs-based methods
achieve better segmentation by hierarchical feature learning.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
12
TABLE VI
F1-SCORE-B (%) OF DIFFERENT METHODS ON SEA–LAND IMAGE DATASET
Best and second best boundary results of segmentation methods are displayed in
bold and blue, respectively. Best values of the outputs of edge network are displayed
in red.
TABLE VII
COMPARATIVE RESULTS OF SHIP DETECTION WITH INSHORE SHIP
DETECTION METHODS
Method Precision Recall
Method in [5] 91.43 54.70
Method in [9] 88.99 82.91
Ours 94.61 92.94
Best values are displayed in bold.
Compared with SegNet, FusionNet obtains smoother land re-
sults as well as finer segmented boundaries by incorporating
edge information.
We exhibit quantitative results in Table V. Our method
achieves the best performance on all of the evaluation metrics.
Specifically, FusionNet has almost 0.25% improvement in land
F-score and 0.18% improvement in sea F-score with respect to
SegNet. LATM and SMS have relative lower land Recall, which
corresponds to more misclassifications within land region.
Evaluation of sea–land boundary is further conducted and
results are presented in Fig. 8 and Table VI. From Fig. 8(a),
we can see that the segmentation results of FusionNet and
SegNet+EdgeNet have the best and second best boundary ac-
curacy. Traditional methods based on threshold selection and
morphological operations behave poor on F-score-b due to the
existence of lots of errors in their segmented land regions. Re-
sults in Fig. 8(b) demonstrate the effectiveness of our edge
network on sea–land problem.
F. Comparative Results With Ship Detection Methods
To better assess the performance of our proposed model,
we compare the ship detection results with two inshore ship
detection methods, i.e., the methods in [5] and [9]. Both methods
are based on ship head detection and only suitable for “V” shape
head. The former is a supervised method, which transforms local
regions around ship head into polar coordinate system to extract
its shape feature. Then, an SVM classifier is trained to predict
ship head. The latter is unsupervised and extracts ship head
by parabola fitting. After acquiring head point, both methods
employ line detection to identify ship body.
We select 30 images from the testing set of harbor image
dataset to evaluate performance of three methods. There are
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING
F-score
68.45
85.84
93.77
Fig. 9. Ship detection results based on segmentation results of FusionNet.
(a) Original image. (b) Result of FusionNet. (c) Ground truth. RBB and ship
direction are drawn in the input.
117 ships with “V” shape head. The segmentation results of
FusionNet are used to extract RBBs. As for method in [9], we
select 80 images from the training set to train the head classifier,
which contain 360 ships.
To better exhibit the detection result of our method, we calcu-
late the RBB and direction of the segmented ships. Specifically,
after acquiring the segmentation map, we first calculate the con-
nected ship regions and compute center locations of them. Then
for each ship region, we traverse the angles between 0 and π. For
each angle, a local coordinate system at center of the connected
region is built, and pixels within the region will be projected
to the short axis. The best angle is found such that the sum of
length of projections is the minimum. The length and width of
ship can be further computed by transforming the ship region
into local coordinate system with the best angle.
Comparative results are listed in Table VII. Method in [5]
has low ship recall because it performs sea–land segmentation
before head detection, which produces unsatisfactory land mask
with coarse boundaries to identify head position. Moreover, both
traditional methods are sensitive to the contrast nearby head re-
gions, which is also a reason to cause low recall. Compared
with them, our proposed method achieves better result for in-
shore ship detection through learning rich hierarchical features
to discriminate between ship and its adjacent region. More de-
tection results of our method are illustrated in Fig. 9.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING 13

Fig. 10. Influence of λ on the testing images of harbor image dataset. The value of λ is selected from coarse to fine. When λ is 10, the F-score has the best value.

Fig. 11. Visual comparison of segmentation results when λ is set at different values. When λ is too big, the boundary results are worse due to the strong smoothing
effect brought by edge aware regularization term.

G. Influence of Parameter λ V. CONCLUSION

In this section, we further discuss the influence of λ in (8)
on the final segmentation results. In (8), the λ is employed to
adjust the smoothing effect that is controlled by outputs of the
edge network. If the value of λ is too small, the edge aware
regularization is negligible and the FusionNet will degenerate
into SegNet+EdgeNet. However, if λ is too big, it will over-
smooth the image and cause decrease of boundary accuracy of
segmentation results.
We have evaluated the influence of λ on the testing images
of harbor image dataset. The average F-score of three class is
presented in Fig. 10. The value of λ is first selected in a coarse
range from 0.1 to 1000. Then, it is selected in a finer range from
10 to 100. It can be seen that when λ is 10, the F-score is the
best. Fig. 11 presents visual effect when λ is too big. We can see
that too large value of λ will decrease the boundary accuracy of
segmentation results because of the strong smoothing effect of
edge aware regularization.
In this paper, an edge aware DCNN has been proposed to
address semantic segmentation of land, sea, and ship. Specifi-
cally, we constructed the edge network by expanding a branch
from the encoder–decoder structure and train in parallel with it
to learn semantic boundaries. The outputs of the edge network
are further employed to refine the entire model with an edge
aware regularization term, which encourages probability prop-
agation among neighboring pixels with constraint of semantic
boundaries.
Extensive experiments have demonstrated the advantages
of FusionNet by comparing with state-of-the-art methods.
Our model performs better in producing spatially consistent
results with well boundary located. We have also evaluated
FusionNet on the sea–land image dataset to address the two-
class problem and achieved better performance in quantita-
tive and qualitative comparisons. Moreover, the ship detec-
tion results based on segmentation results have proved that our
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
14 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING
method is effective in comparison with inshore ship detection
methods.
REFERENCES
[1] E. H. Boak and I. L. Turner, “Shoreline definition and detection: A review,”
J. Coastal Res., vol. 21, no. 4, pp. 688–703, Jul. 2005.
[2] A. A. Fugura, L. Billa, and B. Pradhan, “Semi-automated procedures for
shoreline extraction using single RADARSAT-1 SAR image,” Estuarine,
Coastal Shelf Sci., vol. 95, pp. 395–400, 2011.
[3] F. Baselice and G. Ferraioli, “Unsupervised coastal line extraction
from SAR images,” IEEE Geosci. Remote Sens. Lett., vol. 10, no. 6,
pp. 1350–1354, Nov. 2013.
[4] U. C. Benz, P. Hofmann, G. Willhauck, I. Lingenfelder, and M. Hey-
nen, “Multi-resolution, object-oriented fuzzy analysis of remote sensing
data for GIS-ready information,” ISPRS J. Photogrammetry Remote Sens.,
vol. 58, pp. 239–258, 2004.
[5] G. Liu, Y. Zhang, X. Zheng, X. Sun, K. Fu, and H. Wang, “A new method
on inshore ship detection in high-resolution satellite images using shape
and context information,” IEEE Geosci. Remote Sens. Lett., vol. 11, no. 3,
pp. 617–621, Mar. 2014.
[6] C. Zhu, H. Zhou, R. Wang, and J. Guo, “A novel hierarchical method
of ship detection from spaceborne optical image based on shape and
texture features,” IEEE Trans. Geosci. Remote Sens., vol. 48, no. 9,
pp. 3446–3456, Sep. 2010.
[7] J. Tang, C. Deng, G.-B. Huang, and B. Zhao, “Compressed-domain ship
detection on spaceborne optical image using deep neural network and
extreme learning machine,” IEEE Trans. Geosci. Remote Sens., vol. 53,
no. 3, pp. 1174–1185, Mar. 2015.
[8] X. You and W. Li, “A sea-land segmentation scheme based on statis-
tical model of sea,” in Proc. Int. Congr. Image Signal Process., 2011,
pp. 1155–1159.
[9] S. Li, Z. Zhou, B. Wang, and F. Wu, “A novel inshore ship detection via
ship head classification and body boundary determination,” IEEE Geosci.
Remote Sens. Lett., vol. 13, no. 12, pp. 1920–1924, Dec. 2016.
[10] D. Cheng, G. Meng, S. Xiang, and C. Pan, “Efficient sea-land segmenta-
tion using seeds learning and edge directed graph cut,” Neurocomputing,
vol. 207, pp. 36–47, Sep. 2016.
[11] S. K. Mcfeeters, “The use of the normalized difference water index
(NDWI) in the delineation of open water features,” Int. J. Remote Sens.,
vol. 17, no. 7, pp. 1425–1432, 1996.
[12] T. Kuleli, A. Guneroglu, F. Karsli, and M. Dihkan, “Automatic detection
of shoreline change on coastal Ramsar wetlands of Turkey,” Ocean Eng.,
vol. 38, no. 10, pp. 1141–1149, Jul. 2011.
[13] K. Di, J. Wang, R. Ma, and R. Li, “Automatic shoreline extraction from
high-resolution IKONOS satellite imagery,” in Proc. ASPRS Annu. Conf.,
2003, pp. 1–11.
[14] U. R. Aktas, G. Can, and F. T. Y. Vural, “Edge-aware segmentation in
satellite imagery: A case study of shoreline,” in Proc. IAPR Workshop
Pattern Recognit. Remote Sens., Nov. 2012, pp. 1–4.
[15] H. Liu and K. C. Jezek, “Automated extraction of coastline from satel-
lite imagery by integrating canny edge detection and locally adaptive
thresholding methods,” Int. J. Remote Sens., vol. 25, no. 5, pp. 937–958,
Mar. 2004.
[16] F. Chen, W. Yu, X. Liu, K. Wang, L. Gong, and W. Lv, “Graph-based ship
extraction scheme for optical satellite image,” in Proc. IEEE Int. Geosci.
Remote Sens. Symp., 2011, pp. 491–494.
[17] G. Jubelin and A. Khenchaf, “Multiscale algorithm for ship detection in
mid, high and very high resolution optical imagery,” in Proc. IEEE Int.
Geosci. Remote Sens. Symp., 2014, pp. 2289–2292.
[18] S. Qi, J. Ma, J. Lin, Y. Li, and J. Tian, “Unsupervised ship detection based
on saliency and S-HOG descriptor from optical satellite images,” IEEE
Geosci. Remote Sens. Lett., vol. 12, no. 7, pp. 1451–1455, Jul. 2015.
[19] C. Corbane, F. Marre, and M. Petit, “Using SPOT-5 HRG data in panchro-
matic mode for operational detection of small ships in tropical area,”
Sensors, vol. 8, no. 5, pp. 2959–2973, May 2008.
[20] C. Corbane, L. Najman, E. Pecoul, L. Demagistri, and M. Petit, “A com-
plete processing chain for ship detection using optical satellite imagery,”
Int. J. Remote Sens., vol. 31, no. 22, pp. 5837–5854, Jul. 2010.
[21] T. Ojala and I. Harwood, “A comparative study of texture measures with
classification based on featured distributions,” Pattern Recognit., vol. 29,
no. 1, pp. 51–59, Jan. 1996.
[22] Z. Shi, X. Yu, Z. Jiang, and B. Li, “Ship detection in high-resolution
optical imagery based on anomaly detector and local shape feature,” IEEE
Trans. Geosci. Remote Sens., vol. 52, no. 8, pp. 4511–4523, Aug. 2014.
[23] J. Lin, X. Yang, S. Xiao, Y. Yu, and C. Jia, “A line segment based in-
shore ship detection method,” in Proc. Future Control Autom., 2012,
pp. 261–269.
[24] Z. Deng, H. Sun, S. Zhou, J. Zhao, and H. Zou, “Toward fast and accurate
vehicle detection in aerial images using coupled region-based convolu-
tional neural networks,” IEEE J. Sel. Topics Appl. Earth Observ. Remote
Sens., vol. 10, no. 8, pp. 3652–3664, Aug. 2017.
[25] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies
for accurate object detection and semantic segmentation,” in Proc. Comp.
Vis. Pattern Recognit., 2014, pp. 580–587.
[26] R. Girshick, “Fast R-CNN,” in Proc. Int. Conf. Comput. Vis., 2015,
pp. 1440–1448.
[27] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time
object detection with region proposal networks,” in Proc. Int. Conf. Neural
Inf. Process. Syst., 2015, pp. 91–99.
[28] I. Sevo and A. Avramovic, “Convolutional neural network based auto-
matic object detection on aerial images,” IEEE Geosci. Remote Sens.
Lett., vol. 13, no. 5, pp. 740–744, May 2016.
[29] W. Liu et al., “SSD: Single shot multibox detector,” in Proc. Eur. Conf.
Comput. Vis., 2016, pp. 21–37.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
with deep convolutional neural networks,” in Proc. Int. Conf. Neural Inf.
Process. Syst., 2012, pp. 1097–1105.
[31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” in Proc. ICLR, 2015, pp. 1–14.
[32] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proc. Comp. Vis. Pattern Recognit., 2015,
pp. 3431–3440.
[33] H. Noh, S. Hong, and B. Han, “Learning deconvolution network for seman-
tic segmentation,” in Proc. Int. Conf. Comput. Vis., 2015, pp. 1520–1528.
[34] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille,
“Semantic image segmentation with deep convolutional nets and fully
connected CRFs,” arXiv:1412.7062, 2014.
[35] W. Zhao, S. Du, and W. J. Emery, “Object-based convolutional neural
network for high-resolution imagery classification,” IEEE J. Sel. Top-
ics Appl. Earth Observ. Remote Sens., vol. 10, no. 7, pp. 3386–3396,
Jul. 2017.
[36] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A deep
convolutional encoder-decoder architecture for image segmentation,”
arXiv:1511.00561, 2015.
[37] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing
network,” arXiv:1612.01105, 2016.
[38] J. Dai, K. He, and J. Sun, “Instance-aware semantic segmentation via
multi-task network cascades,” in Proc. Comp. Vis. Pattern Recognit., 2016,
pp. 3150–3158.
[39] Y. Li, C. Tao, Y. Tan, K. Shang, and J. Tian, “Unsupervised multilayer
feature learning for satellite image scene classification,” IEEE Geosci.
Remote Sens. Lett., vol. 13, no. 2, pp. 157–161, Feb. 2016.
[40] Z. Zou and Z. Shi, “Ship detection in spaceborne optical image with
SVD networks,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10,
pp. 5832–5845, Oct. 2016.
[41] F. Zhang, B. Du, and L. Zhang, “Saliency-guided unsupervised feature
learning for scene classification,” IEEE Trans. Geosci. Remote Sens.,
vol. 53, no. 4, pp. 2175–2184, Apr. 2015.
[42] F. Zhang, B. Du, L. Zhang, and M. Xu, “Weakly supervised learn-
ing based on coupled convolutional neural networks for aircraft detec-
tion,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 9, pp. 5553–5563,
Sep. 2016.
[43] N. Otsu, “A threshold selection method from gray-level histograms,” IEEE
Trans. Syst. Man Cybern., vol. SMC-9, no. 1, pp. 62–66, Jan. 1979.
[44] C. Rother, V. Kolmogorov, and A. Blake, “GrabCut—interactive fore-
ground extraction using iterated graph cuts,” in Proc. ACM SIGGRAPH,
2004, pp. 309–314.
[45] Z. Liu, H. Wang, L. Weng, and Y. Yang, “Ship rotated bounding box
space for ship extraction from high-resolution optical satellite images
with complex backgrounds,” IEEE Geosci. Remote Sens. Lett., vol. 13,
no. 8, pp. 1074–1078, Aug. 2016.
[46] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,”
arXiv:1703.06870, 2017.
[47] J. Dai, Y. Li, K. He, and J. Sun, “R-FCN: Object detection via region-based
fully convolutional networks,” arXiv:1605.06409, 2016.
[48] P. O. Pinheiro, R. Collobert, and P. Dollár, “Learning to segment ob-
ject candidates,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2015,
pp. 1990–1998.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHENG et al.: FUSIONNET: EDGE AWARE DEEP CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF REMOTE SENSING
[49] J. Dai, K. He, Y. Li, S. Ren, and J. Sun, “Instance-sensitive fully convolu-
tional networks,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 534–549.
[50] G. Bertasius, J. Shi, and L. Torresani, “Semantic segmentation with
boundary neural fields,” in Proc. Comp. Vis. Pattern Recognit., 2016,
pp. 3602–3610.
[51] L.-C. Chen, J. T. Barron, G. Papandreou, K. Murphy, and A. Yuille,
“Semantic image segmentation with task-specific edge detection using
CNNs and a discriminatively trained domain transform,” in Proc. Comp.
Vis. Pattern Recognit., 2016, pp. 4545–4554.
[52] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network
training by reducing internal covariate shift,” in Proc. Int. Conf. Mach.
Learn., 2015, pp. 448–456.
[53] V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltz-
mann machines,” in Proc. Int. Conf. Mach. Learn., 2010, pp. 807–814.
[54] G. Cheng, Y. Wang, S. Xu, H. Wang, S. Xiang, and C. Pan, “Automatic
road detection and centerline extraction via cascaded end-to-end convolu-
tional neural network,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 6,
pp. 3322–3337, Jun. 2017.
[55] Y. Jia et al., “Caffe: Convolutional architecture for fast feature embedding,”
in Proc. ACM Int. Conf. Multimedia, 2014, pp. 675–678.
Dongcai Cheng received the B.S. degree in electronic
and information engineering from the University of
Science and Technology of China, Hefei, China, in
2011. He is currently working toward the Ph.D. de-
gree in pattern recognition and intelligent system in
the Institute of Automation, Chinese Academy of Sci-
ences, Beijing, China.
His research interests include image processing,
object detection, and remote sensing.
15
Gaofeng Meng (M’17) received the B.S. degree in
applied mathematics from Northwestern Polytechni-
cal University, Xi’an, China, in 2002, the M.S. de-
gree in applied mathematics from Tianjin University,
Tianjin, China, in 2005, and the Ph.D. degree in con-
trol science and engineering from Xi’an Jiaotong Uni-
versity, Xi’an, China, in 2009.
He is currently an Associate Professor in the Na-
tional Laboratory of Pattern Recognition, Institute of
Automation, Chinese Academy of Sciences, Beijing,
China. His research interests include computer vi-
sion, image processing, and machine learning.
Dr. Meng has been an Associate Editor of Neurocomputing since 2014.
Shiming Xiang (M’12) received the B.S. degree
in mathematics from Chongqing Normal University,
Chongqing, China, in 1993, the M.S. degree in com-
puting mechanics from Chongqing University, China,
in 1996, and the Ph.D. degree in computer sciences
from the Institute of Computing Technology, Chinese
Academy of Sciences, Beijing, China, in 2004.
From 1996 to 2001, he was a Lecturer at the
Huazhong University of Science and Technology,
Wuhan, China. He was a Postdoctoral Researcher
with the Department of Automation, Tsinghua Uni-
versity, Beijing, China, until 2006. He is currently a Professor with the Institute
of Automation, Chinese Academy of Sciences. His research interests include
pattern recognition and image processing.
Chunhong Pan received the B.S. degree in automatic
control from Tsinghua University, Beijing, China,
in 1987, the M.S. degree in optic and electronic
engineering from the Shanghai Institute of Optics
and Fine Mechanics, Chinese Academy of Sciences,
Beijing, China, in 1990, and the Ph.D. degree in
pattern recognition and intelligent system from the
Institute of Automation, Chinese Academy of Sci-
ences, in 2000.
He is currently a Professor with the National Lab-
oratory of Pattern Recognition, Institute of Automa-
tion, Chinese Academy of Sciences. His research interests include computer
vision, image processing, computer graphics, and remote sensing.