Parallel Processing Architecture of Remotely Sensed Image Processing System
Based on Cluster
Hangye Liu1,2 Yonghong Fan1 Xueqing Deng1 Song Ji1,2
1 2
Dept. of Remote Sensing Information Engineering
Zhengzhou Institute of Mapping and Surveying
Zhengzhou, PR. China
lhy2004lx@163.com
Abstract—As remote sensing platform and sensor technologies
develop quickly, the resolution of remotely sensed images
becomes much higher. However, the data processing
technologies have been following behind, especially in
computing speed and efficiency. Parallel computing technology
provides an effective way to solve this problem. In this paper,
based on the analysis of parallel computing system, the
architecture of remote sensing parallel processing system
which is based on cluster computing is studied, and a new
hybrid cluster computing architecture for massive remote
sensing data processing is put forward, task scheduling of
remote sensing processing system and parallel processing
workflow are discussed.
Keywords- Parallel Computing; Cluster Computing; Load
Balancing; Geometric correction; Task Scheduling
I. INTRODUCTION
With the rapid development of remote sensing platform
and sensor technologies, the resolution of acquired remotely
sensed images becomes much higher and the amount of data
becomes huge. A common satellite ground receiving station
can receive image data of more than 10G bytes. Comparing
with the fast development of data acquiring technology,
recent remote sensing data processing speed falls far behind,
which means that abundant data cannot be translated into
useful information in time. Parallel computing is able to
improve the computing speed to a great extent in massive
data processing, which makes itself an effective way to
solve the problem of processing efficiency in remote
sensing.
There are very important differences between processing
mode of remote sensing processing under parallel
computing environment and that under traditional
processing mode, which include task assignment and load
balancing. So it is of vital importance to introduce
architecture with highly efficient parallel mechanism to
remote sensing processing. Most of recent remote sensing
parallel processing systems are based on specialized parallel
computers which are very expensive and unfavorable to
popularize. Meanwhile, Cluster computer represents the
development direction of parallel computing, and remote
sensing processing based on cluster computing is of high
efficiency and cost-benefit ratio. Through proper
architecture design and effective algorithm, it can achieve
high computing speed. In this paper, based on the analysis
978-1-4244-4131-0/09/$25.00 ©2009 IEEE
Jiangxi Province Key Lab for Digital Land
East China Institute of Technology
Fuzhou, PR. China
jisong_chxy@163.com
of two main types of parallel computing system, the
architecture of remote sensing parallel processing system is
studied, and the task scheduling and workflow under
specific remote sensing processing application are
discussed.
II. PARALLEL COMPUTER SYSTEMS
Current parallel computer systems may fall into two
basic types, which are shared memory multi-computers and
message passing multi-computers[1].
A. Shared memory muliti-computer
In shared memory multi-computers, memories are shared
among computers, which means multi-computers share a
uniformly coded storage unit and data exchange is realized
by addressing operation. Data communication in shared
memory multi-computers is based on bus, so it is convenient
and faster to access data among computers. Without the
explicit control of the data exchange, it is easier for
programmers to realize a parallel computing program under
this platform. The structure of shared memory multi-
computers is depicted as Figure 1.
Figure 1. Structure of Shared Memory Multi-computers
However, scalability of this system is relatively weak,
when adding massive processors, fast data accessing cannot
be guaranteed. This type of parallel computing system can be
utilized in the applications which don’t need two much
computing nodes but do need much data communication.
B. Message passing multi-computers
Message passing multi-computer system uses network to
connect computers or processors, and each computer has its
own storage unit which cannot be accessed directly by other
computers. Data exchange is implemented by message
passing on networks, and the message passing is under
explicit control. The drawback of this platform is that
programming is more complicated, but in another way, this
structure is more flexible and is extensible. The structure of
message passing multi-computers is depicted as Figure 2.
Figure 2. Structure of Message Passing Multi-computers
Cluster belongs to this kind of parallel computer system.
Cluster system can be built by using low-cost PC computers
or workstations and can be extended easily by adding
computers, disks or other resources. Therefore, by proper
system design and high efficient parallel algorithms, cluster
computing is able to get very high computing speed with
lower cost.
Some mature remote sensing parallel processing systems,
such as pixel factory-[6] and DPGrid[7], adopt specialized
parallel computers as its computing platform. The cost of
whole system is enormously expensive and price is swift to
reach million dollars. The maintenance cost is also very
high. Recently, as the performance of PC computers and
technology of network develop fast, the computing power of
cluster improves sharply. Using cluster computer system as
computing platform is of profound meaning to popularize the
application of remote sensing.
III. ARCHITECTURE OF PARALLEL REMOTE SENSING
PROCESSING SYSTEM
Process architecture of parallel remote sensing
processing system is core of the whole system, of which the
computing efficiency affects the processing speed of whole
system. In order to achieve the best performance, the
computing power of every single PC in the cluster system
should be fully utilized.
Remote sensing processing involves a huge amount of
data and complex computing operations, which makes both
the time of calculation and data communication be costly.
Considering the characteristics of remote sensing processing,
an efficient processing system should allocate computing
load into as many processing nodes as possible to speed up
and avoid as much unnecessary data communication as
possible, or use more efficient methods to improve the data
communicating speed. With the development of multi-
processor technology, there is more than one processor in a
single PC and multi-processors in one PC can work together
efficiently so as to form a mini shared memory parallel
system. Besides, as the graphic processing unit (GPU)
technology develops rapidly, the powerful computing
capability of GPU is able to improve the computing speed
further. Taking two new technologies above into account,
this paper puts forward a new hybrid processing architecture
of remote sensing processing system based on cluster
computing. The processing architecture is showed as Figure
3.
Figure 3. Architecture of remote sensing parallel processing
system based on cluster computing
The whole processing system is divided into several mini
clusters (from cluster 0 to cluster n), which are connected by
gigabit Ethernet to guarantee fast data communication. Each
mini cluster consists of several PCs which may have multi-
processors and multi-GPUs to form a mini shared memory
system. Architecture of mini shared memory system is
showed as Figure 4.
Figure 4. Architecture of mini shared memory system
Traditional cluster system consists of high-network and
independent computers as computing nodes. It is useful
enough since traditional PC has only one processor and the
computing resource can be fully utilized. However, as multi-
processors are widely used in common PC and the floating
computing advantage of GPU becomes prominent, the
potential computing power of common PC is worth
developing.
The architecture above is a kind of hybrid parallel
architecture, which combines the shared memory system
into cluster system. The whole cluster is divided into mini
clusters, so the workload of single management server can
be reduced to avoid communication bottleneck and each
mini cluster takes charge in cooperating PC nodes in it.
Because the number of PC in one mini cluster will be
relatively small, the workload of cooperation will not affect
the computing efficiency and fast network guarantees the
smooth data communication. Each PC in mini cluster
represents a shared memory system, which has better data
communication power. Due to the relatively weak
scalability of shared memory system and restriction of
memory, this type of parallel computing system is not
suitable for massive parallel computing, but when the
processing tasks are already subdivided and necessary
amount of memory is restricted, its strong communication
speed can achieve much better performance on the parallel
computing efficiency.
As mentioned above, remote sensing processing is data-
intensive, massive data communication will be inevitable. In
this architecture, the massive image data and corresponding
processing tasks are divided into mini clusters, and in the
mini cluster, the data and the sub-tasks will be subdivided
further. Through the subdivision, the amount of data in a
single processing node is reduced into such a number that the
data can be processed in a shared memory system, thus the
parallel computing can be performed more efficiently. At the
same time, all the computation resources are fully utilized.
IV. TASK SCHEDULING AND LOAD BALANCING
For further optimization of parallel computing, the task
that performed under the parallel computing platform should
be allocated and controlled according to certain principle.
The aim of coordinating and control the allocation of tasks
is to achieve the best output speed of final results. In task
scheduling, an important norm is whether the load
(computing load, work load etc.) on each computing node is
balanced. If loads that allocated to each computing node are
nearly equal to each other, the computing nodes are likely to
be fully utilized. Otherwise, the nodes that finish working
earlier have to wait for other nodes to proceed to next
processing.
The first problem in task scheduling is adopting what
kind of scheduling policy. Remotely sensed data is of huge
amount, so the number of parallel computing nodes should
be big enough to achieve better efficiency. However, when
adopting centered scheduling policy, too many nodes will
inevitably cause communication bottleneck in centered
management node. Besides, the fully distributed scheduling
policy is hard to implement, especially when there are too
many computing nodes. Therefore, a semi-distributed
scheduling policy should be more suitable for remote sensing
processing. The policy is described in Figure 5.
Figure 5. Semi-distributed scheduling policy
Combined with the traits of hybrid cluster architecture,
each mini cluster system adopts centered scheduling policy.
In mini cluster system, one of the PCs or a server is
designated to be load information management center which
is in charge of collecting load information on each node.
Because the number of nodes in each cluster is restricted, the
communication on load information management center will
not be a bottleneck of overall performance. Much of the
scheduling work being done in mini clusters, the task
scheduling among mini clusters will be less time-consuming
and less complex, so these task scheduling can be performed
distributed or even through a overall centered scheduling
server.
After determining the scheduling policy, load
information collection policy should be considered. Load
information collection involves two aspects, one is the load
index which measures the load on a node, and another is
when to collect the load information. In most situations,
computation in remote sensing processing is repeated on
one single pixel, so number of pixels or blocks of image that
need to be processed serves as load index. But in certain
applications, the computation complexity depends on the
information content, so image blocks with same size would
cause different computation. Thus, computation load
becomes load index. Study shows that choosing single
parameter as load index will be more effective[5], so
according to characteristics of different applications, the
load index would be different. The information collection
policies include periodical collection policy, on-demand
collection and on-state-change collection, among which on-
state-change collection policy collects information only
when the load state on node changes. Remote sensing
processing is computation intensive and most computation
resource should be used in computing. Therefore, the
chance that scheduling will disturb computation should be
as small as possible. On-state-change policy not only
reduces the redundant communication but also avoids the
postponement of executing the scheduling, so it is more
suitable for massive remote sensing processing.
Load migration policy should also be considered. A
specific load migration involves two problems, what kind of
load to migrate and how to implement. In remote sensing,
scheduling should be as simple as possible to make sure the
computing not to be disturbed too much. Data migration
involves only the re-allocation of data, which has lower
complexity and is easy to control, so choosing data migration
to balancing the workload is applicable to remote sensing
parallel processing. For further optimization, a pre--
migration policy can be applied. Because the complexity of
migrating data is relative low, it is possible to perform
computation and load migration at the same time so as to
improve efficiency. Before load migration, a forecast of load
will be performed. The load migration is actually overlapped
with computing, so influence that data communication exerts
on overall computing efficiency is reduced.
V. DESIGN OF WORKFLOW
Since the working mode of remote sensing data
processing is parallel and very different from traditional
remote sensing, the workflow should also be re-designed.
The workflow changes when processing application is
different. Therefore, in this paper, we assume a specific
remote sensing application which starts from image
geometric correction to generation of digital surface model.
The workflow falls into two main modes. One is based on
the data that need to be processed, of which the main
solution is to divide the data into several blocks which are
independent on each other and allocate them to computing
nodes to process through the whole workflow. This mode is
easy to implement but the task scheduling is hard to perform,
because workflow on each node is independent. In order to
control the load balance on each computing node, another
mode should be adopted. The other mode is based on the
specific processing task and the whole workflow is divided
into several parallel processing steps. These steps are
executed on each computing node at the same time and
necessary operations will be done to guarantee
synchronization. The workflow is depicted as Figure 6.
Figure 6. Task-based parallel processing workflow
During each parallel processing step, the task scheduling
will be performed to make sure the loads on computing
nodes are balanced. After each step is finished, an explicit
synchronization will be made to ensure that each computing
node proceed to next step at the same time. Thanks to the
synchronization, in certain time, tasks executed on each
computing node are the same, so task scheduling is easy to
perform. It is quite easy to tell that this workflow needs more
data communication and task scheduling will be more
complex. Combine with architecture which is discussed
above, the task scheduling can be performed hierarchically.
Task scheduling inside the mini clusters is executed more
frequently to make sure the specific computing nodes have
nearly the same workload. Task scheduling among mini
clusters takes place only when loads on mini clusters are
heavily unbalanced so as to reduce the data communication.
VI. SUMMARIZE
Comparing with the fast development of data acquiring
technology, recent remote sensing data processing speed falls
far behind and parallel computing technology is an efficient
way to solve this problem. Aiming at the massive remote
sensing processing, this paper analyzes two main types of
parallel computing platform, and based on that, put forward a
hybrid parallel processing architecture for remote sensing
processing. This architecture is flexible and high efficient to
meet the requirement for computation intensive and data
intensive application. Also, proper task scheduling policy
and workflow are discussed.
ACKNOWLEDGMENT
This work described here is funded by Jiangxi Province
Key Lab for Digital Land (DLLJ200901).
REFERENCES
[1] Barry Wilkinson, Michael Allen., Parallel Programming, China
Machine Press, Beijing, 2002, pp.9-10.
[2] Li Jun, Li Deren, Key Techniques for Distributed Processes of
Remote Sensing Imagery, Geomatics and Information Science of
Wuhan University, 1999, pp. 15-19.
[3] Deng Xueqing. Study on Architecture and Algorithms of Raster
Space Data Service System, Surveying and Mapping Institute, 2003.
[4] Chen Guoliang, An Hong etc. Practice on parallel algorithms. Higher
Education Press, 2004.
[5] Kunz T. The influence of different workload description on a
heuristic load balancing scheme. IEEE Trans. On Software
Engineering, 1991, 17(7).
[6] Bignone F. Processing of Stereo Scanner: Stereo Plotter to Pixel
Factory. Stuttgart. 2003:14l-150
[7] Zhang Zuxun. From Digital Photogrammetry Workstation (DPW) to
Digital Photogrammetry Grid (DPGrid). Geomatics and Information
Science of Wuhan University. 2007: 565-571.