Advanced Software Technologies For Post Peta Scale Computing The Japanese Post Peta CREST Research Project Mitsuhisa Sato

Advanced Software Technologies for
Post Peta Scale Computing The

Japanese Post Peta CREST Research
Project Mitsuhisa Sato
Visit to download the full and correct content document:
https://textbookfull.com/product/advanced-software-technologies-for-post-peta-scale-
computing-the-japanese-post-peta-crest-research-project-mitsuhisa-sato/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Contagious Metaphor 1st Edition Peta Mitchell
https://textbookfull.com/product/contagious-metaphor-1st-edition-
peta-mitchell/
XcalableMP PGAS Programming Language From Programming

Model to Applications Mitsuhisa Sato
https://textbookfull.com/product/xcalablemp-pgas-programming-
language-from-programming-model-to-applications-mitsuhisa-sato/
Basic Japanese Second Edition Eriko Sato
https://textbookfull.com/product/basic-japanese-second-edition-
eriko-sato/
Post-Truth, Post-Press, Post-Europe: Euroscepticism And

The Crisis Of Political Communication 1st Edition Paul
Rowinski
https://textbookfull.com/product/post-truth-post-press-post-
europe-euroscepticism-and-the-crisis-of-political-
communication-1st-edition-paul-rowinski/
Practice Makes Perfect: Complete Japanese Grammar 2nd
Edition Eriko Sato
https://textbookfull.com/product/practice-makes-perfect-complete-
japanese-grammar-2nd-edition-eriko-sato/
Creating and Using Virtual Prototyping Software

Principles and Practices SEI Series in Software
Engineering 1st Edition Post
https://textbookfull.com/product/creating-and-using-virtual-
prototyping-software-principles-and-practices-sei-series-in-
software-engineering-1st-edition-post/
Advanced Computing and Communication Technologies

Jyotsna Kumar Mandal
https://textbookfull.com/product/advanced-computing-and-
communication-technologies-jyotsna-kumar-mandal/
Post-Conflict Education for Democracy and Reform:

Bosnian Education in the Post-War Era, 1995–2015 1st
Edition Brian Lanahan (Auth.)
https://textbookfull.com/product/post-conflict-education-for-
democracy-and-reform-bosnian-education-in-the-post-war-
era-1995-2015-1st-edition-brian-lanahan-auth/
The good research guide for small scale research

projects 5th Edition Denscombe
https://textbookfull.com/product/the-good-research-guide-for-
small-scale-research-projects-5th-edition-denscombe/
Mitsuhisa Sato Editor
Advanced Software
Technologies for
Post-Peta Scale
Computing
The Japanese Post-Peta CREST Research
Project
Advanced Software Technologies for Post-Peta
Scale Computing
Mitsuhisa Sato
Editor
Advanced Software
Technologies for Post-Peta
Scale Computing
The Japanese Post-Peta CREST Research
Project
123
Editor
Mitsuhisa Sato
RIKEN Center for Computational Science
Kobe, Japan
ISBN 978-981-13-1923-5 ISBN 978-981-13-1924-2 (eBook)

https://doi.org/10.1007/978-981-13-1924-2
Library of Congress Control Number: 2018959846
© Springer Nature Singapore Pte Ltd. 2019

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, express or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Computational Science, which enables us to explore uncharted fields of science

through applications of high performance computing, is a third paradigm of
scientific research which has become indispensable for the development of science
and technology of the twenty-first century.
Computational Science has been making great progress rapidly. The computa-
tional capability of supercomputers is now reaching on the verge of surpassing a
Peta-flops (1015 floating operations per second). This advance allows us in making
it possible to explore a wide range of phenomena through computer simulations
which was impossible in the past, for example, the creation and evolution of
the universe, the quantum origin of the functions of nano- and biomaterials
and its implication to life, and global climate changes. At the same time, the
development of intelligent information processing technologies is beginning to
enable a handling and analysis of enormous amount of data, which are accelerating
discoveries in many science disciplines, such as genome analyses, high energy
accelerator experiments, astronomical observations, and satellite observation of geo-
environments. In short, computational science is rapidly developing into a unified
framework in which large-scale modeling and simulation, large-scale data analysis,
and experiments/observation are integrated together to solve grand challenge issues
in various branches of science. As such, computational science has now become
an indispensable tool to better understand nature, life, and environment in order to
create a better future for mankind.
In Japan, the importance of computational sciences was explicitly noted in the
Japanese Government’s Third Basic Plan of Science and Technology (2006–2010),
and a national project for the development of “Next Generation Supercomputer” has
been carried out from 2006 as one of the key technologies of national importance.
The K computer has been produced as a major result of the project, achieving
world’s best performance in TOP500 list in 2011. Currently, the FLAGSHIP 2020
project for the development of the next Japanese flagship supercomputer has been
launched in 2014, and the development is under way. The operation for the public
service will be scheduled around 2020.
v
vi Preface
In 2010, the Japan Science and Technology Agency (JST) has initiated a research
area, titled “Development of System Software Technologies for Post-Peta Scale
High Performance Computing,” as a part of its Strategic Basic Research Program
(CREST). The project was named “JST CREST Post-Petascale software project.”
The research area of the project aimed at developing system software technologies
as well as related systems to be used for high performance computing systems
including the next generations of the Japanese flagship system, the K computer,
which is under development. Several researches and developments were conducted
for system software enabling us to exploit maximum efficiency and performance
from supercomputers composed of general purpose many-core processors as well
as accelerators such as GPUs and FPGA. From 2010 to 2012, 14 research teams
were selected, and 5-year research has been being conducted by each research
team. Many Japanese researchers and graduate students related to HPC have been
participating in these research teams.
This book describes the major outcomes obtained by research teams of the JST
CREST post-petascale software project.
Advanced system software is the key technology for post-petascale and exascale
high performance computing systems which will be developed in next decade. I
hope that the technologies developed in the JST CREST post-petascale software
project will play a role bridging to exascale computing and beyond through system
software technologies and advance the future computational science.
Kobe, Japan Mitsuhisa Sato

May 2018
Contents
1 JST CREST Post-petascale Software Project Bridging

to Exascale Computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Mitsuhisa Sato
2 ppOpen-HPC/pK-Open-HPC: Application Development
Framework with Automatic Tuning (AT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Kengo Nakajima, Masaharu Matsumoto, Masatoshi Kawai,
Takahiro Katagiri, Takashi Arakawa, Hisashi Yashiro,
and Akihiro Ida
3 Scalable Eigen-Analysis Engine for Large-Scale Eigenvalue
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Tetsuya Sakurai, Yasunori Futamura, Akira Imakura, and Toshiyuki
Imamura
4 System Software for Many-Core and Multi-core Architecture . . . . . . . . 59
Atsushi Hori, Yuichi Tsujita, Akio Shimada, Kazumi Yoshinaga,
Namiki Mitaro, Go Fukazawa, Mikiko Sato, George Bosilca,
Aurélien Bouteiller, and Thomas Herault
5 Highly Productive, High Performance Application
Frameworks for Post Petascale Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Naoya Maruyama, Takayuki Aoki, Kenjiro Taura, Rio Yokota,
Mohamed Wahib, Motohiko Matsuda, Keisuke Fukuda,
Takashi Shimokawabe, Naoyuki Onodera, Michel Müller,
and Shintaro Iwasaki
6 System Software for Data-Intensive Science. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Osamu Tatebe, Yoshihiro Oyama, Masahiro Tanaka, Hiroki Ohtsuji,
Fuyumasa Takatsu, and Xieming Li
7 Approaches for Memory-Efficient Communication Library
and Runtime Communication Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Takeshi Nanri
vii
viii Contents
8 A Development Platform for Embedded Domain-Specific

Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Shigeru Chiba, YungYu Zhuang, and Thanh-Chung Dao
9 Xevolver: A User-Defined Code Transformation Approach
to Streamlining Legacy Code Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Hiroyuki Takizawa, Reiji Suda, Daisuke Takahashi,
and Ryusuke Egawa
10 Numerical Library Based on Hierarchical Domain Decomposition . . 183
Ryuji Shioya, Masao Ogino, Yoshitaka Wada, Kohei Murotani,
Seiichi Koshizuka, Hiroshi Kawai, Shin-ichiro Sugimoto,
and Amane Takei
11 Advanced Computing and Optimization
Infrastructure for Extremely Large-Scale Graphs
on Post-peta-scale Supercomputers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Katsuki Fujisawa, Toyotaro Suzumura, Hitoshi Sato, Koji Ueno,
Satoshi Imamura, Ryo Mizote, Akira Tanaka, Nozomi Hata,
and Toshio Endo
12 Software Technology That Deals with Deeper Memory
Hierarchy in Post-petascale Era . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Toshio Endo, Hiroko Midorikawa, and Yukinori Sato
13 Power Management Framework for Post-petascale
Supercomputers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Masaaki Kondo, Ikuo Miyoshi, Koji Inoue, and Shinobu Miwa
14 Project CASSIA —Framework for Exhaustive and Large-Scale
Social Simulation— . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Itsuki Noda, Yohsuke Murase, Nobuyasu Ito, Kiyoshi
Izumi, Hiromitsu Hattori, Tomio Kamada, Hideyuki Mizuta,
and Mikio Takeuchi
15 GPU Accelerated Language and Communication Support
by FPGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Taisuke Boku, Toshihiro Hanawa, Hitoshi Murai, Masahiro Nakao,
Yohei Miki, Hideharu Amano, and Masayuki Umemura
Chapter 1
JST CREST Post-petascale Software
Project Bridging to Exascale Computing
Mitsuhisa Sato
Abstract JST CREST post-petascale software project aimed to establish software

technologies to explore extreme performance computing beyond petascale com-
puting, on the road to exascale computing. Several research and development has
been conducted for system software enabling us to exploit maximum efficiency
and reliability from high-performance computing systems composed of general-
purpose many-core processors as well as accelerators including GPGPU from
the second half of the 2010s to 2020s. The research topics cover from system
software such as programming languages, compilers, runtime systems, operating
systems, communication middleware, and file systems to application development
support software and ultra-large data processing software. As well as conventional
technologies for large-scale numerical computation, the project was also able to
address the storage technology required for big data processing, the complexity
of memory hierarchy, and the power problem. Exploration for the direction of
future high-performance computing is also an urgent and significant agenda in
our research area. This chapter presents the outline of JST CREST post-petascale
software project with brief description of the research topics, followed by summary
of results and achievements.
1.1 Trends of High-Performance Computing
High-performance computing systems used for cutting edge of advanced compu-

tational have reached several petaflops (a million billion calculations per second)
performance and will be targeted to the next generation of exascale systems as a
post-petascale system. Scientific applications require increasing performance for
industrial and societal general improvements.
M. Sato ()
RIKEN Center for Computational Science, Kobe, Japan
e-mail: msato@riken.jp
© Springer Nature Singapore Pte Ltd. 2019 1

M. Sato (ed.), Advanced Software Technologies for Post-Peta Scale Computing,
https://doi.org/10.1007/978-981-13-1924-2_1
2 M. Sato
Japan already has installed several petascale computers including the K com-
puter in RIKEN and now explores the evolution toward future exascale systems.
Following the end of the existing Moore law, the number of core per chip increases
and specialized hardware has been used to accelerate specific type of applications.
The number of processors and the interconnecting network increase also and
we have then to face new programming problems. Post-petascale systems and
future exascale computers are expected to have an ultra-large-scale and high-
performance architecture with nodes of many-core processors and accelerators.
To manage these ultra-large-scale parallel systems, we require new sophisticated
system software technologies, allowing to manage complex parallel computations
with huge distributed data, minimizing the energy consumption, and with fault-
resilient properties.
JST CREST post-petascale software project has been launched to establish
software technologies to explore extreme performance computing beyond petascale
computing, on the road to exascale computing. The ability to manage and program
these future high-performance systems efficiently is considered by all research
national agencies all along the world as a strategic and important issue.
1.2 Outline of JST CREST Post-petascale Software Project
CREST (Core Research for Evolutional Science and Technology) is a funding

program supported by JST (Japan Science and Technology Agency), which is an
independent public body of the Ministry of Education, Culture, Sports, Science
and Technology (MEXT), Japan. CREST is a funding program for team-based
research expected to produce outstanding results to lead scientific and technological
innovation. JST CREST post-petascale software project is one of CREST programs,
starting from 2012.
Our research area was funded under the title of “Development of System
Software Technologies for Post-petascale High-Performance Computing.” It was
launched by the first program research supervisor, Prof. Akinori Yonezawa, RIKEN,
from 2012 to 2014, and it was taken over to the second program research supervisor,
Prof. Mitsuhisa Sato, RIKEN, from 2015 to 2018.
The research area aimed at developing software technologies as well as related
systems to be used for high-performance computing in the post generations of the
Japanese national supercomputer, the K computer. Several research and develop-
ment has been conducted for system software enabling us to exploit maximum
efficiency and reliability from high-performance computing systems composed of
general-purpose many-core processors as well as accelerators including GPGPU
from the second half of the 2010s to 2020s. In addition to the system software
such as programming languages, compilers, runtime systems, operating systems,
communication middleware, and file systems, application development support
software and ultra-large data processing systems are also the targets for our research
and development.
1 JST CREST Post-petascale Software Project Bridging to Exascale Computing 3
The calls for project proposals were issued at every year from 2010 to 2012, and
finally the 14 projects were adopted: 5 projects in the first year, 5 projects in the
second year, and 4 projects in the third year as a result of peer reviews by advisory
committee for the project proposals submitted as responses to the calls. The duration
of each project is 5.5 years. The total budget from 2010 to 2017 was about 60 M
USD. Table 1.1 shows the adopted teams of JST CREST post-petascale software
project.
The advisory committee was organized to advice the research direction of the
project teams by following members:
• Mutsumi Aoyagi, Professor, Research Institute for Information Technology,
Kyushu University (assumption of office period: Oct. 2010–Dec. 2014)
• Yutaka Ishikawa, Project Leader, Flagship 2020 Project, RIKEN Advanced
Institute for Computational Science
• Kouichi Kumon, Member of the board, Fujitsu Laboratories Ltd.
• Kenji Kono, Professor, Keio University
• Hiroaki Kobayashi, Director, Cyberscience Center, Tohoku University
• Mitsuhisa Sato, Professor, Department of Computer Science, University of
Tsukuba (assumption of office period: Oct. 2010–Mar. 2015)
• Shinji Shimojo, Professor, Cybermedia Center, Osaka University
• Keiko Takahashi, Director, Center for Earth Information Science and Technol-
ogy, Japan Agency for Marine-Earth Science and Technology
• Yaoko Nakagawa, Senior Project Manager, Center for Technology Innovation-
Information and Telecommunications, Research & Development Group, Hitachi
Ltd.
• Hiroshi Nakashima, Professor and Director, Academic Center for Computing and
Media Studies, Kyoto University
• Junichiro Makino, Professor, Department of Planetology, Graduate School of
Science, Kobe University
• Satoshi Matsuoka, Professor, Global Scientific Information and Computing
Center, Tokyo Institute of Technology
To carry the projects out strategically, the policies and goals of management in
our research area were defined as follows:
(1) Research and development of highly functional and reliable system software
for sustainable high-performance computing technologies to solve social and
scientific problems
Numerical simulation and data analysis utilizing ultra-large-scale computational
resources and storage have dramatically been increasing the importance of its role
for modern science and technology. In response to this fact, the USA, Europe, China,
and Japan are racing to develop the next generation of supercomputer – exascale
systems – capable of a million trillion calculations a second by around 2020. In
our country, the project, FLAGSHIP2020, has been launched to develop the next
flagship system following the K computer.
4 M. Sato
Table 1.1 Research teams of JST CREST post-petascale software project

PI, title/affiliation Research theme
1. Research teams adopted in 2010
Tetsuya Sakurai Professor, University of Development of an Eigen-Supercomputing
Tsukuba Engine Using a Post-Petascale Hierarchical
Model
Osamu Tatebe Professor, University of System Software for Post-Petascale
Tsukuba Data-Intensive Science
Kengo Nakajima Professor, University of ppOpen-HPC: Open-Source Infrastructure for
Tokyo Development and Execution of Large-Scale
Scientific Applications on Post-Petascale
Supercomputers with Automatic Tuning (AT)
Atsushi Hori Senior Researcher, RIKEN Parallel System Software for Multi-core and
Many-core
Naoya Maruyama Research Team Leader, Highly Productive, High-Performance
RIKEN Application Frameworks for Post-petascale
Computing
Ryuji Shioya Professor, Toyo University Development of a Numerical Library Based on
Hierarchical Domain Decomposition for
Post-petascale Simulation
Hiroyuki Takizawa Associate Professor, An Evolutionary Approach to Construction of a
Tohoku University Software Development Environment for
Massively Parallel Heterogeneous Systems
Shigeru Chiba Professor, The University of Software Development for Post-petascale
Tokyo Supercomputing: Modularity for
Supercomputing
Takeshi Nanri Associate Professor, Kyushu Development of Scalable Communication
University Library with Technologies for Memory Saving
and Runtime Optimization
Katsuki Fujisawa Professor, Kyushu Advanced Computing and Optimization
University Infrastructure for Extremely Large-Scale
Graphs on Post-petascale Supercomputers
Toshio Endo Associate Professor, Tokyo Software Technology that Deals with Deeper
Institute of Technology Memory Hierarchy in Post-petascale Era
Masaaki Kondo Associate Professor, The Power Management Framework for
University of Tokyo Post-petascale Supercomputers
Itsuki Noda Principal Research Manager, Framework for Administration of Social
AIST Simulations on Massively Parallel Computers
Taisuke Boku Professor, University of Research and Development on Unified
Tsukuba Environment of Accelerated Computing and
Interconnection for Post-petascale Era
Under such circumstances, in order to keep the ability to solve several social and
scientific problems by making full use of the supercomputer, it is extremely signif-
icant not only to have technologies to execute a large-scale programs (simulation
programs, data analysis programs, etc.) efficiently on current supercomputers but
also to perform researches on software technologies to bring out the full potential
of the next generation of high-performance systems and technologies in the future.
The sustainable progress of high-performance computing software is essential for
the sustainable contribution to science and technology advancement and innovation
by the high-performance computing.
In our research area, we are developing a high-performance, highly functional,
and high-reliability system software including programming language, compiler,
runtime system, operating system, communication middleware, file system, numer-
ical calculation library, job management system, and ultra-large-scale data process-
ing system software. In addition, from the viewpoint of sustainable contribution by
the high-performance computing mentioned above, it should be not only academic
research to demonstrate simple novel idea and its feasibility but the development of
actual usable software. It is requested for the teams to make the developed software
used in related community and is also to be emphasized in the evaluation.
(2) Exploration for the direction of future high-performance computing research
and development
At the time of calls for the project proposal, it was not clear what kind
of technologies would be used for the future next-generation high-performance
computing systems, general-purpose many-core processor, or specialize hardware
such as GPU. So, we requested to indicate the target system of the research and how
to make the proposed software executed efficiently in the proposed target systems.
The accepted research project teams were supposed to publish their research results
as open-source software.
As the duration of the adopted project was 5 years, in intermediate evaluation at
the third year, the team was requested to demonstrate that the developed software
will be usable realized at the end of the project. By this request, we expected that
the developed software would show the direction of system architecture of the
future high-performance computing systems and be used actually in these systems
including the next-generation system of the K computer or GPU-based accelerator
systems. For this purpose, we aimed for international collaboration and industry-
academia collaboration while sharing information with overseas researchers and
companies.
Furthermore, when each research team is about to be finished (from 2016), we
expected that the practical and usable software developed by the teams as a result of
the project is used to enable advanced large-scale simulation and valuable prediction
using large-scale data in a wide range of science and technology fields.
(3) Fostering of the next-generation leaders in the field of high-performance
computing
In order to keep the ability of sustainable development of high-performance com-
puting systems and promote the direction of future high-performance computing
research and development, human resource who can bear it is indispensable. We
actively appointed young researchers who were expected to be in the future at the
selection of the projects and took into account the development of young research
6 M. Sato
leaders who will be responsible for research and development in the next generation
of high-performance computing technology in Japan, by emphasizing autonomy in
the planning, making teams and management.
1.3 Research Topics of the Project
In this section, the research topics and some highlights of the adopted projects
are described briefly. As mentioned above, the research topics of our research
area covers from system software such as programming languages, compilers,
runtime systems, operating systems, communication middleware, and file systems to
application development support software and ultra-large data processing systems.
Several teams carried researches on programming models and frameworks for
post-petascale systems.
Maruyama’s team (Chap. 5) has been working on several programming frame-
works and libraries to make programming easy for the next generation of high-
performance systems. They developed Daino, a high-level framework for parallel
and efficient AMR on GPUs, and investigated effectiveness of high-level program-
ming techniques such as Gridtools for global climate model simulation.
Endo’s team (Chap. 12) was working on problems for recent trends on deeper
memory hierarchy. For exascale high-performance systems, the “Memory Wall”
problem will become even more severe. His team promotes research toward this
problem via co-design approach among application algorithms, system software,
and architecture. They have developed several libraries to make it easy to use the
system of deeper memory hierarchy.
Chiba’s team (Chap. 8) focused on productive programming models for post-
petascale systems. A single general programming language or framework that
covers all subjects will not be feasible for post-petascale supercomputing. Their goal
is to apply modern techniques for software engineering and theoretical foundations
of programming languages, such as software modularization, to high-performance
computing.
Takizawa’s team (Chap. 9) proposed the Xevolver framework which takes an
evolutionary approach to incremental migration of existing software resources to
new systems. The goal is to establish an effective migration path to new algorithms,
implementation schemes, and programming environments for massively parallel
and heterogeneous systems in an upcoming extreme-scale computing era.
Boku’s team (Chap. 15) has been working on TCA (tightly coupled accelera-
tors) by short-latency communication among GPUs over nodes to achieve strong
scalability on next-generation accelerated computing and its programming model.
They developed prototype system named PEACH2 which implemented by FPGA
for flexible design and suitability for PCIe interface. As its extension, they proposed
aggressive solution named AiS (Accelerator in Switch) to exploit high potential of
recent FPGA. To solve the low productivity on programming by MPI plus CUDA
style, we are developing new language framework XcalableACC (XACC) where

OpenACC description is involved to XcalableMP PGAS language.
The teams of system software cover researches from storage technology to
communication middleware.
Tatebe’s team (Chap. 6) carried researches on the next-generation storage
technologies for post-petascale computing. The performance gap between CPU and
storage is growing wider and wider. Distributed file system using compute-node
local storage is promising to fill the gap. His team promotes node-local distributed
file system and parallel and distributed execution framework for the file system.
Hori’s team (Chap. 4) proposed a new process model for many-core. As in MPI
program, the multiprocess model allows each process to own a private address space,
though processes can allocate explicit shared memory regions. On the other hand,
the multi-threaded model shares all address space by default, though threads can
explicitly move data to thread-private storage. His team proposes a third model
called Process-in-Process (PiP), where multiple processes are mapped into a single
virtual address space, which may make use of many-core processor efficiently.
Nanri’s team (Chap. 7) has been developing a PGAS-based communication
library, ACP, and its applications. Memory efficiency of communication libraries
is becoming important issue in large-scale parallel systems, where the number of
processes is expected to be tens of millions. ACP enables both high-performance
and low memory consumption for post-petascale systems.
Power consumption is now becoming a first class design constraint for devel-
oping future post-petascale computing systems. To achieve exa-flops-level perfor-
mance with realistic power provisioning of 20–30 megawatts, significant power
efficiency improvement over today’s supercomputers is necessary.
Kondo’s team (Chap. 13) has been working on this power issue in the next
generation of high-performance computing system. In order to maximize effective
performance within a power constraint, they proposed the power-constraint adaptive
system design (P-CAS), which allows the system’s peak power to exceed maximum
power provisioning with adaptively controlling power-knows equipped in hardware
components so that effective power consumption at runtime is under the power
constraint.
Researches on numerical algorithms and libraries and software which support
high-performance scientific applications also have been carried out.
Sakurai’s team (Chap. 3) aimed to develop a massively parallel eigenvalue solver,
eigen-supercomputing engine for post-petascale systems. Eigenvalue solver is one
of the most important algorithms in several computational science applications. The
eigen-engine is to be developed based on newly designed algorithms that are suited
to the hierarchical architecture in post-petascale systems.
Nakajima’s team (Chap. 2) has been developing “ppOpen-HPC,” an open-source
infrastructure for development and execution of optimized and reliable simulation
code on post-petascale parallel computers based on many-core architectures, and it
consists of various types of libraries, which cover general procedures for scientific
computation.
8 M. Sato
Shioya’s team (Chap. 10) has been developing an open-source software called
ADVENTURE system. The ADVENTURE system is a general-purpose parallel
finite element analysis system and can simulate a large-scale analysis model with
supercomputer like the Earth Simulator or K computer. In the ADVENTURE
system, HDDM (hierarchical domain decomposition method), a very effective
technique for large-scale analysis, was developed. They aimed to develop a numer-
ical library based on HDDM that is extended to pre- and post-processing parts,
including mesh generation and visualization of large-scale data, for the post-
petascale simulation.
Fujisawa’s team (Chap. 11) has been developing advanced computing and
optimization infrastructures for extremely large-scale graphs on post-petascale
supercomputers. The large-scale graph analysis has attracted significant attention as
a new application of the next-generation supercomputer. It is, however, extremely
difficult to realize a high-speed graph processing in various application fields
by utilizing previous methods. They aimed to develop advanced computing and
optimization infrastructures for extremely large-scale graphs on the next-generation
supercomputers.
Noda’s team (Chap. 14) has been working on Project CASSIA (Comprehensive
Architecture of Social Simulation for Inclusive Analysis) which aims to develop a
framework to administer to execute large-scale multiagent simulations exhaustively
to analyze socially interactive systems. The framework realizes engineering envi-
ronment to design and synthesize social systems like traffics, economy, and politics.
1.4 Results and Achievements
The JST CREST post-petascale project has ended at the end of March 2018. At
the end, the final evaluation was done, and the overall results were evaluated as
“excellent” by the evaluation committee.
As planned at the beginning of the project, the goals described in Sect. 1.2 have
been achieved.
Regarding research and development of highly functional and reliable system
software for sustainable high-performance computing technologies, at first of all,
the researchers of each team have published many excellent technical papers and
valuable software as well as research presentations at prominent academic societies.
Furthermore, making use of the developed software, large-scale applications are
executed in many practical application fields from tsunami simulation, weather sim-
ulation to graph analysis, economic simulation, resulting in valuable contribution
to solving social and scientific problems. In particular, it was showing not only
the conventional large-scale numerical simulation but also the application of high-
performance computing to new and important field such as big data analysis and
social simulation. And, in the team having close to the real application, collaboration
and joint research with the industries were actively carried out.
Exploration for the future direction of future high-performance computing

research and development is an urgent and significant agenda in our research area.
Although the subjects of this research area were mainly focused on software,
at the time of setting research area, there was discussion on whether or not
to conduct hardware research. During the project, the research done by many
researchers has caught the trend of large-scale systems such as with many-core
processors or accelerators from software’s point of view, and consequently the
researches is applied to the current systems, indicating the direction to the future
high-performance computing systems. As well as conventional technologies for
large-scale numerical computation, the project was also able to address the storage
technology required for big data processing, the complexity of memory hierarchy,
and the power problem. Furthermore, we have handled quickly to new trends such
as FPGA after the start of the project.
As mentioned in Sect. 1.2 the project for the development of the next Japanese
flagship supercomputer, the Post-K (Flagship 2020 project), was started from
FY2014, and the development is under way. The operation for the public service
will be scheduled around 2020. Many software developed in our project, such as
numerical libraries and power control software, will be actually deployed and used
for the post-K.
From the viewpoint of fostering human resources of the leaders in the next
generation of high-performance computing, many excellent young researchers were
produced as many young researchers in the project are awarded such as the “Young
Encouragement Prize” and the “Research Award” in the community. It is thought as
a big achievement that the research team took the initiative to organize numerous
international and domestic symposiums, workshops, and seminars related to the
research area, resulting in improvement of international awareness of leaders and
researchers of the teams.
As a major international collaboration, the project collaborated with SPPEXA
with Germany and France. Under the framework of the “Software for Exascale
Computing (SPPEXA)” program which is implemented by DFG (Germany), JST
(Japan) and ANR (France) agreed to support trilateral projects for high-performance
computing. Some of teams have been awarded as a SPPEXA partner supported
by JST. The research duration of the awarded teams was extended by 1 or 2
years, and the research results could be further improved through the international
collaborative research.
The JST CREST post-petascale software project was expected to play a role
bridging to post-petascale and exascale computing through system software. Cur-
rently, research and development of supercomputers aiming at the exascale comput-
ing is going on in several countries. It is important to continue several dissemination
activities as well as for post-K. We hope that the technologies and software
developed in the project should be used in several applications even after the
research period and further drive the direction of research for the future high-
performance computing.
Chapter 2
ppOpen-HPC/pK-Open-HPC:
Application Development Framework
with Automatic Tuning (AT)
Kengo Nakajima, Masaharu Matsumoto, Masatoshi Kawai,

Takahiro Katagiri, Takashi Arakawa, Hisashi Yashiro, and Akihiro Ida
Abstract ppOpen-HPC and pK-Open-HPC are open source infrastructures for

development and execution of large-scale scientific applications on post-petascale
(pp) supercomputers with automatic tuning (AT). Both of ppOpen-HPC and pK-
Open-HPC focus on parallel computers based on many-core architectures and
consist of various types of libraries covering general procedures for scientific
computations. The source code, developed on a PC with a single processor, is linked
with these libraries, and the parallel code generated is optimized for post-petascale
systems. In this article, recent achievements and progress of the ppOpen-HPC and
pK-Open-HPC project are summarized.
2.1 Overview of ppOpen-HPC
“ppOpen-HPC [1, 2]” is an open source infrastructure for development and exe-
cution of optimized and reliable simulation code on post-petascale (pp) parallel
computers based on many-core architectures, and it consists of various types of
libraries, which cover general procedures for scientific computation. Source code
K. Nakajima () · M. Matsumoto · M. Kawai · A. Ida

The University of Tokyo, Tokyo, Japan
e-mail: nakajima@cc.u-tokyo.ac.jp; matsumoto@is.s.u-tokyo.ac.jp; masatoshi.kawai@riken.jp;
ida@cc.u-tokyo.ac.jp
T. Katagiri
Nagoya University, Nagoya, Aichi, Japan
e-mail: katagiri@cc.nagoya-u.ac.jp
T. Arakawa
Research Organization for Information Science and Technology (RIST), Tokyo, Japan
e-mail: arakawa@rist.jp
H. Yashiro
RIKEN Advanced Institute for Computational Science (AICS), Kobe, Hyogo, Japan
e-mail: h.yashiro@riken.jp
© Springer Nature Singapore Pte Ltd. 2019 11

M. Sato (ed.), Advanced Software Technologies for Post-Peta Scale Computing,
https://doi.org/10.1007/978-981-13-1924-2_2
12 K. Nakajima et al.
developed on a PC with a single processor is linked with these libraries, and the
parallel code generated is optimized for post-petascale systems. The target post-
petascale system is many-core-based systems, such as the Oakforest-PACS system
operated by JCAHPC [3]. ppOpen-HPC is part of a 5-year project (FY.2011–2015)
spawned by the “Development of System Software Technologies for Post-petascale
High-Performance Computing [4]” funded by JST-CREST.
The framework covers various types of procedures for scientific computations,
such as parallel I/O of datasets, matrix assembly, linear solvers with practical
and scalable preconditioners, visualization, adaptive mesh refinement, and dynamic
load balancing, in various types of computational models, such as FEM (finite
element method), FDM (finite difference method), FVM (finite volume method),
BEM (boundary element method), and DEM (discrete element method). Automatic
tuning (AT) technology enables automatic generation of optimized libraries and
applications under various types of environments. We release the most updated
version of ppOpen-HPC as open source software every year in November (2012–
2015), which is available at the home page of ppOpen-HPC [2].
In 2016, the team of ppOpen-HPC joined ESSEX-II (Equipping Sparse Solvers
for Exascale) project (Leading P.I. Professor Gerhard Wellein (University of
Erlangen-Nuremberg)) [5]), which is funded by JST-CREST and the German
DFG Priority Programme 1648 “Software for Exascale Computing” (SPPEXA)
[6] under Japan (JST)-Germany (DFG) collaboration until FY2018. In ESSEX-II,
we develop pK-Open-HPC (extended version of ppOpen-HPC, framework for exa-
feasible applications), preconditioned iterative solvers for quantum sciences, and a
framework for automatic tuning (AT) with performance model.
Original ppOpen-HPC includes the following four components (Fig. 2.1):
• ppOpen-APPL Frameworks for development of applications by FEM, FDM,
FVM, BEM, and DEM (Fig. 2.2) [1, 2]
User’s Program
Framework for
pK-Open-FVM ppOpen-APPL FEM FDM FVM BEM DEM Application
Development
pK-Open-SOL ppOpen-MATH MG GRAPH VIS MP Math Library
pK-Open-AT Automatic
ppOpen-AT STATIC DYNAMIC
Tuning (AT)
pK-Open-HPC ppOpen-SYS COMM FT

System
Software
ppOpen-HPC
Optimized Application with

Optimized ppOpen-APPL, ppOpen-MATH
Fig. 2.1 Overview of ppOpen-HPC and pK-Open-HPC [1, 2]

2 ppOpen-HPC/pK-Open-HPC: Application Development Framework. . . 13
FEM FDM FVM

Finite Element Method Finite Diﬀerence Method Finite Volume Method
BEM DEM
Boundary Element Method Discrete Element Method
Fig. 2.2 Target applications of ppOpen-HPC and pK-Open-HPC
• ppOpen-MATH Math library

• ppOpen-AT Capability of automatic tuning
• ppOpen-SYS System software
In, pK-Open-HPC, we are focusing on the following issues and developed new
libraries (Fig. 2.2)
• pK-Open-FVM Frameworks for development of applications by FVM with AMR
(adaptive mesh refinement)
• pK-Open-SOL Robust and efficient preconditioned iterative solvers
• pK-Open-AT Capability of automatic tuning
In this article, we introduce recent developments and activities in ppOpen-HPC
and pK-Open-HPC. The structure of this article with information of authors is as
follows:
1. Overview of ppOpen-HPC (Kengo Nakajima)
2. Development of pK-Open-FVM (Masaharu Matsumoto)
3. Robust and massively parallelized preconditioner for quantum systems
(Masatoshi Kawai) (related to pK-Open-SOL)
4. Automatic tuning (AT) in ppOpen-HPC and pK-Open-HPC (Takahiro Katagiri)
5. Development of a multi-physics coupler ppOpen-MATH/MP (Takashi Arakawa,
Hisashi Yashiro)
6. Efficient structures of H-matrices on distributed memory computer systems
(Akihiro Ida) (related to ppOpen-APPL/BEM and pK-Open-SOL)
7. Summary (Kengo Nakajima)
2.2 Development of pK-Open-FVM
Adaptive mesh refinement (AMR) technique [7, 8] can provide efficient numerical
calculation by generating hierarchical layers with different cell sizes at the local
regions where high resolution is needed. It is, however, generally difficult to
implement the AMR treatment in conventional simulation codes discretized by finite
volume method, finite difference method, and so on. To overcome this problem, a
block-based AMR framework, pK-Open-FVM, with which the AMR technique can
be relatively easily ported to generic simulation programs which hire the uniform
cell system has been developed. In this framework, the AMR technique is applied
to a block-structured region consisting of the fixed number of cells, as shown in
Fig. 2.3. A generic simulation program using uniform cell size can be implemented
in each block in the AMR framework [9]. Once a situation occurs where high
resolution is needed in a local region, the corresponding block-structured region is
divided into eight for three-dimensional case and new block-structured regions with
uniform cell with half size of the original one are generated. In the AMR framework,
the simulation domain is divided into multiple sub-domains, and they are assigned to
a number of processes for parallel computing using MPI. A sub-domain is composed
of multiple block-structured regions each of which has the fixed number of grids.
When high resolution is required at a certain region in the sub-domain, a block-
structured region with refined cells, which is called child block, is locally created.
For the application of the AMR framework, (1) Vlasov-Poisson plasma simu-
lations, which are composed of 1D uniform direction and 1D AMR direction, (2)
AMR plasma particle simulations for the development of reactive plasma deposition
equipment, and (3) particle-based sugarscape model simulations are introduced as
follows.
Fig. 2.3 Example of

block-AMR
2.2.1 Application to Vlasov-Poisson Simulation
On large-scale computing, high-efficient calculation by AMR technique is one of the

most attractive features. In particular, high-dimensional (≥4 dimension) simulation
is suitable for the AMR simulation. Numerical simulation of Vlasov equation is
one of the famous high-dimensional simulations in the plasma physics field. In
Vlasov equation, a collision term on the right hand side of Boltzmann equation
is eliminated.
∂f ∂f F ∂f
+v· + · =0
∂t ∂x m ∂v
Here, f = f (x, v, t) is velocity distribution function, x, v are position, velocity, m
is particle mass, and F is external force, respectively. This equation includes position
coordinate, 3 dimension, + velocity coordinate, 3 dimension = total 6 dimension.
It is difficult to solve this equation under the condition with six-dimensional
coordinate because very large amount of memory is needed. Therefore, the function
of AMR framework is extended to combine uniform mesh direction and AMR
direction, and the extended function is applied to two-dimensional Vlasov-Poisson
system (one-dimensional space and one-dimensional velocity). In this simulation,
spatial direction is set to uniform 1D mesh because Poisson equation is solved to
estimate electric field as an external force by fast Fourier transform, and velocity
direction is set to adaptive mesh. Figure 2.4 shows the contour figure of distribution
function at t = 0, 128, 176 s under the typical simulation conditions. The model
calculated here is called “beam instability.” In this model, twin electron beams with
Fig. 2.4 Application of AMR to 1D-1D Vlasov-Poisson simulation

different components of velocity generate a vortex and combine into single electron
beam as time proceeds. As criteria for mesh refinement, the gradient of velocity
is set. When adaptive mesh direction (namely, velocity) have large gradient, mesh
refinement occurs, and high-resolution simulation can be conducted. The lower
right of Fig. 2.4 shows enlarged view of vortex. Lv. 3 block is generated where
the gradient of velocity is large. The execution time of the AMR simulation can
reduce 30% compared to the simulation with uniform mesh at velocity direction.
2.2.2 AMR Particle Simulation for the Development

of Reactive Plasma Deposition Equipment
Plasma particle/fluid hybrid simulation code for the development of reactive plasma
deposition (RPD) equipment (Fig. 2.5) has been developed by using pK-Open-
FVM [10]. In this simulation, the interaction of electromagnetic fields and plasma
particles can be simulated by calculating electromagnetic fields defined on the
computational mesh and mesh-free plasma particles, simultaneously. In such a
plasma particle simulation, in general, several hundreds of particles per one mesh
are needed because the influence of statistical error depending on the number of
particles is reduced.
In a conventional simulation with uniform mesh, the statistical error increases
with increasing the difference of plasma density in the computational domain. In
order to reduce statistical error, much more particles are needed and the execution
time of simulation is also increased. On the other hand, by using the AMR
Fig. 2.5 RPD equipment

Fig. 2.6 Distribution of ion

density and computational
mesh
Fig. 2.7 The number of

particles used in the
simulation with/without
AMR technique
framework, cell size can be adjusted according to the number of particles (plasma
density) (Fig. 2.6). That is, cell size becomes fine at high plasma density region
or becomes coarse at low plasma density region. Furthermore, Fig. 2.7 shows the
comparison of the number of particles used in the simulation with/without AMR
technique. As a result, the total number of particles and execution time can be
reduced by using AMR technique. The simulation using indium tin oxide as a
material source which can’t be conducted in conventional code can evaluate an
experimental result (dependency of coil current), quantitatively.
2.2.3 Application to Particle-Based Sugarscape Model
Test simulations by adopting the sugarscape model which is proposed for the
simulation for an artificial society by using many agents representing inhabitants
in a certain area are conducted [11]. The inhabitants are treated as a bunch of
particles and the sugar amount are assigned at each grid as the environment in a
Fig. 2.8 Initial distribution

of sugar amount in the color
map and inhabitants with dots
two-dimensional simulation domain. In the simulation, initially, two peaks of sugar

are placed and randomly distribute the inhabitants in Fig. 2.8. In the PSS model, the
inhabitant agents can move freely in the computational domain. The following are
the equations of motion to be solved:
∂x
∂t = v
∂v
∂t = mFs − nFd
Here, Fs is the spatial gradient of the sugar amount obtained at inhabitant

position, and Fd is the spatial gradient of the agent density obtained at inhabitant
position. Therefore, from these equations, the inhabitant agents receive forces to the
direction of high-sugar density region and low agent density region, as shown in
Fig. 2.9.
2.3 Robust and Massively Parallelized Preconditioner for

Quantum Systems
2.3.1 Objective
The objective of this research is to develop an iterative solver with robustness and
exascale parallelism.
In the field of quantum systems, researchers are very interested in electrical,
structural, and chemical properties of materials such as graphene and topological
insulators. To clarify the physical properties of the target materials, we must
solve the generalized eigenvalue problems. A collaborating project of ppOpen-HPC
Fig. 2.9 Distribution of

inhabitants number density
and refinement block
Fig. 2.10 Example of

SS/FEAST method Arbitrary area
Integral points
(founded by CREST) and ESSEX-II (founded by SPPEXA) focuses on solving the

problems using the Sakurai-Sugiura (SS) [12] or FEAST [13] method. By using
these methods, we can calculate eigenvalue-eigenvector pairs that lie in an arbitrary
area (Fig. 2.10). The area is described using a liner integral, and we must solve a sys-
tem of simultaneous linear equations (SLEs) on integral points. Coefficient matrices
Az of the SLEs on each integral point are calculated as Az = zB – A. Then, matrices A
and B are defined by an application. These matrices derived from our target applica-
tions are sparse and large scale. The values of z are determined from the coordinate
of each integral point. There is a possibility that ill-conditioned coefficient matrices
are provided. Therefore, the requirements of the iterative solver are:
• Robustness
• Massive parallelism
In this study, we select an ILU preconditioned Krylov subspace method as the
solver. In addition, we applied regularization methods to incomplete LU (ILU)
preconditioner for the robustness [14]. For the massive parallelism, we proposed
hierarchical parallelization for multicoloring algorithms [15].
2.3.2 Regularizations for Robustness
For robustness of the ILU preconditioned Krylov subspace method, we applied

two regularization methods. The first method is a blocking technique, and the
second is diagonal shifting. If conventional ILU factorization methods are applied
to real-world applications, problems such as accumulation of rounding error or a
breakdown of factorization are known to occur. The applied regularization methods
increase the robustness for overcoming such problems.
2.3.2.1 Blocking Technique
The ILU preconditioner using a blocking technique is a well-known approach to

improving the convergence. In our approach, we focus on the increasing robustness
as well. The breakdown of ILU factorization occurs if diagonal entries of the target
matrix are small. This is because the rounding errors accumulate by dividing the off-
diagonal entries by diagonal entries. By applying a blocking technique, we obtain
larger diagonal submatrices. The diagonal blocks include off-diagonal entries, as
shown in Fig. 2.11.
2.3.2.2 Diagonal Shifting
This is a strong and direct method to make diagonal dominant matrices. If we

apply diagonal shifting, the matrix Ãz , which should be incompletely factorized,
is calculated as:
Ãz = AZ + αI I = identical matrix
Then, the constant value α is decided by the user. α is then added to the diagonal
entries of the target matrix AZ , making it the diagonal dominant matrix Ãz . The
larger value of α makes a more diagonal dominant matrix. However, the effect of
ILU preconditioner gets smaller because of larger difference between AZ and Ãz .
The best parameter α depends on the target application and shift value z.
Small entry Non-small entry
Blocking
Small diagonal entries Diagonal blocks including

non–small entries
Fig. 2.11 The effect of the blocking technique
2.3.2.3 Numerical Evaluation
For numerical evaluations, we prepared 128 datasets. There are two types of
different model. One model simulates the graphene, and the other simulates the
topological insulator. Each model has four kinds of degrees of freedom (DoF) data,
and each set of data has 16 data shifts. These models are symmetrical and complex
values. For solving them, we implemented the block IC preconditioned conjugate
orthogonal conjugate gradient (COCG) method. The BIC-COCG solver converged
when the relative residual norm was less than 10−7 . If we applied only a blocking
technique with a small size, we would solve only 64 datasets. However, by applying
a larger block size (64) and (0.0, 1.0) data shifts, we solved all datasets (Fig. 2.12).
2.3.3 Hierarchical Parallelization of Multicoloring Algorithms

for Massive Parallelism
The multicoloring is often used for parallelization of ILU factorizations and

forward-backward substitutions. However, the coloring algorithms themselves are
not parallelized in most reported research. To support the exascale system, we must
parallelize them. In addition, the coloring algorithms have a significant impact on
the convergence rate and performance of the ILU preconditioned Krylov subspace
method. If local colorings are applied to parallelizing the ILU preconditioners
for each MPI process, the convergence rate degrades as the number of processes
increases. The multicoloring with hierarchical parallelization has a small influence
on the convergence rate, because the proposed method provides global coloring.
Fig. 2.12 Numerical

evaluation of the regularized
IC-COCG
Fig. 2.13 Example of five-colored matrix for four processes
2.3.3.1 Parallelization of the ILU Preconditioner with Multicoloring
Generally, the ILU factorization and forward-backward substitution have sequen-

tiality. To parallelize them, the multicoloring algorithms are widely applied. Figure
2.13 shows the example of a “three-colored matrix for four processes” paralleliza-
tion. The diagonal submatrices colored with the same color have no relationships.
Another random document with
no related content on Scribd:
Cimbex she finds a similar arrangement, but there are ten chambers,
and no aorta.
The dorsal vessel is connected with the roof of the body by some
short muscles, and is usually much surrounded by fat-body into
which tracheae penetrate; by these various means it is kept in
position, though only loosely attached; beneath it there is a delicate,
incomplete or fenestrate, membrane, delimiting a sort of space
called the pericardial chamber or sinus; connected with this
membrane are some very delicate muscles, the alary muscles,
extending inwards from the body wall (b, Fig. 72): the curtain formed
by these muscles and the fenestrate membrane is called the
pericardial diaphragm or septum. The alary muscles are not directly
connected with the heart.
Fig. 72.—Dorsal vessel (c), and alary muscles (b), of Gryllotalpa (after
Graber); a, aorta. N.B.—The ventral aspect is here dorsal, and
nearly the whole of the body is removed to show these parts.
Fig. 73.—Diagram of transverse section of pericardial sinus of

Oedipoda coerulescens. (After Graber, Arch. Mikr. Anat. ix.) H,
heart; s, septum; m, muscles—the upper suspensory, the lower
alary.
It has been thought by some that delicate vessels exist beyond the
aorta through which the fluid is distributed in definite channels, but
this does not appear to be really the case, although the fluid may
frequently be seen to move in definite lines at some distance from
the heart.
There is still much uncertainty as to some of the details of the action

of the heart, and more especially as to the influence of the alary
muscles. The effect of the contraction of these must be to increase
the area of the pericardial chamber by rendering its floor or septum
less arched, as shown in our diagram (Fig. 73), representing a
transverse section through the pericardial chamber, H being the
dorsal vessel with m its suspensory muscles, and s its septum, with
m the alary muscles. The contraction of these latter would draw the
septum into the position of the dotted line, thus increasing the area
of the sinus above; but as this floor or septum is a fenestrated
structure, its contraction allows fluid to pass through it to the
chamber above; thus this arrangement may be looked on as a
means of keeping up a supply of fluid to the dorsal vessel, the
perforated septum, when it contracts, exerting pressure on the
tissues below; these are saturated with fluid, which passes through
the apertures to the enlarged pericardial chamber.
Some misconception has prevailed, too, as to the function of the

pericardial chamber. This space frequently contains a large quantity
of fat-body—pericardial tissue—together with tracheae, and this has
given rise to the idea that it might be lung-like in function; but, as
Miall and Denny[58] have pointed out, this is erroneous; the tissues in
Insects have their own ample supplies of air. It has also been
supposed that the alary muscles cause the contraction of the heart,
but this is not directly the case, for they are not attached to it, and it
pulsates after they have been severed. It has been suggested that
the contractions of this vessel are regulated by small ganglia placed
on, or in, its substance. However this may be, these contractions
vary enormously according to the condition of the Insect; they may
be as many, it is said, as 100 or more in a minute, or they may be
very slow and feeble, if not altogether absent, without the death of
the Insect ensuing.
The expulsion of the blood from the front of the dorsal vessel seems
to be due to the rhythm of the contraction of the vessel as well as to
its mechanical structure. Bataillon says,[59] confirming an
observation of Réaumur, that at the period when the silkworm is
about to change to the chrysalis condition, the circulation undergoes
periodical changes, the fluid moving during some intervals of about
ten minutes' duration in a reversed direction, while at other times the
blood is expelled in front and backwards simultaneously, owing
apparently to a rhythmical change in the mode of contraction of the
dorsal vessel.
As the dorsal vessel consists of a number of distinct chambers, it

has been suggested that there is normally one of these for each
segment of the body; and it appears that the total number is
sometimes thirteen, which is frequently that of the segments of the
body without the head. The number of chambers differs, however,
greatly, as we have previously stated, and cannot be considered to
support the idea of an original segmental arrangement of the
chambers. The dorsal vessel, though in the adult a single organ,
arises in the embryo from two lateral, widely separated parts which
only in a subsequent stage of the embryonic development coalesce
in the median line.
Fat-Body.
In discussing the tracheae we remarked on the importance of their

function and on their abundant presence in the body. Equally
conspicuous, and perhaps scarcely less important in function, is the
fat-body, which on opening some Insects, especially such as are in
the larval stage, at once attracts attention. It consists of masses of
various size and indefinite form distributed throughout the body,
loosely connected together, and more or less surrounding and
concealing the different organs. The colour varies according to the
species of Insect. This fat-body is much connected with fine tracheal
twigs, so that an organisation extending throughout the body is thus
formed. It may be looked on as a store of nutritious matter which
may be added to or drawn on with great rapidity; and it is no doubt
on this that many of the internal parasites, so common in the earlier
stages of Insects' lives, subsist before attacking the more permanent
tissues of their hosts. There is some reason to suppose that the fat-
body may have some potency in determining the hunger of the
Insect, for some parasitised larvae eat incessantly.
The matter extracted from the food taken into the stomach of the
Insect, after undergoing some elaboration—on which point very little
is known—finds its way into the body-cavity of the creature, and as it
is not confined in any special vessels the fat-body has as unlimited a
supply of the nutritive fluid as the other organs: if nutriment be
present in much greater quantity than is required for the purposes of
immediate activity, metamorphosis or reproduction, it is no doubt
taken up by the fat-body which thus maintains, as it were, an
independent feeble life, subject to the demands of the higher parts of
the organisation. It undoubtedly is very important in metamorphosis,
indeed it is possible that one of the advantages of the larval state
may be found in the fact that it facilitates, by means of the fat-body,
the storage in the organisation of large quantities of material in a
comparatively short period of time.
A considerable quantity of fat tissue is found in the pericardial sinus,

where it is frequently of somewhat peculiar form, and is spoken of as
pericardial cells, or pericardial tissue. Some large cells, frequently of
pale yellow colour, and containing no fat, are called oenocytes by
Wielowiejski. They are connected with the general fat-body, but are
not entirely mingled with it; several kinds have been already
distinguished, and they are probably generally present. The
phagocytes, or leucocytes, the cells that institute the process of
histolysis in the metamorphosis of Muscidae, are a form of blood
cell; though these cells are amoeboid some writers derive them from
the fat-body. The cells in the blood have no doubt generally an
intimate relation with the fat-body, but very little accurate information
has been obtained as to these important physiological points, though
Graber has inaugurated their study.[60]
Organs of Sex.
The continuation of the species is effected in Insects by means of

two sexes, each endowed with special reproductive organs. It has
been stated that there are three sexes in some Insects—male,
female, and neuter; but this is not correct, as the so-called neuters
are truly sexed individuals,—generally females,—though, as a rule,
they are not occupied with the direct physiological processes for
continuing the species.
The offspring is usually produced in the shape of eggs, which are

formed in ovaries. These organs consist of egg-tubes, a cluster of
which is placed on each side of the body, and is suspended,
according to Leydig[61] and others, to the tissue connected with the
heart by means of the thread-like terminations of the tubes.
Fig. 74.—Sex organs of female of Scolia interrupta (after Dufour); a,

egg-tubes; b, oviducts; c, poison glands; d, duct of accessory
gland (or spermatheca); e, external terminal parts of body.
The number of egg-tubes varies greatly in different Insects; there

may be only one to each ovary (Campodea), but usually the number
is greater, and in the queen-bee it is increased to about 180. In the
Queens of the Termitidae, or white ants, the ovaries take on an
extraordinary development; they fill the whole of the greatly
distended hind-body. Three thousand egg-tubes, each containing
many hundred eggs, may be found in a Queen Termite, so that, as
has been said by Hagen,[62] an offspring of millions in number is
probable. There is considerable variety in the arrangements for the
growth of the eggs in the egg-tubes. Speaking concisely, the tubes
may be considered to be centres of attraction for nutritive material, of
which they frequently contain considerable stores. Next to the
terminal thread, of which we have already spoken, there is a greater
or smaller enlargement of the tube, called the terminal chamber; and
there may also be nutriment chambers, in addition to the dilatations
which form the egg-chambers proper. Korschelt[63] distinguishes
three principal forms of egg-tubes, viz. (1) there are no special
nutriment chambers, a condition shown in Figure 74; (2) nutriment
chambers alternate with the egg-chambers, as shown in our Figure
of an egg-tube of Dytiscus marginalis; (3) the terminal chamber
takes on an unusual development, acting as a large nutriment
chamber, there being no other special nutriment chambers. This
condition is found in Rhizotrogus solstitialis. The arrangements as to
successive or simultaneous production of the eggs in the tubes
seem to differ in different Insects. In some forms, such as the white
ants, the process of egg-formation (oogenesis) attains a rapidity that
is almost incredible, and is continued, it is said, for periods of many
months. There is no point in which Insects differ more than in that of
the number of eggs produced by one female. The egg-tubes are
connected with a duct for the conveyance of the eggs to the exterior,
and the arrangements of the tubes with regard to the oviduct also
vary much. An interesting condition is found in Machilis (see Fig. 94,
p. 188), where the seven egg-tubes are not arranged in a bunch, but
open at a distance from one another into the elongated duct. The
two oviducts usually unite into one chamber, called the azygos
portion or the uterus, near their termination. There are a few Insects
(Ephemeridae) in which the two oviducts do not unite, but have a
pair of orifices at the extremity of the body. Hatchett-Jackson has
recently shown[64] that in Vanessa io of the Order Lepidoptera, the
paired larval oviducts are solid, and are fixed ventrally so as to
represent an Ephemeridean stage; that the azygos system of ducts
and appended structures develop separately from the original
oviducts, and that they pass through stages represented in other
Orders of Insects to the stage peculiar to the Lepidoptera. Machilis,
according to Oudemans, is a complete connecting link between the
Insects with single and those with paired orifices.
There are in different Insects more than one kind of diverticula and
accessory glands in connexion with the oviducts or uterus; a
receptaculum seminis, also called spermatheca, is common. In the
Lepidoptera there is added a remarkable structure, the bursa
copulatrix, which is a pouch connected by a tubular isthmus with the
common portion of the oviduct, but having at the same time a
separate external orifice, so that there are two sexual orifices, the
opening of the bursa copulatrix being the lower or more anterior. The
organ called by Dufour in his various contributions glande sébifique,
is now considered to be, in some cases at any rate, a spermatheca.
The special functions of the accessory glands are still very obscure.
Fig. 75.—Egg-tube of Dytiscus marginalis; e.c, egg-chamber; n.c,

nutriment chamber; t.c, terminal chamber; t.t, terminal thread.
(After Korschelt.)
The ovaries of the female are replaced in the male by a pair of

testes, organs exhibiting much variety of form. The structure may
consist of an extremely long and fine convoluted tube, packed into a
small space and covered with a capsule; or there may be several
shorter tubes. As another extreme may be mentioned the existence
of a number of small follicles opening into a common tube, several of
these small bodies forming together a testis. As a rule each testis
has its own capsule, but cases occur—very frequently in the
Lepidoptera—in which the two testes are enclosed in a common
capsule; so that there then appears to be only one testis. The
secretion of each testis is conveyed outwards by means of a slender
tube, the vas deferens, and there are always two such tubes, even
when the two testes are placed in one capsule. The vasa deferentia
differ greatly in their length in different Insects, and are in some
cases many times the length of the body; they open into a common
duct, the ductus ejaculatorius. Usually at some part of the vas
deferens there exists a reservoir in the form of a sac or dilatation,
called the vesicula seminalis. There are in the male, as well as in the
female, frequently diverticula, or glands, in connexion with the sexual
passages; these sometimes exhibit very remarkable forms, as in the
common cockroach, but their functions are quite obscure. There is,
as we have already remarked, extreme variety in the details of the
structure of the internal reproductive apparatus in the male, and
there are a few cases in which the vasa deferentia do not unite
behind, but terminate in a pair of separate orifices. The genus
Machilis is as remarkable in the form of the sexual glands and ducts
of the male as we have already mentioned it to be in the
corresponding parts of the female.
Fig. 76.—Tenthredo cincta. a, a, testes; b, b, vasa deferentia; c, c,

vesiculæ seminales; d, extremity of body with copulatory
armature. (After Dufour.)
Although the internal sexual organs are only fully developed in the
imago or terminal stage of the individual life, yet in reality their
rudiments appear very early, and may be detected from the embryo
state onwards through the other preparatory stages.
The spermatozoa of a considerable number of Insects, especially of

Coleoptera, have been examined by Ballowitz;[65] they exhibit great
variety; usually they are of extremely elongate form, thread-like, with
curious sagittate or simply pointed heads, and are of a fibrillar
structure, breaking up at various parts into finer threads.
External Sexual Organs.—The terminal segments of the body are

usually very highly modified in connexion with the external sexual
organs, and this modification occurs in such a great variety of forms
as to render it impossible to give any general account thereof, or of
the organs themselves. Some of these segments—or parts of the
segments, for it may be dorsal plates or ventral plates, or both—may
be withdrawn into the interior, and changed in shape, or may be
doubled over, so that the true termination of the body may be
concealed. The comparative anatomy of all these parts is especially
complex in the males, and has been as yet but little elucidated, and
as the various terms made use of by descriptive entomologists are of
an unsatisfactory nature we may be excused from enumerating
them. We may, however, mention that when a terminal chamber is
found, with which both the alimentary canal and the sexual organs
are connected, it is called a cloaca, as in other animals.
Parthenogenesis.
There are undoubted cases in Insects of the occurrence of

parthenogenesis, that is, the production of young by a female without
concurrence of a male. This phenomenon is usually limited to a
small number of generations, as in the case of the Aphididae, or
even to a single generation, as occurs in the alternation of
generations of many Cynipidae, a parthenogenetic alternating with a
sexual generation. There are, however, a few species of Insects of
which no male is known (in Tenthredinidae, Cynipidae, Coccidae),
and these must be looked on as perpetually parthenogenetic. It is a
curious fact that the result of parthenogenesis in some species is the
production of only one sex, which in some Insects is female, in
others male; the phenomenon in the former case is called by
Taschenberg[66] Thelyotoky, in the latter case Arrhenotoky;
Deuterotoky being applied to the cases in which two sexes are
produced. In some forms of parthenogenesis the young are
produced alive instead of in the form of eggs. A very rare kind of
parthenogenesis, called paedogenesis, has been found to exist in
two or three species of Diptera, young being produced by the
immature Insect, either larva or pupa.
Glands.
Insects are provided with a variety of glands, some of which we have

alluded to in describing the alimentary canal and the organs of sex;
but in addition to these there are others in connexion with the outer
integument; they may be either single cells, as described by Miall in
Dicranota larva,[67] or groups of cells, isolated in tubes, or pouches.
The minute structure of Insect glands has been to some extent
described by Leydig;[68] they appear to be essentially of a simple
nature, but their special functions are very problematic, it being
difficult to obtain sufficient of their products for satisfactory
examination.
CHAPTER V
DEVELOPMENT
EMBRYOLOGY–EGGS–MICROPYLES–FORMATION OF EMBRYO–VENTRAL
PLATE–ECTODERM AND ENDODERM–SEGMENTATION–LATER STAGES–
DIRECT OBSERVATION OF EMBRYO–METAMORPHOSIS–COMPLETE AND
INCOMPLETE–INSTAR–HYPERMETAMORPHOSIS–METAMORPHOSIS OF
INTERNAL ORGANS–INTEGUMENT–METAMORPHOSIS OF BLOWFLY–
HISTOLYSIS–IMAGINAL DISCS–PHYSIOLOGY OF METAMORPHOSIS–
ECDYSIS.
The processes for the maintenance of the life of the individual are in
Insects of less proportional importance in comparison with those for
the maintenance of the species than they are in Vertebrates. The
generations of Insects are numerous, and the individuals produced
in each generation are still more profuse. The individuals have as a
rule only a short life; several successive generations may indeed
make their appearances and disappear in the course of a single
year.
Although eggs are laid by the great majority of Insects, a few species
nevertheless increase their numbers by the production of living
young, in a shape more or less closely similar to that of the parent.
This is well known to take place in the Aphididae or green-fly Insects,
whose rapid increase in numbers is such a plague to the farmer and
gardener. These and some other cases are, however, exceptional,
and only emphasise the fact that Insects are pre-eminently
oviparous. Leydig, indeed, has found in the same Aphis, and even in
the same ovary, an egg-tube producing eggs while a neighbouring
tube is producing viviparous individuals.[69] In the Diptera pupipara
the young are produced one at a time, and are born in the pupal
stage of their development, the earlier larval state being undergone
in the body of the parent: thus a single large egg is laid, which is
really a pupa.
The eggs are usually of rather large size in comparison with the
parent, and are produced in numbers varying according to the
species from a few—15 or even less in some fossorial Hymenoptera
—to many thousands in the social Insects: somewhere between 50
and 100 may perhaps be taken as an average number for one
female to produce. The whole number is frequently deposited with
rapidity, and the parent then dies at once. Some of the migratory
locusts are known to deposit batches of eggs after considerable
intervals of time and change of locality. The social Insects present
extraordinary anomalies as to the production of the eggs and the
prolongation of the life of the female parent, who is in such cases
called a queen.
The living matter contained in the egg of an Insect is protected by

three external coats: (1) a delicate interior oolemm; (2) a stronger,
usually shell-like, covering called the chorion; (3) a layer of material
added to the exterior of the egg from glands, at or near the time
when it is deposited, and of very various character, sometimes
forming a coat on each egg and sometimes a common covering or
capsule for a number of eggs. The egg-shell proper, or chorion, is
frequently covered in whole or part with a complex minute sculpture,
of a symmetrical character, and in some cases this is very highly
developed, forming an ornamentation of much delicacy; hence some
Insects' eggs are objects of admirable appearance, though the
microscope is of course necessary to reveal their charms. One of the
families of butterflies, the Lycaenidae, is remarkable for the complex
forms displayed by the ornamentation of the chorion (see Fig. 78, B).
Fig. 77.—Upper or micropylar aspect of egg of Vanessa cardui. (After

Scudder.)
The egg-shell at one pole of the egg is perforated by one or more

minute orifices for the admission to the interior of the spermatozoon,
and it is the rule that the shell hereabouts is symmetrically sculptured
(see Fig. 77), even when it is unornamented elsewhere: the
apertures in question are called micropyles. They are sometimes
protected by a micropyle apparatus, consisting of raised processes,
or porches: these are developed to an extraordinary extent in some
eggs, especially in those of Hemiptera-Heteroptera (see Fig. 78, C).
Some of these peculiar structures have been described and figured
by Leuckart.[70] The purpose they serve is quite obscure.
Fig. 78.—Eggs of Insects: A, blowfly (after Henking); B, butterfly,

Thecla (after Scudder); C, Hemipteron (Reduviid).
Formation of Embryo.
The mature, but unfertilised, egg is filled with matter that should
ultimately become the future individual, and in the process of
attaining this end is the seat of a most remarkable series of changes,
which in some Insects are passed through with extreme rapidity. The
egg-contents consist of a comparatively structureless matrix of a
protoplasmic nature and of yolk, both of which are distributed
throughout the egg in an approximately even manner. The yolk,
however, is by no means of a simple nature, but consists, even in a
single egg, of two or three kinds of spherular or granular
constituents; and these vary much in their appearance and
arrangement in the early stages of the development of an egg, the
yolk of the same egg being either of a homogeneously granular
nature, or consisting of granules and larger masses, as well as of
particles of fatty matter; these latter when seen through the
microscope looking sometimes like shining, nearly colourless,
globules.
Fig. 79.—Showing the two extruded polar bodies P1, P2 now nearly
fused and reincluded, and the formation of the spindle by junction
of the male and female pronuclei. (After Henking.)
The nature of the matrix—which term we may apply to both the

protoplasm and yolk as distinguished from the minute formative
portions of the egg—and the changes that take place in it have been
to some extent studied, and Kowalewsky, Dohrn,[71] Woodworth,[72]
and others have given some particulars about them. The early
changes in the formative parts of the mature egg have been
observed by Henking in several Insects, and particularly in
Pyrrhocoris, his observations being of considerable interest. When
the egg is in the ovary and before it is quite mature,—at the time, in
fact, when it is receiving nutriment from ovarian cells,—it contains a
germinal vesicle including a germinal spot, but when the egg is
mature the germinal vesicle has disappeared, and there exists in its
place at one portion of the periphery of the egg-contents a cluster of
minute bodies called chromosomes by Henking, whom we shall
follow in briefly describing their changes. The group divides into two,
each of which is arranged in a rod or spindle-like manner, and may
then be called a directive rod or spindle. The outer of these two
groups travels quite to the periphery of the egg, and there with some
adjacent matter is extruded quite outside the egg-contents (not
outside the egg-coverings), being in its augmented form called a
polar or directive body. While this is going on the second directive
spindle itself divides into two groups, the outer of which is then
extruded in the manner we have already described in the case of the
first polar body, thus completing the extrusion of two directive bodies.
The essential parts of the bodies that are successively formed during
these processes are the aggregates, called chromosomes; the
number of these chromosomes appears to be constant in each
species; their movements and dispositions are of a very interesting
character, the systems they form in the course of their development
having polar and equatorial arrangements. These we cannot further
allude to, but may mention that the extrusion of the directive bodies
is only temporary, they being again included within the periphery of
the egg by the growth and extension of adjacent parts which meet
over and thus enclose the bodies.
The arrangements and movements we have briefly alluded to have

been limited to the unfertilised condition of the egg (we should rather
say, the fertilising element has taken no part in them), and have as
their result the union of the chromosomes existing after the extrusion
of the two polar bodies, into a small body called the female
pronucleus or egg-nucleus (Eikern), while the position of the
movements has been an extremely minute portion of the egg near to
its outer surface or periphery. The introduction of a sperm, or male,
element to the egg through the micropyle gives rise to the formation
of another minute body placed more in the interior of the egg, and
called the sperm-nucleus. The egg-nucleus, travelling more into the
interior of the egg, meets the sperm-nucleus; the two amalgamate,
forming a nucleus or body that goes through a series of changes
resulting in its division into two daughter-bodies. These two again
divide, and by repetitions of such division a large number of nuclei
are formed which become arranged in a continuous manner so as to
form an envelope enclosing a considerable part (if not quite the
whole) of the egg-mass. This envelope is called the blastoderm, and
together with its contents will form the embryo. We must merely
allude to the fact that it has been considered that some of the nuclei
forming the blastoderm arise directly from the egg-mass by a
process of amalgamation, and if this prove to be correct it may be
admitted that some portions of the embryo are not entirely the result
of division or segmentation of combined germ and sperm-nuclei.
Wheeler states[73] that some of the nuclei formed by the first
differentiation go to form the vitellophags scattered throughout the
yolk. We should also remark that, according to Henking, the
blastoderm when completed shows at one part a thickening,
immediately under which (i.e. included in the area the blastoderm
encloses) are the two polar bodies, which, as we have seen, were
formed by the germinating body at an earlier stage of its activity. Fig.
79 represents a stage in the development of Pyrrhocoris, showing
the interior of the egg after a body has been formed by the union of
the sperm and egg-nuclei; this body is about to undergo division or
segmentation, and the equatorial arrangement where this will take
place is seen. The two polar bodies P1, P2, after having been
excluded, are nearly reincluded in the egg.
The Ventral Plate.
The next important change after the formation of the blastoderm is

the partial detachment of a part of its periphery to become placed in
the interior of the other and larger portion. The way in which this
takes place will be gathered from the accompanying diagrammatic
figures taken from Graber: a thickened portion (a b) of the
blastoderm becomes indrawn so as to leave a fold (c d) at each point
of its withdrawal, and these folds afterwards grow and meet so as to
enclose the thickened portion. The outer envelope, formed in part by
the original blastoderm and in part by the new growth, is called the
serosa (e f), the inner layer (g) of the conjoined new folds being
termed the amnion: the part withdrawn to the interior and covered by
the serosa and amnion is called the ventral plate, or germinal band
(Keimstreif), and becomes developed into the future animal. The
details of the withdrawal of the ventral plate to the interior are very
different in the various Insects that have been investigated.
Fig. 80.—Stages of the enclosure of the ventral plate: A, a, b, ventral plate;
B, c, d, folds of the blastoderm that form the commencement of the
amnion and serosa; C, e, f, part of the serosa; g, amnion.
One of the earliest stages in the development is a differentiation of a

portion of the ventral plate into layers from which the future parts of the
organisation will be derived. This separation of endoderm from ectoderm
takes place by a sort of invagination, analogous with that by which the
ventral plate itself is formed. A longitudinal depression running along the
middle of the ventral plate appears, and forms a groove or channel, which
becomes obliterated as to its outer face by the meeting together of the
two margins of the groove (except on the anterior part, which remains
open). The more internal layer of the periphery of this closed canal is the
origin of the endoderm and its derivatives. Subsequently the ventral plate
and its derivatives grow so as to form the ventral part and the internal
organs of the Insect, the dorsal part being completed much later by
growths that differ much in different Insects; Graber, who has specially
investigated this matter, informing us[74] that an astonishing
multifariousness is displayed. It would appear that the various modes of
this development do not coincide with the divisions into Orders and
Families adopted by any systematists.
We should observe that the terms ectoderm, mesoderm, and endoderm

will probably be no longer applied to the layers of the embryo when
embryologists shall have decided as to the nature of the derived layers,
and shall have agreed as to names for them. According to the
nomenclature of Graber[75] the blastoderm differentiates into Ectoblast
and Endoblast; this latter undergoing a further differentiation into
Coeloblast and Myoblast. This talented embryologist gives the following
table of the relations of the embryonic layers and their nomenclature, the
first term of each group being the one he proposed to use:—
Nussbaum considers[76] that "there are four layers in the cockroach-
embryo, viz. (1) epiblast, from which the integument and nervous system
are developed; (2) somatic layer of mesoblast, mainly converted into the
muscles of the body-wall; (3) splanchnic layer of mesoblast, yielding the
muscular coat of the alimentary canal; and (4) hypoblast, yielding the
epithelium of the mesenteron."
Fig. 81.—Early stages of the segmentation of a beetle (Lina): A,

segmentation not visible, 1 day; B, segmentation of head visible; C,
segmentation still more advanced, 2¼ days; PC, procephalic lobes; g1,
g2, g3, segments bearing appendages of the head; th, thorax; th1, th2,
th3, segments of the thorax; a1, a2, anterior abdominal.
Turning our attention to the origin of the segmentation, that is so marked

a feature of Insect structure, we find that evidence of division or
arrangement of the body into segments appears very early, as shown in
our Figure of some of the early stages of development of Lina (a beetle),
Fig. 81. In A the segmentation of the ectoderm has not commenced, but
the procephalic lobes (P C) are seen; in B the three head segments are
distinct, while in C the thoracic segmentation has occurred, and that of
the abdomen has commenced. Graber considers that in this species the
abdomen consists of ten segmental lobes, and a terminal piece or telson.
According to Graber[77] this is not a primitive condition, but is preceded by
a division into three or four parts, corresponding with the divisions that will
afterwards be head, thorax, and abdomen. This primary segmentation, he
says, takes place in the Hypoblast (Endoderm) layer of the ventral plate;
this layer being, in an early stage of the development of a common
grasshopper (Stenobothrus variabilis), divided into four sections, two of
which go to form the head, while the others become thorax and abdomen
respectively. In Lina the primary segmentation is, Graber says, into three
instead of four parts. Graber's opinion on the primary segmentation does
not appear to be generally accepted, and Wheeler, who has studied[78]
the embryology of another Orthopteron, considers it will prove to be
incorrect. When the secondary segmentation occurs the anterior of the
two cephalic divisions remains intact, while the second divides into the
three parts that afterwards bear the mouth parts as appendages. The
thoracic mass subsequently segments into three parts, and still later the
hind part of the ventral plate undergoes a similar differentiation so as to
form the abdominal segments; what the exact number of these may be is,
however, by no means easy to decide, the division being but vague,
especially posteriorly, and not occurring all at once, but progressing from
before backwards.
The investigations that have been made in reference to the segmentation

of the ventral plate do not at present justify us in asserting that all Insects
are formed from the same number of embryonic segments. The matter is
summarised by Lowne, to the effect that posterior to the procephalic lobes
there are three head segments and three thoracic segments, and a
number of abdominal segments, "rarely less than nine or more than
eleven." It will be seen by referring to Figure 81 that the segmentation
appears, not simultaneously, but progressively from the head backwards;
this of course greatly increases the difficulty of determining by means of a
section the real number of segments.
Fig. 82.—Embryo of a moth (Zygaena) at the fifth day (after Graber): am,
amnion; s, serosa; p, procephalic lobes; st, stomodaeum; pr,
proctodaeum; g1, g2, g3, the mouth parts or head appendages; th1, th2,
th3, appendages of the thoracic segments; a1-a10, abdominal
segments; s.g, salivary gland.
The later stages in the development of Insects are already proved to be

so various that it would be impossible to attempt to follow them in detail;
but in Fig. 82 we represent a median section of the embryo of Zygaena
filipendula at the fifth day. It shows well some of the more important of the
general features of the development at a stage subsequent to those
represented in Fig. 81, A, B, C. The very distinct stomodaeum (st) and
proctodaeum (pr) are seen as inflexions of the external wall of the body;
the segmentation and the development of the ventral parts of the embryo
are well advanced, while the dorsal part of the embryo is still quite
incomplete.
The method of investigation by which embryologists chiefly carry on their

researches is that of dividing the egg after proper preparation, into a large
number of thin sections, which are afterwards examined in detail, so as to
allow the arrangement to be completely inferred and described. Valuable
as this method is, it is nevertheless clear that it should, if possible, be
supplemented by direct observation of the processes as they take place
in the living egg: this method was formerly used, and by its aid we may
still hope to obtain exact knowledge as to the arrangements and
rearrangements of particles by which the structures develop. Such
questions as whether the whole formative power in the egg is absolutely
confined to one or two small centres to which the whole of the other egg
contents are merely, as it were, passive accessories, or whether an egg is
a combination in which some portion of the powers of rearrangement is
possessed by other particles, as well as the chromosomes, in virtue of
their own nature or of their position at an early period in the whole, can
scarcely be settled without the aid of direct observation of the processes
during life.
The importance of the yolk is recognised by most of the recent writers.

Nussbaum states (loc. cit.) that "scattered yolk-cells associate themselves
with the mesoblast cells, so that the constituents of the mesoblast have a
twofold origin." Wheeler finds[79] that amoeboid cells—he styles them

Advanced Software Technologies For Post Peta Scale Computing The Japanese Post Peta CREST Research Project Mitsuhisa Sato

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Software Technologies For Post Peta Scale Computing The Japanese Post Peta CREST Research Project Mitsuhisa Sato

Uploaded by

Copyright:

Available Formats

Advanced Software Technologies for

Post Peta Scale Computing The

Contagious Metaphor 1st Edition Peta Mitchell

XcalableMP PGAS Programming Language From Programming

Basic Japanese Second Edition Eriko Sato

Post-Truth, Post-Press, Post-Europe: Euroscepticism And

Creating and Using Virtual Prototyping Software

Advanced Computing and Communication Technologies

Post-Conflict Education for Democracy and Reform:

The good research guide for small scale research

ISBN 978-981-13-1923-5 ISBN 978-981-13-1924-2 (eBook)

Library of Congress Control Number: 2018959846

© Springer Nature Singapore Pte Ltd. 2019

Computational Science, which enables us to explore uncharted fields of science

Kobe, Japan Mitsuhisa Sato

1 JST CREST Post-petascale Software Project Bridging

8 A Development Platform for Embedded Domain-Specific

Abstract JST CREST post-petascale software project aimed to establish software

1.1 Trends of High-Performance Computing

High-performance computing systems used for cutting edge of advanced compu-

© Springer Nature Singapore Pte Ltd. 2019 1

1.2 Outline of JST CREST Post-petascale Software Project

CREST (Core Research for Evolutional Science and Technology) is a funding

Table 1.1 Research teams of JST CREST post-petascale software project

1.3 Research Topics of the Project

style, we are developing new language framework XcalableACC (XACC) where

1.4 Results and Achievements

Exploration for the future direction of future high-performance computing

Kengo Nakajima, Masaharu Matsumoto, Masatoshi Kawai,

Abstract ppOpen-HPC and pK-Open-HPC are open source infrastructures for

2.1 Overview of ppOpen-HPC

K. Nakajima () · M. Matsumoto · M. Kawai · A. Ida

© Springer Nature Singapore Pte Ltd. 2019 11

pK-Open-SOL ppOpen-MATH MG GRAPH VIS MP Math Library

pK-Open-HPC ppOpen-SYS COMM FT

Optimized Application with

Fig. 2.1 Overview of ppOpen-HPC and pK-Open-HPC [1, 2]

FEM FDM FVM

Fig. 2.2 Target applications of ppOpen-HPC and pK-Open-HPC

• ppOpen-MATH Math library

2.2 Development of pK-Open-FVM

Fig. 2.3 Example of

2.2.1 Application to Vlasov-Poisson Simulation

On large-scale computing, high-efficient calculation by AMR technique is one of the

Fig. 2.4 Application of AMR to 1D-1D Vlasov-Poisson simulation

2.2.2 AMR Particle Simulation for the Development

Fig. 2.5 RPD equipment

Fig. 2.6 Distribution of ion

Fig. 2.7 The number of

2.2.3 Application to Particle-Based Sugarscape Model

Fig. 2.8 Initial distribution

two-dimensional simulation domain. In the simulation, initially, two peaks of sugar

Here, Fs is the spatial gradient of the sugar amount obtained at inhabitant

2.3 Robust and Massively Parallelized Preconditioner for

Fig. 2.9 Distribution of

Fig. 2.10 Example of

(founded by CREST) and ESSEX-II (founded by SPPEXA) focuses on solving the

2.3.2 Regularizations for Robustness

For robustness of the ILU preconditioned Krylov subspace method, we applied

2.3.2.1 Blocking Technique

The ILU preconditioner using a blocking technique is a well-known approach to

2.3.2.2 Diagonal Shifting

This is a strong and direct method to make diagonal dominant matrices. If we