Parallel Computing in Optimization

Applied Optimization
Volume 7

Series Editors:
Panos M. Pardalos
University of Florida, U.S.A.

Donald Hearn
University of Florida, U.SA.

The titles published in this series are listed at the end of this volume.
Parallel Computing
in Optimization
Edited by

Athanasios Migdalas
LinJcijping Institute of Technology

Panos M. Pardalos
University ofFlorida
and

Sverre Stor0y
University ofBergen

KLUWER ACADEMIC PUBLISHERS

DORDRECHT I BOSTON I LONDON
A C.l.P. Catalogue record for this book is available from the Library of Congress.

ISBN-13: 978-1-4613-3402-6 e-ISBN-13:978-1-4613-3400-2

001: 10.1007/978-1-4613-3400-2

Published by Kluwer Academic Publishers,

P.O. Box 17,3300 AA Dordrecht, The Netherlands.

Kluwer Academic Publishers incorporates

the publishing programmes of
D. Reidel, Martinus Nijhoff, Dr W. Junk and MTP Press.

Sold and distributed in the U.S.A. and Canada

by Kluwer Academic Publishers,
101 Philip Drive, Norwell, MA 02061, U.S.A.

In all other countries, sold and distributed

by Kluwer Academic Publishers,
P.O. Box 322, 3300 AH Dordrecht, The Netherlands.

Printed on acid-free paper

All Rights Reserved

© 1997 Kluwer Academic Publishers
Softcover reprint of the hardcover 1st edition 1997
No part of the material protected by this copyright notice may be reproduced or
utilized in any form or by any means, electronic or mechanical,
including photocopying, recording or by any information storage and
retrieval system, without written permission from the copyright owner.
"there is no rest for the messenger
till the message is delivered"

The Rescue: A Romance in the Shallows

(by Joseph Conrad)

 CONTENTS

PREFACE xvii

1 MODELS FOR PARALLEL ALGORITHM

DESIGN: AN INTRODUCTION
Afonso Ferreira and Michel Morvan 1
1 Introduction 1
2 Shared memory model: PRAM 4
3 Distributed memory models: DMM 13
4 The coarse grained multicomputer model: CGM 21
5 Summary 24
6 EXERCISES 24

2 PARALLEL ALGORITHMS AND COMPLEXITY

Martin Furer 27
1 Introduction 28
2 Models of Parallel Computers 30
3 Limits of Parallelism 34
4 Classification of some Important Graph Problems 38
5 Basic Techniques 40
6 Parallel Algorithms Toolbox 44
7 Approximating the Minimum Degree Spanning Tree Problem 49
8 EXERCISES 52

3 A PROGRAMMER'S VIEW OF PARALLEL

COMPUTERS
TorS¢revik 57
1 Introduction 57

Vll
Vlll PARALLEL COMPUTING IN OPTIMIZATION

2 The Memory Hierarchy 59

3 Communication Network 65
4 Future trends 68
5 EXERCISES 69

4 SCALABLE PARALLEL ALGORITHMS FOR

SPARSE LINEAR SYSTEMS
Anshul Gupta, George Karypis, and Vipin Kumar 73
1 Introduction 73
2 Parallel Direct Cholesky Factorization 77
3 Multilevel Graph Partitioning 84
4 EXERCISES 91

5 OBJECT ORIENTED MATHEMATICAL

MODELLING AND COMPILATION TO
PARALLEL CODE
Niclas Andersson and Peter Fritzson 99
1 Introduction 100
2 Object Math 102
3 Background to Parallel Code Generation 121
4 Definitions 136
5 Towards a Parallelising Compiler 141
6 Equation System Level 145
7 Equation Level 151
8 Clustered Task Level 159
9 Explicit Parallelism 170
10 Summary 177
11 EXERCISES 178

6 PARALLEL ALGORITHMS FOR NETWORK

PROBLEMS
Olof Damberg, Athanasios Migdalas and Sverre Stor¢y 183
1 Introduction 183
2 Parallel processing paradigms 184
3 The shortest path problem 186
4 Linear problems over bipartite graphs 190
Contents IX

5 Convex problems over singlecommodity networks 201

6 Convex problems over multicommodity networks 206
7 EXERCISES 227

7 PARALLEL BRANCH AND BOUND -

PRINCIPLES AND PERSONAL EXPERIENCES
Jens Clausen 239
1 Introduction 240
2 Sequential B&B 241
3 Parallel B&B 249
4 Personal Experiences with GPP and QAP 256
5 Ideas and Pitfalls for Parallel B&B users 262
6 EXERCISES 265

8 PARALLELIZED HEURISTICS FOR

COMBINATORIAL SEARCH
Kristina Holmqvist, Athanasios Migdalas, Panos M.
Pardalos 269
1 Heuristics for Combinatorial Search 269
2 Local Search 271
3 Simulated Annealing 273
4 Tabu Search 278
5 Genetic Algorithms 282
6 Greedy Randomized Adaptive Search Procedures 287
7 Conclusions 289
8 EXERCISES 290

9 PARALLEL COST APPROXIMATION

ALGORITHMS FOR DIFFERENTIABLE
OPTIMIZATION
Michael Patriksson 295
1 Introduction 295
2 Sequential Cost Approximation Algorithms 308
3 Synchronized Parallel Cost Approximation Algorithms 320
4 Partially Asynchronous Parallel Cost Approximation Algorithms328
5 Concluding Remarks 334
6 EXERCISES 335
x PARALLEL COMPUTING IN OPTIMIZATION

10 PARALLEL COMPUTATION OF
VARIATIONAL INEQUALITIES AND
PROJECTED DYNAMICAL SYSTEMS WITH
APPLICATIONS
Anna Nagurney 343
1 Introduction 343
2 The Variational Inequality Problem 345
3 Projected Dynamical Systems 359
4 Variational Inequality Applications 370
5 Projected Dynamical Systems Applications 390
6 Summary and Conclusions 405
7 EXERCISES 406

11 PARALLEL ALGORITHMS FOR LARGE-SCALE

STOCHASTIC PROGRAMMING
Hercules Vladimirou and Stavros A. Zenios 413
1 Introduction 413
2 Stochastic Programs with Recourse 416
3 Algorithmic Approaches 425
4 Algorithmic Comparisons 445
5 Conclusions 460
6 EXERCISES 461

12 PARALLEL CONTINUOUS NON-CONVEX

OPTIMIZATION
Kristina Holmqvist, Athanasios Migdalas and Panos M.
Pardalos 471
1 Introduction 472
2 Local Search Heuristics 473
3 Deterministic and Stochastic Refinements of Local Search 487
4 Summary of General Principles for Local Search Parallelization 503
5 Exact Methods: Deterministic Approaches 504
6 EXERCISES 516
Contents Xl

13 DETERMINISTIC AND STOCHASTIC

LOGARITHMIC BARRIER FUNCTION
METHODS FOR NEURAL NETWORK
TRAINING
Theodore B. Trafalis and Tarek A. Tutunji 529
1 Introduction 530
2 Newton-type and Logarithmic Barrier Methods 532
3 Application to Neural Network Training 539
4 Ill-Conditioning 549
5 Computational Results 554
6 Conclusions and Future Research 565
7 EXERCISES 567

INDEX 575
 CONTRIBUTORS

Niclas Andersson Peter Fritzson

PELAB PELAB
Computer Science Department Computer Science Department
Linkoping Institute of Technology Linkoping Institute of Technology
S-581 Linkoping S-581 Linkoping
SWEDEN SWEDEN
nicanGida.liu.se petfrGida.liu.se

Jens Clausen Martin Fiirer

DIKU Department of Computer Science and
Department of Computer Science Engineering
University of Copenhagen 220 Pond Laboratory
Universitetsparken 1 Pennsylvania State University
DK-2100 Copenhagen 0 University Park
DENMARK Pennsylvania 16802-6106
clausenGdiku.dk USA
furerGcse.psu.edu
Olof Damberg
Division of Optimization Anshul Gupta
Department of Mathematics Department of Computer Science
Linkoping Institute of Technology University of Minnesota
S-581 83 Linkoping Minneapolis, MN 55455
SWEDEN USA
oldamGmath.liu.se aguptaGcs.umn.edu
Afonso Ferreira Kristina Holmqvist
CNRS Division of Optimization
LIP-ENS Lyon Department of Mathematics
69364 Lyon Cedex 07 Linkoping Institute of Technology
FRANCE S-581 83 Linkoping
ferreiraGlip.ens-lyon.fr SWEDEN
krholGmath.liu.se
xiv CONTRIBUTORS

George Karypis Michael Patriksson

Department of Computer Science
University of Minnesota
Minneapolis, MN 55455
USA
karypisGcs.umn.edu
Vipin Kumar
Department of Computer Science
University of Minnesota
Minneapolis, MN 55455
USA
kumarGcs.umn.edu
Athanasios Migdalas
Division of Optimization
Department of Mathematics
Linkoping Institute of Technology
8-581 83 Linkoping
SWEDEN
samigGmath.liu.se
Michel Morvan
LITP/IBP
Universite Paris 7
75251 Paris Cedex 07
FRANCE
morvanGlitp.ibp.fr
Anna Nagurney
School of Management
Department of General Business and
Finance University of Massachusetts
Amhers, Massachusetts
USA
annaGumass. edu
Panos M. Pardalos
Department of Industrial and
Systems Engineering
University of Florida
Gainesville, FL32611
USA
pardalosGmath.ufl.edu
Division of Optimization
Department of Mathematics
Linkoping Institute of Technology
S-581 83 Linkoping
SWEDEN
mipatGmath.liu.se
Sverre StorfllY
Department of Informatics
University of Bergen
Thorml/lhlensgate 55
N-5020 Bergen
NORWAY
Sverre.StoroyGii.uib.no
Tor Sfllrevik
Parallab
Department of Informatics
University of Bergen,
Thorml/lhlensgate 55
N-5020 Bergen
NORWAY
Tor.SorevikGii.uib.no
Theodore Trafalis
School of Industrial Engineering
202 West Boyd, Room 124
Oklahoma 73019-0631
USA
trafalisGmailhost.ecn.uoknor.edu
Tarek A. Tutunji
Halliburton Energy Services
1100 Everman Road
Forth Worth, Texas 76140
USA
Contributors xv

Hercules Vladimirou
Department of Public and
Business Administration
University of Cyprus
75 Kallipoleos Street
P.O. Box 537
1678 Nicosia
CYPRUS
herculesOatlas.pba.ucy.ac.cy

Stavros Zenios
Department of Public and
Business Administration
University of Cyprus
75 Kallipoleos Street
P.O. Box 537
1678 Nicosia
CYPRUS
zeniossOjupiter.cca.ucy.cy
 PREFACE

During the last three decades, breakthroughs in computer technology have

made a tremendous impact on optimization. In particular, parallel computing
has made it possible to solve larger and computationally more difficult prob-
lems.

This volume contains mainly lecture notes from a Nordic Summer School held
at the Linkoping Institute of Technology, Sweden in August 1995. In order
to make the book more complete, a few authors were invited to contribute
chapters that were not part of the course on this first occasion.

The purpose of this Nordic course in advanced studies was three-fold. One
goal was to introduce the students to the new achievements in a new and very
active field, bring them close to world leading researchers, and strengthen their
competence in an area with internationally explosive rate of growth.

A second goal was to strengthen the bonds between students from different
Nordic countries, and to encourage collaboration and joint research ventures
over the borders. In this respect, the course built further on the achievements of
the "Nordic Network in Mathematical Programming" , which has been running
during the last three years with the support ofthe Nordic Council for Advanced
Studies (NorFA).

The final goal was to produce literature on the particular subject, which would
be available to both the participating students and to the students of the "next
generation" .

The school was funded by NorFA, while activities related to the preparation
of the application to NorFA, and the co-ordination of the lecturers and the
editors were supported in part by the Linkoping Center for Transfer of Infor-
mation Technology (CENIIT). The National Supercomputer Center (NSC) in
Linkoping freely supplied its computer resources and laboratories to the par-
ticipating lecturers and students. More than 30 students and seniors from the
Nordic countries participated in the course.

xvii
xviii PARALLEL COMPUTING IN OPTIMIZATION

An international group of invited speakers presented lectures dealing with novel

programming and algorithmic aspects of parallel computing as well as techni-
cal advances in parallel optimization. In addition, the Summer School pro-
vided a concentrated forum to discuss progress on enhancing the number of
optimization applications that take advantage of emerging parallel computing
architectures.

The volume contains 13 chapters. These can be divided into three groups.
The first two chapters, contributed by A. Ferreira & M. Morvan and M. Fiirer
respectively, discuss theoretical models for parallel algorithm design and their
complexity. T. SI/Jrevik, on the other hand, in chapter 3 gives the perspective
of the programmer practicing parallel algorithm development on real world
platforms.

Solving systems of linear equations efficiently is of great importance on their

own because they arise in many scientific and engineering applications but also
because algorithms for solving many optimization problems need to call system
solvers as subroutines. The fourth chapter by A. Gupta, G. Karypis and V.
Kumar is devoted to scalable parallel algorithms for sparse linear systems, while
the fifth chapter by N. Andersson and P. Fritzson discusses computer based
mathematical modeling aspects of applications such as bearing and automatic
generation of parallel code for the solution of the resulting system of equations.

Chapters six to thirteen are dedicated to optimization problems and methods.

More specifically, in Chapter six O. Damberg, A. Migdalas and S. Storl/Jy give an
introductory survey of parallel algorithms for network problems with emphasis
on linear bipartite assignment problems and a thorough discussion on the im-
plementation of a parallel solver for the traffic assignment problem. In Chapter
seven, J. Clausen reviews the principles of sequential Branch-and-Bound (B&
B) and sketches the main trends in parallel B& B and the problems experienced
with the latter approach. In chapter eight, K. Holmqvist, A. Migdalas and P.
M. Pardalos provide an introductory survey in the parallelization of modern
heuristic methods for combinatorial optimization problems.

In Chapter nine, M. Patriksson presents a unified analysis of decomposition

algorithms for differentiable optimization problems. Chapter ten, written by A.
Nagurney, presents parallel algorithms for the computation of solutions to finite
dimensional variational inequality problems and projected dynamical systems.
H. Vladimirou and S. A. Zenios present in the eleventh chapter the state-of-
the-art in parallel algorithms for stochastic programming. The twelfth chapter,
authored by K. Holmqvist, A. Migdalas and P. M. Pardalos, surveys exact and
heuristic algorithms of deterministic and stochastic type for global optimization
Preface XIX

problems. Finally in Chapter thirteen, T. Trafalis and T. Tutunji present

deterministic and stochastic logarithmic barrier function algorithms for neural
network training.

Although the entire spectrum of Parallel Computing in Optimization cannot

be covered in a volume, it is hoped that these chapters communicate the rich-
ness and diversity in this important research domain with an ever increasing
practical application area.

We would like to take this opportunity to thank the authors, the anonymous
referees, CENIIT for funding the coordination process, and NorFA for funding
the Summer School and helping us produce this book with state-of-the-art
chapters.

A. Migdalas, P. M. Pardalos and S. Stor~y

August, 1996