Introduction Turbulence Simulations

Direct Numerical Simulation of Turbulent Flows

Michael Wilczek

Institute for Theoretical Physics, University of M

unster

22.07.09

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Is turbulence a solved problem?

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Is turbulence a solved problem?

Engineering: conceptually yes

(depends on the complexity of your problem)
relevant scenarios are computationally
accessible

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Is turbulence a solved problem?

Engineering: conceptually yes

(depends on the complexity of your problem)
relevant scenarios are computationally
accessible

Physics: no
statistical description from first principles is
missing
non-equilibrium thermodynamics?

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Is turbulence a solved problem?

Engineering: conceptually yes

(depends on the complexity of your problem)
relevant scenarios are computationally
accessible

Physics: no
statistical description from first principles is
missing
non-equilibrium thermodynamics?

Mathematics: no
boundedness of solutions with smooth inital
conditions?

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Introduction

Problem: turbulence . . .
is described by nonlinear equations
exhibits spatio-temporal chaos
involves large space- and time-scales

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Introduction

Problem: turbulence . . .
is described by nonlinear equations
exhibits spatio-temporal chaos
involves large space- and time-scales

Possible solutions:
understanding of structures
formulating a statistical theory

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Introduction

Problem: turbulence . . .
is described by nonlinear equations
exhibits spatio-temporal chaos
involves large space- and time-scales

Possible solutions:
understanding of structures
formulating a statistical theory

Tools:
any kind of mathematics, that will do
computer simulations

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

Introduction Turbulence Simulations

DNS

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: Visualization

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: equations

Navier-Stokes equations:
u
(x, t) + u(x, t) u(x, t) = p(x, t) + u(x, t) + f(x, t)
t
u(x, t) = 0

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: equations

Navier-Stokes equations:
u
(x, t) + u(x, t) u(x, t) = p(x, t) + u(x, t) + f(x, t)
t
u(x, t) = 0

Vorticity: (x, t) = u(x, t)

Vorticity equation:


(x, t) = u(x, t) (x, t) + (x, t) + f (x, t)
t

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: numerics I

aim: forced (stationary) homogeneous, isotropic turbulence

temporal discretization: RK3 TVD
spatial discretization: box-length 2, dim grid points, periodic
boundary conditions
pseudospectral code

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: numerics II

(k, t) = ik F{u(x, t) (x, t)} + f(k, t)
(k, t) + k2
t

adaptive time-stepping (Courant-Friedrichs-Levy criterion)

pseudospectral: forward/backward FFT is computationally
cheaper than convolution (N log N vs. N 2 )
aliasing: smooth Fourier filter
viscosity is treated exactly (integrating factor)
forcing: freezing of low modes
code is MPI parallel

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: computational costs I

integral scale

k 5/3
E(k)

forcing
dissipative scale

inertial range

extends with Re
dissipation

k
Michael Wilczek Direct Numerical Simulation of Turbulent Flows
 Introduction Turbulence Simulations

DNS: computational costs II

forcing scale and dissipative scale should be well seperated

inertial range extends with increasing Re
size of smallest structures decreases with Re
smallest structures should be well-resolved by the grid
turbulent field should be accurately advanced in time

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: computational costs III

to be more precisely . . .

 3/4
uL
= L = Re3/4 L

x
2 3
   3  4/3
2 9/4 L
Nx Re Re Nx4/9
x L 2
T T T
Nt Re3/4
t x/u l/u
   3
T 2
$$$ Nx Nt Re3
l/u l

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: computational costs IV

1283 2563 5123 10243

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: computational costs V

2563 5123 10243

and beyond
$$$

Re
serial
OpenMP
MPI
Michael Wilczek Direct Numerical Simulation of Turbulent Flows
 Introduction Turbulence Simulations

DNS: computational costs V

2563 5123 10243

and beyond
$$$

Re
serial
OpenMP
MPI
Michael Wilczek Direct Numerical Simulation of Turbulent Flows
 Introduction Turbulence Simulations

DNS: computational costs V

2563 5123 10243

and beyond
$$$

Re
serial
OpenMP
MPI
Michael Wilczek Direct Numerical Simulation of Turbulent Flows
 Introduction Turbulence Simulations

DNS: computational costs V

2563 5123 10243

and beyond
$$$

Re
serial
OpenMP
MPI
Michael Wilczek Direct Numerical Simulation of Turbulent Flows
 Introduction Turbulence Simulations

DNS: OpenMP vs. MPI

OpenMP MPI
fork and join principle decomposition of
easy computational domain
incremental communicate when
parallelization necessary principle
medium number of cores broad range of
limited degree of application
scalability high scalability

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Amdahls Law

consider serial code with runtime 1

P denotes the parallelizable fraction of code
P
runtime on n cores: (1 P ) + N
1
maximum expectable speed up S(N ) = P
(1P )+ N

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Amdahls Law

consider serial code with runtime 1

P denotes the parallelizable fraction of code
P
runtime on n cores: (1 P ) + N
1
maximum expectable speed up S(N ) = P
(1P )+ N

100 P=0.50
P=0.75
P=0.90
P=0.95
P=0.99

20
speed up

10

2 8 32 128 512 2048 8192 32768

number of cores

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: parallelization via MPI

core 3
core 2
core1 DD
core 0

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

DNS: scaling

3
scaling performance of VOrTEx on HLRB II, 1024 grid points, single precision

119.17
32
time per step (sec)

60.15
34.55
16 18.5
9.19
5.28
8
speedup

32 64 128 256 512 1024

4 two partitions in use

three partitions in use

1 reference point
high density blades
ideal scaling

32 64 128 256 512 1024

number of cores

Michael Wilczek Direct Numerical Simulation of Turbulent Flows

 Introduction Turbulence Simulations

Demos

Hello OpenMP/ MPI world

HLRBII @ LRZ
remote visualization

Michael Wilczek Direct Numerical Simulation of Turbulent Flows