OpenFOAM open source CFD on ANSELM

Tom Brzobohat

Supercomputing for industry SC4Industry

Research Programme 3 - Libraries for Parallel Computing, VB-TU Ostrava

open source CFD on ANSELM

OpenFOAM

Outline

- Introduction - Features - OpenFOAM Case

- Running applications - Running applications in parallel - Compiling applications and libraries

OpenFOAM on ANSELM

Benchmarks

What OpenFOAM is?

The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox based on C++.

open source CFD on ANSELM OpenFOAM - Introduction

OpenFOAM is free and open source software, released under the GNU General Public License. OpenFOAM includes over 80 solver applications that simulate specific problems in engineering mechanics and over 170 utility applications that perform pre- and post-processing tasks, e.g. meshing, data visualization, etc. Friendly syntax for partial differential equations Unstructured polyhedral grid capabilities Automatic parallelization of applications written using OpenFOAM high-level syntax Commercial support and training provided by the developers

open source CFD on ANSELM OpenFOAM - Introduction

OpenFOAM syntax for tensor operations and partial differential equations

+ - = p

For example the equation is represented by the code:

Solve ( fvm::ddt(rho,U) + fvm::div(phi,U) - fvm::laplacian(mu,U) == - fvc::grad(p) );

History

open source CFD on ANSELM OpenFOAM - Introduction

The original development of OpenFOAM started in the late 1980s at Imperial College, London. The FOAM, was sold by UK company Nabla Ltd. before being released as open source in 2004. On 15 August 2011, OpenCFD announced its acquisition by Silicon Graphics International (SGI). On September 12, 2012, the ESI Group announced the acquisition of OpenFOAM Ltd from SGI.

Where we can find info?

Homepage Wiki CFD Forum - http://www.openfoam.com/ - http://openfoamwiki.net/ - http://www.cfd-online.com/Forums/openfoam/

open source CFD on ANSELM

OpenFOAM Features

Incompressible flows
adjointShapeOptimizationFoam icoFoam MRFSimpleFoam nonNewtonianIcoFoam pimpleDyMFoam pimpleFoam pisoFoam porousSimpleFoam

open source CFD on ANSELM OpenFOAM - Features

simpleFoam SRFSimpleFoam

Incompressible flows
adjointShapeOptimizationFoam icoFoam MRFSimpleFoam nonNewtonianIcoFoam pimpleDyMFoam pimpleFoam pisoFoam porousSimpleFoam

open source CFD on ANSELM OpenFOAM - Features

simpleFoam SRFSimpleFoam

Multiphase flows
bubbleFoam cavitatingFoam compressibleInterFoam interFoam interDyMFoam interMixingFoam interPhaseChangeFoam LTSInterFoam MRFInterFoam MRFMultiphaseInterFoam multiphaseInterFoam porousInterFoam settlingFoam twoLiquidMixingFoam twoPhaseEulerFoam simpleFoam SRFSimpleFoam

Compressible flows
rhoCentralFoam rhoCentralDyMFoam rhoPimpleFoam rhoPorousMRFLTSPimpleFoam rhoPorousSimpleFoam rhoPorousMRFPimpleFoam rhoSimplecFoam rhoSimpleFoam

open source CFD on ANSELM OpenFOAM - Features sonicDyMFoam sonicLiquidFoam

Compressible flows
rhoCentralFoam rhoCentralDyMFoam rhoPimpleFoam rhoPorousMRFLTSPimpleFoam chemFoam coldEngineFoam dieselEngineFoam dieselFoam rhoPorousMRFSimpleFoam rhoPorousMRFPimpleFoam rhoSimplecFoam rhoSimpleFoam engineFoam fireFoam PDRFoam reactingFoam

open source CFD on ANSELM OpenFOAM - Features sonicDyMFoam sonicLiquidFoam

Combustion
rhoReactingFoam XiFoam

Compressible flows
rhoCentralFoam rhoCentralDyMFoam rhoPimpleFoam rhoPorousMRFLTSPimpleFoam chemFoam coldEngineFoam dieselEngineFoam dieselFoam rhoPorousMRFSimpleFoam rhoPorousMRFPimpleFoam rhoSimplecFoam rhoSimpleFoam engineFoam fireFoam PDRFoam reactingFoam

open source CFD on ANSELM OpenFOAM - Features sonicDyMFoam sonicLiquidFoam

Combustion
rhoReactingFoam XiFoam

Conjugate Heat Transfer

buoyantBaffleSimpleFoam buoyantBoussinesqPimpleFoam buoyantBoussinesqSimpleFoam buoyantPimpleFoam buoyantSimpleFoam chtMultiRegionFoam

RAS Turbulence Models

laminar kEpsilon kOmega kOmegaSST RNGkEpsilon NonlinearKEShih LienCubicKE qZeta LaunderSharmaKE LamBremhorstKE

open source CFD on ANSELM OpenFOAM - Features

LienCubicKELowRe LienLeschzinerLowRe LaunderGibsonRSTM realizableKE SpalartAllmaras

RAS Turbulence Models

laminar kEpsilon kOmega kOmegaSST RNGkEpsilon NonlinearKEShih LienCubicKE qZeta LaunderSharmaKE LamBremhorstKE

open source CFD on ANSELM OpenFOAM - Features

LienCubicKELowRe LienLeschzinerLowRe LaunderGibsonRSTM realizableKE SpalartAllmaras

LES Turbulence Models

laminar Smagorinsky Smagorinsky2 dynSmagorinsky dynLagrangian scaleSimilarity mixedSmagorinsky dynMixedSmagorinsky kOmegaSSTSAS oneEqEddy dynOneEqEddy locDynOneEqEddy

spectEddyVisc LRDDiffStress DeardorffDiffStress SpalartAllmaras SpalartAllmarasDDES SpalartAllmarasIDDES

Transport models
Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerLaw interfaceProperties

open source CFD on ANSELM OpenFOAM - Features

Transport models ThermoProperties

Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerLaw interfaceProperties icoPolynomial perfectGas hConstThermo janafThermo specieThermo constTransport

open source CFD on ANSELM OpenFOAM - Features

polynomialTransport sutherlandTransport hPolynomialThermo

Transport models ThermoProperties

Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerLaw interfaceProperties icoPolynomial perfectGas hConstThermo janafThermo specieThermo constTransport P1 fvDOM viewFactor

open source CFD on ANSELM OpenFOAM - Features

polynomialTransport sutherlandTransport hPolynomialThermo

Radiation Models

Transport models ThermoProperties

Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerLaw interfaceProperties icoPolynomial perfectGas hConstThermo janafThermo specieThermo constTransport P1 fvDOM viewFactor

open source CFD on ANSELM OpenFOAM - Features

polynomialTransport sutherlandTransport hPolynomialThermo

Radiation Models Reaction Models

hPsiMixtureThermo hsPsiMixtureThermo hRhoMixtureThermo hsRhoMixtureThermo

homogeneousMixture inhomogeneousMixture veryInhomogeneousMixture dieselMixture

basicMultiComponentMixture multiComponentMixture reactingMixture egrMixture

Mesh Generation
blockMesh extrudeMesh extrude2DMesh extrudeToRegionMesh snappyHexMesh

open source CFD on ANSELM OpenFOAM - Features

Mesh Generation Mesh Manipulation

blockMesh extrudeMesh extrude2DMesh extrudeToRegionMesh snappyHexMesh checkMesh mergeMeshes moveMesh polyDualMesh refineMesh renumberMesh splitMesh transformPoints

open source CFD on ANSELM OpenFOAM - Features

autoRefineMesh modifyMesh refineHexMesh refinementLevel refineWallLayer

Mesh Generation Mesh Manipulation

blockMesh extrudeMesh extrude2DMesh extrudeToRegionMesh snappyHexMesh checkMesh mergeMeshes moveMesh polyDualMesh refineMesh renumberMesh splitMesh transformPoints

open source CFD on ANSELM OpenFOAM - Features

Mesh Conversion
ansysToFoam cfx4ToFoam fluentMeshToFoam foamMeshToFluent foamToStarMesh gambitToFoam gmshToFoam ideasUnvToFoam mshToFoam netgenNeutralToFoam plot3dToFoam star4ToFoam tetgenToFoam writeMeshObj

autoRefineMesh modifyMesh refineHexMesh refinementLevel refineWallLayer

Pre-Processing
applyBoundaryLayer applyWallBoundaryConditions changeDictionary mapFields

open source CFD on ANSELM OpenFOAM - Features

setFields viewFactorsGen

Pre-Processing
applyBoundaryLayer applyWallBoundaryConditions changeDictionary mapFields

open source CFD on ANSELM OpenFOAM - Features

setFields viewFactorsGen

Post-Processing
foamDataToFluent foamToEnsight foamToEnsightParts

foamToGMV foamToTecplot360 foamToVTK

Co Lambda2 Mach

OpenFOAM case
Constant - contains a full description of the case mesh in a
subdirectory polyMesh and files specifying physical properties for the application concerned, e.g.transportProperties.

open source CFD on ANSELM OpenFOAM case

case constant
properties

system
controlDict fvSchemes fvSolution decomposeParDict

System - for setting parameters associated with the solution

procedure itself.

Time directories - containing individual files of data for

particular fields

polyMesh
Points Faces Cells boundary

Processors - directories are named processorN

where N=0,1 represents a processor number and contains a time directory, containing the decomposed field descriptions, and a constant/polyMesh directory containing the decomposed mesh description.

0 time directories > 0

processors*

open source CFD on ANSELM

OpenFOAM on ANSELM

Available engineering modules

[brz021@login2.anselm ~]$ module avail

open source CFD on ANSELM

------------------- /opt/modules/modulefiles/engineering -------------------ansys/14.5.x matlab/R2013a-EDU comsol/43b-COM openfoam/2.2.1-gcc481-openmpi1.6.5-DP comsol/43b-EDU openfoam/2.2.1-gcc481-openmpi1.6.5-SP lsdyna/7.x.x openfoam/2.2.1-icc-openmpi1.6.5-DP matlab/R2013a-COM

To create OpenFOAM environment on ANSELM give the commands:

[brz021@login2.anselm ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP [brz021@login2.anselm ~]$ source $FOAM_BASHRC

Getting Started

open source CFD on ANSELM

Create a project directory within the $HOME/OpenFOAM directory named <USER>-2.2.1 and create a directory named run within it, e.g. by typing: [brz021@login2.anselm ~]$ mkdir -p $FOAM_RUN

Project directory is now available by typing

[brz021@login2.anselm ~]$ cd /home/brz021/OpenFOAM/brz021-2.2.1/run

or
[brz021@login2.anselm ~]$ cd $FOAM_RUN

Copy the tutorial examples directory in the OpenFOAM distribution to the run directory
[brz021@login2.anselm ~]$ cp -r $FOAM_TUTORIALS $FOAM_RUN

Run the first example case of incompressible laminar flow in a cavity:

[brz021@login2.anselm ~]$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity [brz021@login2.anselm ~]$ blockMesh [brz021@login2.anselm ~]$ icoFoam

Running Serial Applications

Create a Bash script:

open source CFD on ANSELM

pokus.sh
#!/bin/bash module load openfoam/2.2.1-icc-openmpi1.6.5-DP source $FOAM_BASHRC # source to run functions . $WM_PROJECT_DIR/bin/tools/RunFunctions cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity runApplication blockMesh runApplication icoFoam

Job Submission
$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh

open source CFD on ANSELM

OpenFOAM on ANSELM
Running applications in parallel

Running applications in parallel

#!/bin/bash #PBS N motorBike #PBS l select=2:ncpus=16 #PBS l walltime=01:00:00 #PBS q qprod #PBS A IT4I-1-4

open source CFD on ANSELM

pokusParallel.pbs

module load openfoam/2.2.1-icc-openmpi1.6.5-DP source $FOAM_BASHRC cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike nproc = 32 mpirun hostfile ${PBS_NODEFILE} np $nproc snappyHexMesh overwrite parallel | tee snappyHexMesh.log mpirun hostfile ${PBS_NODEFILE} np $nproc potentialFoam -noFunctionObject-writep parallel | tee potentialFoam.log mpirun hostfile ${PBS_NODEFILE} np $nproc simpleFoam parallel | tee simpleFoam.log

Running applications in parallel

decomposeParDict The mesh and fields are decomposed using the decomposePar utility.
properties

open source CFD on ANSELM

case constant polyMesh

Points Faces Cells boundary

system
controlDict fvSchemes fvSolution decomposeParDict

0 time directories > 0

processors*

Running applications in parallel

numberOfSubdomains 32;
method simpleCoeffs { n delta } hierarchicalCoeffs { n delta order } manualCoeffs { dataFile } simple;

open source CFD on ANSELM

decomposeParDict

( 4 4 2 ); 0.001;

( 1 1 1 ); 0.001; xyz;

"";

Running applications in parallel

open source CFD on ANSELM

simple - simple geometric decomposition in which the domain is split into pieces by direction

hierarchical - hierarchical geometric decomposition which is the same as simple except the user specifies the order in which the directional split is done
scotch - scotch decomposition which requires no geometric input from the user and attempts to minimise the number of processor boundaries. manual - manual decomposition, where the user directly specifies the allocation of each cell to a particular processor.

Running applications in parallel

Create a Bash script:

open source CFD on ANSELM

pokus.sh
#!/bin/bash module load openfoam/2.2.1-icc-openmpi1.6.5-DP source $FOAM_BASHRC # source to run functions . $WM_PROJECT_DIR/bin/tools/RunFunctions cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike runApplication blockMesh runApplication decomposePar

Job Submission
$ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh $ qsub pokusParallel.pbs

open source CFD on ANSELM

OpenFOAM on ANSELM
Compile an own solvers

Compile an own solver

Initialize OpenFOAM environment before compiling your solver

open source CFD on ANSELM

[brz021@login2.anselm ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-DP [brz021@login2.anselm ~]$ source $FOAM_BASHRC $ cd $FOAM_RUN/

Create directory applications/solvers in user directory

$ mkdir p applications/solvers $ cd applications/solvers

Copy icoFoam solvers source files

$ cp r $FOAM_SOLVERS/incompressible/icoFoam/ My_icoFoam $ cd My_icoFoam

Rename icoFoam.C to My_icoFOAM.C

$ mv icoFoam.C My_icoFoam.C

Compile an own solver

Edit files file in Make directory:
icoFoam.C

open source CFD on ANSELM

EXE = $(FOAM_APPBIN)/icoFoam
My_icoFoam.C EXE = $(FOAM_USER_APPBIN)/My_icoFoam

In directory My_icoFoam give the compilation command:

$ wmake

open source CFD on ANSELM

BENCHMARKS

open source CFD on ANSELM Benchmarks

Boeing 787 Dreamliner

Boeing 787 Dreamliner

106.2 M Cells 637.2 M Unknowns Mesh generated by snappyHexMesh simpleFoam k-omegaSST 2000 time step iter.
p U k omega

open source CFD on ANSELM Benchmarks

solver parameters
- GAMG - smoothSolver - smoothSolver - smoothSolver

Tolerance relativeTolerance

1e-7 0.1

Boeing 787 Dreamliner

open source CFD on ANSELM Benchmarks

open source CFD on ANSELM Benchmarks

Lamborghini Aventador

Lamborghini Aventador
225.3 M Cells 1351.8 M Unknowns Mesh generated by snappyHexMesh simpleFoam k-omegaSST 2000 time step iter.
p U k omega

open source CFD on ANSELM Benchmarks

solver parameters
- GAMG - smoothSolver - smoothSolver - smoothSolver

Tolerance relativeTolerance

1e-8 0.001

Lamborghini Aventador

open source CFD on ANSELM Benchmarks

Hydraulic Valve
52.3 M Cells Mesh generated by ANSYS Workbench simpleFoam k-omegaSST 1000 time step iter.

open source CFD on ANSELM Benchmarks

Hydraulic Valve

open source CFD on ANSELM Benchmarks

Hydraulic Valve
2000 1500 Time [s] 1000 500 0 6 14 N Cells [M]

open source CFD on ANSELM Benchmarks

32

64

256

52

Thermal Sensor
5.3 M Cells Mesh generated by ANSYS Workbench chtMultiRegionSimpleFoam k-epsilon 1000 time step iter. 11 regions

open source CFD on ANSELM Benchmarks

Thermal Sensor

open source CFD on ANSELM Benchmarks

Lamp
17.3 M Cells Mesh generated by ANSYS Workbench chtMultiRegionSimpleFoam fvDOM Radiation Model laminar 20000 time step iter. 6 Regions

open source CFD on ANSELM Benchmarks

Lamp

open source CFD on ANSELM Benchmarks

Lamborghini Aventador
Transient solver - pisoFoam 23 M Cells nCorrectors 2 k-omegaSST solver parameters
p U k omega - PCG - PBiCG - PBiCG - PBiCG

open source CFD on ANSELM Benchmarks

Lamborghini Aventador

open source CFD on ANSELM Benchmarks

20 000 time steps solution time - 40 hr 256 cores

VSB Campus

open source CFD on ANSELM Benchmarks

VSB Campus
18 M Cells Mesh generated by snappyHexMesh simpleFoam k-epsilon 2000 time step iter.

open source CFD on ANSELM Benchmarks

VSB Campus

open source CFD on ANSELM Benchmarks

open source CFD on ANSELM

snappyHexMesh

Stonavka - Ole

open source CFD on ANSELM Benchmarks

Stonavka - Ole

open source CFD on ANSELM Benchmarks

snappyHexMesh
1,51

open source CFD on ANSELM Benchmarks

VSB Campus

1,01

0,51

0,01

20

40

60

80

100

120

140

Boeing 787 Dreamliner

snappyHexMesh scalability
N cores Time [h] 256 1.5 512 2

open source CFD on ANSELM Benchmarks

1024 1.9

Lamborghini Aventador
snappyHexMesh scalability
N cores Time [h] 128 3.7 256 2.5

open source CFD on ANSELM Benchmarks

512 3.1

1024 3.7

Supermarine Spitfire

open source CFD on ANSELM Benchmarks

VW Beetle

open source CFD on ANSELM Benchmarks

Other Open Source

Elmer Code Aster Code Saturne Palabos SU 2

open source CFD on ANSELM Benchmarks

benchmarket.it4i.cz

open source CFD on ANSELM Benchmarks