Considerations For HPC Cluster Compute Types - IBM Cloud Docs

19/04/2024, 10:00 Considerations for HPC cluster compute types | IBM Cloud Docs
Product guide
English
IBM Spectrum LSF On this page
Benchmarking of
Get started
Considerations for HPC cluster specific workloads
Getting started with IBM Spectrum

LSF
compute types Electronic Design
Automation (EDA)
Last updated 2023-04-21
About IBM Spectrum LSF Weather (WRF Model)
Considerations for HPC cluster HPC workloads have varying requirements in terms of CPU, DoE (Department of
Energy)
compute types memory, network, and storage resource requirements.
benchmarking
Deployment values You start with the following: Virtual machine use
cases
Integrating existing infrastructure
core count
resources
memory per core
Release notes
network bandwidth and latency
Tutorials processor clock speed
Setting up an HPC cluster The goal is to pick a compute configuration that returns the best
Integrating OpenLDAP with IBM price performance. An HPC workload can be as simple as a single
Spectrum LSF core job or as complex as a job that needs hundreds to thousands
of cores. HPC aggregates computing power (clustering) and can
Integrating Active Directory with deliver much higher performance and solve large problems.
Most of HPC workloads fall below the 1000 core range. There are
Expand all | Collapse all
some workloads that require a high number of cores up to and
including 10k-50k. Execution time for an HPC workload can be
https://cloud.ibm.com/docs/ibm-spectrum-lsf?topic=ibm-spectrum-lsf-considerations-for-HPC-custer-compute-types 1/12
simple and last only a few seconds, or it can be complex and take
several days.
Product guide As an example, EDA (Electronic Design Simulation) workloads have

IBM Spectrum LSF component level simulation jobs that require millions of such jobs
ran every day, but each single job requires a single core and
approximately 10 seconds. Others such as Optical Proximity
Correction can take multiple hours or even days depending on the
size of the chip and the size of the HPC cluster.
LSF
About IBM Spectrum LSF Typically HPC clusters contain a set of virtual machines and use the
cluster to run multiple workloads. It all depends on the workload
Considerations for HPC cluster resource requirements and duration.
compute types
To address this range of characteristics, IBM Cloud® provides
Deployment values
various VPC virtual machine (VM) configurations in different modes:
Balanced configurations that provide average 4 GB memory

per core and can range 4 - 64 Gbps in network bandwidth
Release notes
Compute-intensive configurations that provide 2 GB memory
per core and a network of 4 - 80 Gbps
Memory-intensive configurations that provide 8 - 28 GB

Setting up an HPC cluster
memory per core and a network of 2 - 80 Gbps
Integrating OpenLDAP with IBM
Spectrum LSF For more information, see Instance profiles.
Integrating Active Directory with For all of the configurations, the range of core count is 2 - 128 per
virtual machine. There is a special ultra high memory virtual
machine type that might be applicable for workloads that require
|
more memory per core. This type can go up to 200 cores and as
high as 28 GB per core.
The network bandwidth on a single NIC can reach maximum 16

Gbps. If a higher bandwidth is wanted, more NIC configurations
that go up to 80 Gbps might be needed. Under these
Product guide circumstances, 5 NICs would need to be configured for the virtual
IBM Spectrum LSF
machine.
By default, hyper-threading is enabled on an IBM Cloud virtual

machine, so you get 2 vCPUs per physical core. But this can be
disabled easily.
LSF
About IBM Spectrum LSF

Note: Most HPC applications perform best with one
Considerations for HPC cluster process or thread per physical core.
compute types
Deployment values For communication-intensive workloads that can fit on a single

virtual machine, it might be good to pick the best match and go up
to 128 core virtual machine instance instead of splitting the
workload across multiple virtual machine instances of a smaller
Release notes core count. This allows the process to take advantage of faster
communication through shared memory on a single virtual machine
rather than communication across multiple virtual machines over
Setting up an HPC cluster an ethernet network.
Integrating OpenLDAP with IBM To put it in perspective, two processes running on the same virtual
Spectrum LSF machine might be able to communicate in a fraction of a
Integrating Active Directory with microsecond (for example, 0.3 microseconds) whereas across two
virtual machine instances it can take more than 30 microseconds. A
factor of 100 times faster communication when it's through shared
|
memory in a single virtual machine.
A very cost effective configuration is cx2-128x256, which allows

128 cores and 2 GB memory per core. This can cover a broad range
of MPI workloads.
Product guide Note: Scalable MPI jobs can be set up that require
IBM Spectrum LSF multiple virtual machines that are configured at up to 80
Gbps apiece, but that requires multiple NICs and might
not be desirable. It is recommended to pick a
configuration that provides the best network bandwidth
per core with single NIC. bx2-16x64 might be a good
LSF
starting point for your MPI benchmarking.
Considerations for HPC cluster

compute types Benchmarking of specific workloads
Deployment values Electronic Design Automation (EDA)

IBM Systems and IBM Research work in this industry domain and
have successfully used IBM Cloud for such workloads. The
following graph displays a scale test for up to 30 K cores. To
Release notes
showcase how cloud zones can be used as a single data center, we
built a large HPC cluster aggregating the resources across three
IBM Cloud locations. The setup also uses IBM Storage Scale as a
scratch-based, high-performing file system along with IBM
Integrating OpenLDAP with IBM Spectrum LSF for HPC cluster configuration. We have used BX2-
Spectrum LSF 48x192 for IBM Spectrum LSF worker nodes and MX2d-16x128 for
storage nodes under the IBM Storage Scale.
Integrating Active Directory with
Zoom
|
Product guide
IBM Spectrum LSF

LSF

compute types
Deployment values
Release notes Weather (WRF Model)

IBM Cloud shows linear performance, performs favorably, and can
scale well into thousands of cores. The virtual machine
Figure 1. EDA diagram
configuration that is used for this benchmark is bx2-16x64. The
Integrating OpenLDAP with IBM WRF model is not sensitive to network latency as it packs many
Spectrum LSF variables into each message, resulting in fairly large messages, and
not many small messages.
Zoom
Product guide
IBM Spectrum LSF

LSF

compute types
Deployment values
The red line represents the HPC environment with the InfiniBand
Release notes HDR that gives highest bandwidth, lowest latency, and is the best
configuration for such workloads. The green line shows IBM Cloud
with the Lon2 data center for the benchmarking. The blue line is
Figure 2. WRF diagram
Setting up an HPC cluster Summit super computer. In summary, any workloads that have
characteristics similar to the WRF model should scale well with IBM
Spectrum LSF Cloud. As you can see IBM Cloud shows reasonable performance
against state-of-the-art HPC systems.
DoE (Department of Energy) benchmarking

|
SNAP and Quicksilver are two applications that the DoE uses for
benchmarking and making decisions on specific commodity
technology systems.
The following graphs show results on how IBM Cloud compares

with the state-of-the-art HPC system.
Product guide On IBM Cloud, the benchmarks used two different configurations:
IBM Spectrum LSF
bx2-8x32
bx2-16x64
Getting started with IBM Spectrum As you can see, SNAP results show that bx2-8x32 provides more
LSF performance because of higher effective network bandwidth ratio
per core; where Quicksilver does well with bx2-16x64 as it has
moderate communication requirements and is mostly near-
Considerations for HPC cluster neighbor to track particle motion across the global domain.
compute types
Zoom
Deployment values
Release notes

Spectrum LSF
Product guide
IBM Spectrum LSF
Zoom

LSF

compute types
Deployment values
Release notes
Figure 3. SNAP scaling diagram


Spectrum LSF

Figure 4. Quicksilver scaling diagram
Even though the scaling is not as good as the weather model, IBM
|
Cloud can scale up to thousands of cores reasonably with a linear
curve.
Virtual machine use cases

The choice of virtual machine profile type for IBM Cloud depends
Product guide on your workload details on cores, memory, and network
IBM Spectrum LSF requirements.
Single node virtual machine use cases

Getting started with IBM Spectrum This should be the first evaluation as it can provide the best price
LSF and performance for running such jobs on IBM Cloud.

You can choose from the following set of virtual machine profiles:
CX2-16x32 to CX2-128x256
compute types
BX2-16x64 to BX2-128x512
Deployment values
MX2-16x128 to MX2-128x1024
Depending on the memory required per core, you might pick the
MX2 configuration, which can support up to 1 TB on the MX2-
Release notes 128x1024 profile.
If the memory required per core is less than 2 GB, an appropriate

Setting up an HPC cluster CX2 profile might give you the best price and performance. The
advantages of faster communication over shared memory help with
Integrating OpenLDAP with IBM the performance if the workload can run on a single virtual
Spectrum LSF
machine.
Some examples of such workloads:
| Local area weather forecasting, not high resolution, large

models but modest in size
OpenFOAM computational fluid dynamics, size of the problem
between 2 million - 10 million grid cells

Design rule checking (DRC) in Electronic Design Automation
(EDA) as part of chip designs
Product guide
IBM Spectrum LSF EDA single component simulation and verification jobs
Scale out use case with multiple virtual machines
Getting started with IBM Spectrum This is the second category of workloads where the requirement is
LSF to have multiple virtual machines for execution. Even for this
category, depending on the network bandwidth and latency
About IBM Spectrum LSF requirements, a significant set of HPC workloads would scale well
Considerations for HPC cluster on IBM Cloud.

compute types
There are cases where small updates might be needed to the
Deployment values application code to adjust it well to run in a cloud-like environment,
and these changes are not specific to IBM Cloud but would be
desirable to burst to any cloud provider.
Release notes IBM carries deep HPC expertise and can provide specific
recommendations to get you the best price and performance for an
HPC cloud environment.

IBM Cloud has been successful in the following examples:
Optical Proximity Correction (OPC) in EDA
Spectrum LSF
Full chip Integrated Circuit Validator (ICV) in EDA
Any Hadoop map/reduce or Spark workload
MPI workloads that cannot fit on a single virtual machine
|
The recommendation for such workloads is to start with BX2-16x64

as it gives the best performance in tests so far. Based on your
results, you might get insight into alternative options that might
better suit your specific workload. Some workloads might be

network-latency sensitive, and IBM Cloud configurations might not
appear promising; yet even for these cases, engage with the
Product guide offering owner and your sales team so the HPC experts can
IBM Spectrum LSF
evaluate your specific requirements and provide assistance.

Contribute in GitHub
LSF
Open doc issue Edit topic

compute types
Deployment values
Release notes

Spectrum LSF
Product guide
IBM Spectrum LSF

LSF

compute types
Deployment values
Release notes

Spectrum LSF

Considerations For HPC Cluster Compute Types - IBM Cloud Docs

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Considerations For HPC Cluster Compute Types - IBM Cloud Docs

Uploaded by

Copyright:

Available Formats

19/04/2024, 10:00 Considerations for HPC cluster compute types | IBM Cloud Docs

Getting started with IBM Spectrum

Product guide As an example, EDA (Electronic Design Simulation) workloads have

Balanced configurations that provide average 4 GB memory

Memory-intensive configurations that provide 8 - 28 GB

The network bandwidth on a single NIC can reach maximum 16

By default, hyper-threading is enabled on an IBM Cloud virtual

About IBM Spectrum LSF

Deployment values For communication-intensive workloads that can fit on a single

A very cost effective configuration is cx2-128x256, which allows