Kohei Fukuda

Last Update: October 23rd, 2018

ANSYS on GRAHAM
LOGIN THROUGH LINUX TERMINAL COMMAND - (LINK)

ssh -Y username@graham.computecanada.ca

LOGIN USING PUTTY (SSH CLIENT) - (LINK)

Put "graham.computecanada.ca" as your host name. Login using your Compute Canada Data Base (CCDB) user ID and password.
This can be the same as your sharcnet ID and password.

UPLOAD YOUR DATA TO GRAHAM - (LINK)

Use Globus as they recommended or use software like WinSCP to transfer your data from local PC to GRAHAM and vice versa.

CONFIGURATING THE LICENSE

In order to access to the license server, a text file "ansys.lic" needs to be created in $HOME/.license. (The folder may be hidden.
Check the folder with the command "ls -a") To do this,
1. Make directory ".licenses" (dot licenses) in the home directory or maybe it is already available.
2. Move to the directory ".licenses" and create a text file with following texts and save it as "ansys.lic".
setenv("ANSYSLMD_LICENSE_FILE", "1055@license3.sharcnet.ca")
setenv("ANSYSLI SERVERS", "2325@license3.sharcnet.ca")

The availability of the license can be checked by;

lmutil lmstat -a -c 1055@license3.sharcnet.ca

CONFIGURATING JOURNAL FILE

The file location specified in a journal file is different whether a job is submitted on Graham or Orca.
For Graham
/file/read-case "/scratch/kfukuda/05/05.cas
/file/read-data "/scratch/kfukuda/05/05.dat
/file/auto-save/data-frequency 100
/file/auto-save/root-name "/scratch/kfukuda/05/05
/parallel/partition/method cartesian-axes 32
/file/confirm-overwrite y
/solve/set/data-sampling yes 1 yes yes
/solve/dual-time-iterate 10000 20
exit
yes

SUBMITTING JOBS TO THE QUEUE – ON ANY SYSTEM

Jobs are submitted through Slurm scheduler with extension ".sh".
[someuser@host ~]$ sbatch simple job.sh
The ".sh" file contains the number of CPUs, the size of memory, job time, the module that you want to run, your simulation file, etc.
The script in the ".sh" file looks like below;

For ANSYS Fluent:

 Kohei Fukuda
Last Update: October 23rd, 2018

#!/bin/bash
#SBATCH --time=0-01:00 # time (DD-HH:MM)
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --mem-per-cpu 2000M
module load ansys/18.2

slurm_hl2hl.py --format ANSYS-FLUENT > machinefile

NCORE=$((SLURM_NTASKS * SLURM_CPUS_PER_TASK))

fluent 3ddp -t $NCORE -cnf=machinefile -mpi=intel -g -i fluent.jou

TIME LIMITS
Graham will accept jobs of up to 28 days in run-time. However, jobs of that length will be restricted to use only a small fraction of the
cluster. (Approximately 10%, but this fraction is subject to change without notice.)
There are several partitions for jobs of shorter and shorter run-times. Currently there are partitions for jobs of
 3 hours or less,
 12 hours or less,
 24 hours (1 day) or less,
 72 hours (3 days) or less,
 7 days or less, and
 28 days or less.
Because any job of 3 hours is also less than 12 hours, 24 hours, and so on, shorter jobs can always run in partitions with longer time-
limits. A shorter job will have more scheduling opportunities than an otherwise-identical longer job.

HOWTO MONITORING SUBMITTED JOBS

squeue -u <USERID>

To list only running jobs or pending jobs

squeue -u <USERID> -t RUNNING
squeue -u <USERID> -t PENDING

To show detailed information for a specific job

scontrol show job -dd <jobID>
To show completed jobs
sacct -j <jobID>

HOWTO CANCEL JOBS

scancel <jobID>

In order to cancel all jobs or all pending jobs

scancel -u <USERID>
scancel -t PENDING -u <USERID>

HOWTO CHECK DISK USAGE

 Kohei Fukuda
Last Update: October 23rd, 2018

diskusage_report
or

quota

HOW TO RECEIVE A NOTIFICATION AT JOB START

#SBATCH --mail-user=<email_address>
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=REQUEUE
#SBATCH --mail-type=ALL

HOWTO MONITOR SOLUTION RESIDUALS

tail -f slurm-<jobID>.out

HOW TO OPEN A SIMULATION WITH GUI

Use tigervnc to access the visual workstation and open a simulation with GUI.

The workstation is "gra-vdi.computecanada.ca".

[kfukdua@gra-vdi4 ~] module load sn

[kfukdua@gra-vdi4 ~] module load ansys/19.1
[kfukdua@gra-vdi4 ~] fluent-gui

or

[kfukdua@gra-vdi4 ~] cfx-gui

HOW TO LIST THE FILES WHICH IS IN THE SCRATCH AND IS EXPIRING

Scratch file expiration currently occurs on Sunday mornings each week,

and all files older than the current scratch expiration period for this
cluster of 62 days at that time will be expired.

You can view the list of currently due to expire files by logging into the saw
cluster, changing directories to the relevant scratch directory (ie: "cd /scratch/userid")
and typing the following command:

find . -type f -mtime +62 -ls

This will give you the full details of all files in your directory older than 62 days.
If you wish to see the files due to expire on Sunday morning, you should reduce the number
above from 62 by the number of days until Sunday. For example, on a Friday, you
 Kohei Fukuda
Last Update: October 23rd, 2018
would use:

find . -type f -mtime +60 -ls

It should be noted that in scratch directories that contain very large numbers of
files and/or subdirectories, this command can take some time to run.

HOW TO CHECK USER STATUS

id <userid>