Submitted by Gergana Slavova...

(/en-us/user/198987) on Mon, 12/21/2009 - 14:31

Share (https://www.facebook.com/sharer/sharer.php?u=https://software.intel.com/en-us/articles/integrating-intel-mpi-sge)

Tweet (https://twitter.com/intent/tweet?text=Check out the Intel IDZ%3A&url=https://software.intel.com/en-us/articles/integrating-intel-mpi-sge

3
Share (https://plus.google.com/share?url=https://software.intel.com/en-us/articles/integrating-intel-mpi-sge)
So, you want to use Intel MPI Library with the Sun* Grid Engine* (SGE) batch scheduler?
The below instructions describe how to run Intel MPI Library jobs using Sun Grid Engine. This document
relates to Linux*. While there are some differences and additional steps when using Microsoft* Windows*, in
general the procedure is the same.
All optional steps are recommended but not necessary for successful integration.
[Optional] Visit sun.com (http://www.sun.com/software/gridware/) and get a brief overview of SGE 1.
Installation
See the Installation Guide (http://docs.sun.com/app/docs/doc/820-0697?a=load) from sun.com for
details. Roughly, the steps are as follows:
Install Master Host (see How to Install the MasterHost section);
Install Execution Host (see How to Install ExecutionHosts section);
Register Administration Hosts (see the corresponding section in the Installation Guide);
Register Submit Hosts (see corresponding section);
Verify the installation (see corresponding section).
IMPORTANT NOTES:
To finalize the installation process, youll have to configure the network services manually (by
modifying /etc/services), which requires root privileges.
Its possible to install/run SGE as a non-privileged user, but
there are some limitations in that case; 1.
you need root privileges for the complete installation process (at least, for modifying
/etc/services).
2.
2.
Create a new Parallel Environment (PE) for Intel MPI Library
Create the appropriate configuration file for the new PE. It should contain the following lines:
pe_name impi
slots 999
user_lists NONE
xuser_lists NONE
start_proc_args NONE
stop_proc_args NONE
allocation_rule $round_robin
control_slaves FALSE
job_is_first_task FALSE
urgency_slots min
1.
Add the new PE using the following command: 2.
3.
Integrating Intel MPI Library with Sun Grid Engine
Developer Zone
=
Development
^
Tools
^
Resources
^ ~ 9
Join Today Log in ^
Look for us on:
M
English^
What can we help you find today?
Integrating Intel MPI Library with Sun Grid Engin... https://software.intel.com/en-us/articles/integratin...
1 of 4 09/05/2014 07:51 PM
qconf Ap <config_file>
USEFUL COMMANDS:
* qconf spl view all PEs currently available;
* qconf sp <PE_name> - view settings for a particular PE;
* qconf dp <PE_name> - remove a PE;
* qconf mp <PE_name> - modify an existing PE.
Also see the Managing Special Environment section in the Administration Guide
(http://docs.sun.com/app/docs/doc/820-0698?a=load) from sun.com if you need more details about
PE configuration.
Associate a queue with the new PE
Use the following commands for that:
qconf sql to see all queues available; 1.
qconf mq <queue_name> - to modify the queues settings. Find the pe_list property in the
open window and add the impi string to that property.
2.
USEFUL COMMANDS:
* qconf sq <queue_name> - view the queues settings.
See the Administration Guide if you need more details about the queue configuration process.
4.
Add Intel MPI Library environment to your current environment by sourcing the appropriate
mpivars.[c]sh script located in the <install_dir>/bin[64] directory
5.
Build the MPI application to be run 6.
[Optional] Make sure that Intel MPI Library works fine on the desired hosts. For this, manually run your
application on the desired hosts individually
7.
Submit your MPI job to SGE
Use the following command for that:
qsub -N <job_name> -pe impi <num_of_processes> \
-V <mpirun_absolute_name> -r ssh -np
<num_of_processes> <app_absolute_name>
where
-V option is used so that all environment variables available in the current shell are exported to a job.
USEFUL COMMANDS to monitor and control jobs:
* qstat show status of SGE jobs and queues;
* qstat j show detailed information about jobs (can be useful for pending jobs);
* qdel remove existing job.
After submitting the job you can monitor its status using the qstat command. When the job is finished,
you can find the jobs output and error output in your HOME directory just look for
<job_name>.o<jobID> and <job_name>.e<jobID> files.
See the Users Guide (http://docs.sun.com/app/docs/doc/820-0699?a=load), if you need more
information about the job submission process.
8.
Closer integration with SGE
Read the 'Tight Integration of Parallel Environments and Grid Engine Software' section in SGE's
Administration Guide (http://docs.sun.com/app/docs/doc/820-0698?a=load) first.
To enable tight integration for Intel MPI, use the same procedure as the one mentioned above, but use a
different configuration file for the PE at step #3.
The configuration file should contain the following lines:
pe_name impi_tight
slots 999
Look for us on:
M
English^
Integrating Intel MPI Library with Sun Grid Engin... https://software.intel.com/en-us/articles/integratin...
2 of 4 09/05/2014 07:51 PM
user_lists NONE
xuser_lists NONE
start_proc_args <SGE_install_dir>/mpi/startmpi.sh -catch_rsh $pe_hostfile
stop_proc_args <SGE_install_dir>/mpi/stopmpi.sh
allocation_rule $round_robin
control_slaves TRUE
job_is_first_task FALSE
urgency_slots min
Categories: Intel MPI Library (/en-us/search/site/language/en?query=Intel%C2%AE%20MPI%20Library) ,
Linux* (/en-us/search/site/language/en?query=Linux%2A)
Tags: sge (/en-us/search/site/field_tags/sge-21618/language/en?query) , sun grid engine (/en-us/search
/site/field_tags/sun-grid-engine-21619/language/en?query)
For more complete information about compiler optimizations, see our Optimization Notice (optimization-
notice).
Comments (2)
*Top
said on Wed, 05/26/2010 - 03:02
reuti_at_intel said on Mon, 10/07/2013 - 09:54
These instructions will lead as a first setup to a loose integration of Intel MPI into SGE. To allow SGE to have full control of all
started slave tasks and also record a correct accounting (also called a tight integration then), some additional steps are necessary,
which will then make use of "control_slaves TRUE":
gridengine.sunsource.net/howto/mpich2-integration/mpich2-integration.html (mpd startup method)
+
I posted the above comment when Intel MPI was still using `mpdboot` to set up the daemons and SGE lost control over them as
they forked into daemon land. The mentioned Howto for this older version to keep them under SGE control can still be found at
http://arc.liv.ac.uk/SGE/howto/mpich2-integration/mpich2-integration.html
With version 4.1 of Intel MPI they also changed to start up by Hydra and just a plain definition of a parallel environment (PE) of:
$ qconf -sp intel-mpi
pe_name intel-mpi
slots 999
user_lists NONE
xuser_lists NONE
start_proc_args NONE
stop_proc_args NONE
allocation_rule $round_robin
control_slaves TRUE
job_is_first_task TRUE
urgency_slots min
accounting_summary TRUE
will do. This PE is identical to the one which can be used with Open MPI.
Disclaimer: There is a bug in MPICH2/3 in case you defined several queues in GridEngine per exechost and it may create a wrong
allocation: http://lists.mpich.org/pipermail/discuss/2013-September/001455.html
+
Look for us on:
M
English^
Integrating Intel MPI Library with Sun Grid Engin... https://software.intel.com/en-us/articles/integratin...
3 of 4 09/05/2014 07:51 PM
Terms of Use *Trademarks Privacy Cookies
(For technical discussions visit our developer forums. For site or software product issues contact support.)
Please sign in to add a comment. Not a member? Join today^
Publications^
Look for us on:
M
English^
Integrating Intel MPI Library with Sun Grid Engin... https://software.intel.com/en-us/articles/integratin...
4 of 4 09/05/2014 07:51 PM