Informatics Facility Documentation
The hostname of the cluster located at the Informatics Facility is:
xeuser.ivec.org
You will need an ssh client to connect to the system.
Linux and MacOSX users should be able to use command-line ssh and scp.
If you are connecting from a Windows machine, a freely available ssh client is PuTTY. To transfer files to/from the system you can use the freely available WinSCP.
Directory structure
There are several file systems available depending on user requirements. The appropriate file system should be use whenever possible.
- /home/username Home directory
Small quota, backed up. Intended for source code, executables and input files
- /scratch/project Project directory
Large quota, not backed up. Intended to be used for job data that must live beyond the lifetime of the job
Queues
There is currently only one queue available on the system:
- Normal - this is the default queue designed for production use, and has a charging rate of 1 SU per processor-hour (walltime)
Environment Modules
Environment modules are used to customise your shell environment, allowing for dynamic changes to your path environment variables. They work for all supported shells. This is the preferred way to use software that is installed on the system.
How to use modules
- module avail
will list the software currently supported by modules. If you require a package that is not listed, please contact the iVEC systems administration staff to request its installation (subject to licensing restrictions).
- module load package_name
will modify the environment settings required by package_name.
- module unload package_name
will return the environment to it's state prior to the module load command.
- module list
will show what modules you currently have loaded.
For example, the command module load intel-cc/9.0.026 will adjust your PATH environment variable to include the location of the Intel C/C++ compiler executables, your MANPATH environment variable to include the location of the Intel C/C++ compiler manpages and your LD_LIBRARY_PATH environment variable to include the location of the Intel C/C++ compiler libraries.
Module commands can be added to your shell startup scripts and to jobscripts.
Resource limits
Each interactive process you run or batch job you submit has imposed on it a time limit, a memory use limit, and a processor count limit.
Monitoring resource usage
Some commands to monitor system resource usage are:
- gbalance -p project displays the current usage for the given project, as well as the number of credits remaining (in CPU seconds). You will need to load the gold package: module load gold
- quota -v displays your disk usage and quota in your home directory
- qstat -a displays the status of all running and queued jobs.
Torque
The batch system in use on xe is Torque. You submit jobs to Torque specifying the number of CPUs, the amount of memory, and the length of time needed (and possibly other resources). Torque runs the job when the resources are available, subject to constraints on maximum resource usage.
Basic Torque commands
- qstat - Standard queue status command supplied by Torque.
- qdel jobid - Delete your unwanted jobs from the queues.
- qsub - Submit jobs to the queues. Options to qsub can be specified on the command line as follows:
% qsub -P aa99 -q normal -l walltime=20:00:00,vmem=300MB -w ./a.out ^D (control-D)
or via a jobscript:
% cat jobscript #!/bin/csh #PBS -P a99 #PBS -q normal #PBS -l walltime=20:00:00,vmem=300MB #PBS -w ./a.out
which can be submitted via the command qsub jobscript.
MPI jobs
When you want to run a MPI job, you must load either the openmpi module (openmpi compiled with GCC) or the openmpi-intel module (openmpi compiled with Intel C compiler).
You don't need to specify the number of processors to run using the -np switch, you specify the number using qsub
Each node has eight cores, so if we want to run a 16 processor job, we would run on two nodes with eight cores per node. So we would submit the following:
% cat jobscript module load openmpi mpirun a.out
using qsub -l nodes=2:ppn=8 jobscript
