Skip to main content

Secondary navigation:

Research Services

Information Technology Services

Torque (PBS) User Guide

How to use the queues

The Scorpio Linux Cluster is using Torque as the queuing system. Torque uses the same commands and has the same structure as PBS, another common system.  Along with Torque, we use the Moab job scheduler.

To use torque, first load the torque module (module load torque).

To submit a job to the queues, you need to create a command file (also called a script).    Here we will use the command file called test.pbs.

To submit your job:

  qsub test.pbs

Here is a basic command file:

#!/bin/tcsh
#PBS -l mem=1gb,nodes=1:ppn=1,walltime=1:00:00
#PBS -m abe -M your-email-address

module add pathscale
cd work-directory
./a.out

This file consists of three parts.

1.  The shell to use.  In this case

 #!/bin/tcsh

means to use the tcsh.  You can also use bash or sh.

2.  PBS arguments to specify wall-time, number of cores, memory and more.  These commands start with #PBS. The command

#PBS -l mem=1gb,nodes=1:ppn=1,walltime=1:00:00

requests 1 gigabyte of memory, 1 core, and 1 hour of wall clock time (wall clock time is the time from when the job start to when it completes).

Units to specify memory include kb (kilobytes), mb (megabytes) and gb (gigabytes).  The unit must be an integer.

The "nodes=1:ppn=1" requests one core.  This means you requesting one node and one core per node.  The ppn notation is historic and means processors per node. It is from a time when every processor had one core.  See the section on Parallel Jobs for instructions on how to request multiple cores.

The line #PBS -m abe is not required.  It specifies notification options.
You can use any combination of a,b, and e.  The meaning is as follows:

a  mail is sent when the job is aborted by the batch system.
b  mail is sent when the job begins execution.
e  mail is sent when the job terminates.

3.  The third part of the command file consists of commands that you want to use.  Typically, the module to load, then the actual command to execute the code you want to run. 

Notes: 

1.  PBS starts from your home directory.  Adding a cd command to take you to the directory where your files are is usually good to add to the commands.

2. Be sure the last line of the pbs file has a carriage return in it (Be sure it has a blank line at the end)

Requesting Specific Nodes

Some software, such as aimall is licensed only on specific nodes.  To request nodes in which Aimall is licensed,

#PBS -l mem=1gb,nodes=aimall+1:ppn=1,walltime=1:00:00

You may also request the type of node.  If you don't request a type of node, your job will run on any available core.  To see the type of nodes, type

  pbshosts

The "properties" column shows the types of nodes that you can request

To specify  4 HPintel8 nodes using 16 cores per node with 32 GB of total memory:

#PBS -l mem=32g,nodes=HPintel8+4:ppn=8,walltime=1:00:00

Parallel Jobs

To request more than one core, you can request nodes and cores per node (recommended) or just request the total number of cores

a) To request 4 cores, on a single node:

#!/bin/tcsh
#PBS -l pmem=4gb,nodes=1:ppn=4,walltime=1:00:00
#PBS -m abe -M your-email-address

b) To request 8 cores, on two nodes:

#!/bin/tcsh
#PBS -l pmem=1gb,nodes=2:ppn=4,walltime=1:00:00
#PBS -m abe -M your-email-address

Notes: 

  1.  In this example, we used pmem, not mem.  This request memory per process, you can also use mem, which is total memory requested.
  2. It is best to specify either intel8 or HPintel8 when running multiprocessor jobs on multiple nodes.  This will give you nodes with the same processor speed.

Optional Job Control Parameters

The following parameters are optional.  If you don't specify them, torque will use default names.

 a) To name your job, add:

#PBS -N your-choice-of-job-name

b) To specify the name of the output file (where the standard output is sent), add:

#PBS -o output-file-name

This will send the output file to the output-file-name in the directory from which the job was submitted.  If you want to the output file to go to another directory,  use the complete path to the file.

#PBS -o complete-path/output-file-name

c) To specify the name of the error file. You can use just a file name, or give a complete path to the file where you want the standard error as for the output file.

#PBS -e error-file-name

Environment Variables

You can use the environment variable PBS_JOBID  in your script.  For example

#PBS -o output.$PBS_JOBID

will send the output to a file named output.Job-Number.nas-1-0.local.

Other useful commands

To delete a job:

qdel job-number

To see the status of the queues, use the Moab command

showq

To check the status of your job, use the Moab command

checkjob -v job-number

To see the number of cores per node, and the number of free cores,

           pbshosts

or to see information about only nodes with free cores

          pbshosts -f

Note:  This command is not part of the Torque distribution.