FAQ about PBS
FAQ about PBS
1. What is PBS?
Portable Batch System (or simply PBS) is the name of computer software that performs job scheduling. Its primary task is to allocate computational tasks, i.e., batch jobs, among the available computing resources. It is often used in conjunction with UNIX cluster environments. (fromPBS Wiki page [1])
2. How to submit a task to PBS?
You could submit a task to PBS with qsub from command line, e.g.,
qsub -q <queue_name> -N <job_name> -l nodes=1:ppn=1,mem=1gb,walltime=01:00:00 \
-o <path_to_out_log> -e <path_to_err_log> <script_to_run>or through a qsub script (recommended). One simple qsub script is like ( modified from this script [2])
#!/bin/bash
# declare a name for this job
#PBS -N <job_name>
# request the queue for this job
#PBS -q <queue_name>
# request a total of x processors for this job (y nodes and z processors per node)
#PBS -l nodes=1:ppn=1
# request memory
#PBS -l mem=1gb
# specify walltime
#PBS -l walltime=01:00:00
# out log file
#PBS -o <path_to_out_log>
# err log file
#PBS -e <path_to_err_log>
# By default, PBS scripts execute in your home directory, not the
# directory from which they were submitted. The following line
# places you in the directory from which the job was submitted.
cd $PBS_O_WORKDIR
# run the program
/path_to_executable/program_name arg1 arg2 ...
exit 03. Frequently used PBS commands
copied from this page [3]
#qsub #submit a job, see man qsub
#qdel -p jobid #will force purge the job if it is not killed by qdel
#qstat #list information about queues and jobs
#showq #calculated guess which job will run next
#xpbs #GUI to PBS commands
#qstat -q #list all queues on system
#qstat -Q #list queue limits for all queues
#qstat -a #list all jobs on system
#qstat -s #list all jobs with status comments
#qstat -r #list all running jobs
#qstat -f jobid #list full information known about jobid
#qstat -Qf queueid #list all information known about queueid
#qstat -B #list summary information about the PBS server
#qstat -iu userid #get info for queued jobs of userid
#qstat -u userid #get info for all the jobs of userid
#qstat -n -1 jobid #will list nodes on which jobid is running in one line
#checkjob jobid #will list job details4. Use conda in PBS system
4.1 Activate conda environment inside a bash script
Sometimes we failed to run a bash script in PBS when required Apps were installed into a conda environment. One solution is to use absolute pathes to the Apps, which is not convenient, especially when the number of the Apps is big. So it would be nice if there was a way to activate conda environment inside a bash script.
Here's one solution (credit to this page [4])
EDIT: seems the issue [5] has been fixed in the new version of conda, so if you are using conda v4.9.0 or above (version below v4.9.0 not tested), no need to add eval "$(conda shell.bash hook)" in script.
source ~/.bashrc # this cmd will activate conda base env.
eval "$(conda shell.bash hook)" # this will allow runing conda inside shell scripts.
/path/to/conda activate $conda_env
codes of required Apps here ...
/path/to/conda deactivate4.2 One example: install R package to certain conda environment
Sometimes installing R packages could be a challenge on institutional HPC clusters for it may violate the CPU or memory limits. Submitting the installing task to PBS is a solution. e.g. to install package ggplot2 to conda env named R-4.0.2,
#!/bin/bash
#PBS -N install_r_pkg
#PBS -q <queue_name>
#PBS -l nodes=1:ppn=1
#PBS -l mem=10gb
#PBS -l walltime=01:00:00
#PBS -o install_r_pkg.out
#PBS -e install_r_pkg.err
source ~/.bashrc
eval "$(conda shell.bash hook)"
conda activate R-4.0.2 # replace R-4.0.2 with your own conda env name
Rscript -e 'install.packages("ggplot2", repos = "http://cran.r-project.org")'
conda deactivatesave the codes above to a script named install_ggplot.sh and then run
chmod u+x install_ggplot.sh
qsub install_ggplot.sh[1] https://en.wikipedia.org/wiki/Portable_Batch_System [2] https://www.dartmouth.edu/~rc/HPC/Sample_Qsub_script.html [3] http://physics.princeton.edu/it/faq/pbs-maui-cmds.html [4] https://github.com/conda/conda/issues/7980 [5] https://github.com/conda/conda/issues/7980