Running MITgcm simulations

From Darwin

Basic information on running MITgcm can be found at the MITgcm web site

Specific steps you need to follow on beagle are listed below:

Running interactively

• requested some CPU's to run on (we ask for 8 in the example below)

[charles@beagle] $ qrsh -now no -pe mpich_mx 8

• make sure you have the appropriate modules loaded

[charles@compute-2-1 ~]$ source /etc/profile.d/modules.sh
[charles@compute-2-1 ~]$ module add mitgcm

• compile the model using the optfile linux_amd64_ifort+mpi_beagle (or linux_amd64_ifort_beagle for a single-process run), e.g.,

[charles@compute-2-1 ~]$ ../tools/genmake2 -mpi -optfile ../tools/build_options/linux_amd64_ifort+mpi_beagle ...
[charles@compute-2-1 ~]$ make depend
[charles@compute-2-1 ~]$ make

• locate the machine file (it contains the names of the hosts you've been assigned when running qrsh). It is called something like /tmp/40320.1.darwin/machines (the number being the jobid of your job).

• run mitgcm

[charles@compute-2-1 ~]$ mpirun -np 8 -machinefile /tmp/40320.1.darwin/machines --mx-kill 30 --mx-copy-env ./mitgcmuv

Running in batch

Running repeated batch jobs

#!/bin/bash
#
# SGE queue system job script that requests some four-core nodes,
# waits 10 seconds, submits itself again and then ends.
# SGE directives begin with #$ in column 1
# o Directive # #$ -j y
#   means put standard output and standard error in the same file.
# o Directive #$ -cwd 
#   means start the job in the directory from which it was 
#   submitted.
# o Directive #$ -q darwin
#   selects the queue named darwin - currently the only queue we have.
# o Directive #$ -pe mpich_mx 60
#   means run with exactly 60 "slots" using the parallel 
#   environment defined by the keyword "mpich_mx". This gives
#   the job 60 CPU's with the CPU's selected to be on the
#   minimum number of nodes.
# o Directive #$ -N sge_demo_001
#   means name this job "sge_demo_001".
#
#$ -j y
#$ -cwd
#$ -q darwin
#$ -pe mpich_mx 4
#$ -N sge_demo_001
#
# Lots more SGE information can be found by reading
# the SGE man pages - see
# man qsub
# man qconf
# man sge_pe
# However, these are very dense reading.
#
echo '*********************************************************'
THEDATE=`date`
echo 'Start job '$THEDATE
echo 'NHOSTS = '$NHOSTS
echo 'NSLOTS = '$NSLOTS
echo '======= PE_HOSTFILE ======='
cat $PE_HOSTFILE
echo '==========================='
echo '======= $TMPDIR/machines ===='
cat $TMPDIR/machines

# Run the model
mpirun -np $NSLOTS -machinefile $TMPDIR/machines --mx-kill 30 --mx-copy-env ./mitgcmuv

# Submit myself again (this assumes your job script is called job.sge)
ssh beagle "cd ${SGE_CWD_PATH}; qsub job.sge"

echo '==========================='
echo 'End job '$THEDATE
echo '*********************************************************'