Differences

This shows you the differences between two versions of the page.

--- software:specfem [2017/03/29 11:39]
wphase
+++ software:specfem [2018/03/05 16:05]
wphase
@@ Line 1: / Line 1: @@
 ====== SPECFEM3D_GLOBE ======
-==== Running SPECFEM3D_GLOBE on the Strasbourg HPC cluster with intel17 and cuda7.5 ====
+==== Running SPECFEM3D_GLOBE on the Strasbourg HPC cluster with gnu 4.8 and cuda 7.5 ====
 === Setup the environment ===
@@ Line 9: / Line 10: @@
 module load batch/slurm
 module load compilers/cuda-7.5
-module load mpi/openmpi-basic
-export CUDA_LIB=/usr/local/cuda/cuda-7.5/lib64
 export CUDA_INC=/usr/local/cuda/cuda-7.5/include
+export CUDA_LIB=/usr/local/cuda/cuda-7.5/lib64
+export PATH=/rpriv/ipgs/zac/openmpi-1.10.7/bin:$PATH
+export LD_LIBRARY_PATH=/rpriv/ipgs/zac/openmpi-1.10.7/lib:$LD_LIBRARY_PATH
+</code>
+Notice that we use default gnu compiler of the operating system:
+<code>
+$ gfortran --version
+GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
 </code>
 === Compilation ===
-First make sure that required modules are loaded and CUDA_LIB, CUDA_INC environment variables are declared (see previous section). Create a run directory including directories ''DATABASE_MPI'', ''OUTPUT_FILES'', ''bin'' and ''DATA''.
+Before compilation, make sure that required modules are loaded and CUDA_LIB, CUDA_INC environment variables are declared (see previous section). Create a run directory including directories ''DATABASE_MPI'', ''OUTPUT_FILES'', ''bin'' and ''DATA''.
 In the directory ''DATA'', create ''CMTSOLUTION'', ''Par_file'' and ''STATIONS'' file (cf., SPECFEM3D_GLOBE documentation).
@@ Line 45: / Line 52: @@
 ln -s DATA/topo_bathy $rundir/DATA/topo_bathy
 </code>
+Below is a script doing all the configuration and compilation:
+<code>
+#!/bin/bash
+# Load modules
+module purge
+module load batch/slurm
+module load compilers/cuda-7.5
+module load mpi/openmpi-basic
+export CUDA_LIB=/usr/local/cuda/cuda-7.5/lib64
+export CUDA_INC=/usr/local/cuda/cuda-7.5/include
+# source directory
+rootdir=/b/home/ipgs/cmorales/specfem3d_globe
+# setting up run directory
+currentdir=`pwd`
+mkdir -p DATABASES_MPI
+mkdir -p OUTPUT_FILES
+rm -rf DATABASES_MPI/*
+rm -rf OUTPUT_FILES/*
+# configure and compile in the source directory
+cd $rootdir
+# configure
+./configure -with-cuda=cuda5
+# compiles for a forward simulation
+cp $currentdir/DATA/Par_file DATA/Par_file
+make clean
+make all
+# backup of constants setup
+cp setup/* $currentdir/OUTPUT_FILES/
+cp DATA/Par_file $currentdir/OUTPUT_FILES/
+# Copy executables/Model in the current directory
+cd $currentdir
+# copy executables
+mkdir -p bin
+cp $rootdir/bin/xmeshfem3D ./bin/
+cp $rootdir/bin/xspecfem3D ./bin/
+# Links data necessary directories
+# The example below is for s362ani... this part should be changed if another model is used
+cd DATA/
+ln -s $rootdir/DATA/crust2.0
+ln -s $rootdir/DATA/s362ani
+ln -s $rootdir/DATA/QRFSI12
+ln -s $rootdir/DATA/topo_bathy
+cd ../
+</code>
 === Run with CPU ===
@@ Line 56: / Line 121: @@
 #SBATCH -t 23:00:00            # Wall time
+# Load modules
+module purge
+module load batch/slurm
+module load mpi/openmpi-basic
+#
 echo Master on host `hostname`
 echo Time is `date`
@@ Line 136: / Line 207: @@
 #SBATCH -t 12:00:00            # Wall time
 #SBATCH --cpu_bind=verbose
+# Load modules
+module purge
+module load batch/slurm
+module load compilers/cuda-7.5
+module load mpi/openmpi-basic
 # ID of each GPU (should be adapted if using a different number of GPUs)
@@ Line 204: / Line 281: @@
 echo CPUtime  : $(squeue -j $SLURM_JOBID -o "%M" -h)   # HH:MM:SS
 </code>
+==== Running simulations in parallel ====
+Some instructions to use custom scripts enabling parallel SEM simulations on the HPC cluster
+=== Preparing the input files ===
+First, create an event list "Events.txt" with 3 collumns:
+  * 1st column: event_id (will also be the name of the run directory)
+  * 2nd column: path to ''CMTSOLUTION'' file for this event
+  * 3nd column: path to ''STATION'' file for this event (can be the same for all events)
+Then you must setup a ''Par_file'' (be careful to use a version of ''Par_file'' that is compatible with your SEM version)
+Finally, you must setup hostfiles named ''nodelistN'' files where N=0,...,Np-1 (Np, the number of parallel SEM simulations). These files must specify host names and number of slots per node. Here is an example:
+<code>
+$ cat nodelist0
+hpc-n443 slots=8
+hpc-n444 slots=8
+hpc-n445 slots=8
+</code>
+(see ''/b/home/eost/zac/jobs/specfem/parallelSEM/nodelist0'')
+=== Running the simulations in parallel ===
+Parallel SEM simulations are handled using 3 scripts:
+  * ''parallelSEM.sh'': is the main script, that compiles the code and run the simulations
+  * ''run_gpu_nodelist.sh'' is the script used to run the mesher and solver
+  * ''sleep.slurm'' is a script to reserve the GPU nodes
+All these scripts are available in ''/b/home/eost/zac/jobs/specfem/parallelSEM''
+Before running your job, make sure that the input parameters in ''parallelSEM.sh'' are consistent with the input parameters stated above (see ''INPUT PARAMS'' in the main script). Specifically:
+  * ''SPECFEMDIR'': path to SPECFEM3D_GLOBE directory
+  * ''Par_file'': path to the Par_file used in simulations
+  * ''Nparallel'': Number of SEM simulations in parallel (make sure enough GPUs are available)
+  * ''event_list'': List of events with the format given above
+Then run your simulations:
+<code>
+./parallelSEM.sh
+</code>
+The script will make sure that the GPU nodes are available before launching SPECFEM3D_GLOBE.

Seismo Wiki

User Tools

Site Tools

Differences

Page Tools