User Tools

Site Tools


software:specfem

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
software:specfem [2017/03/29 11:47]
wphase
software:specfem [2018/03/05 16:05]
wphase
Line 1: Line 1:
 ====== SPECFEM3D_GLOBE ====== ====== SPECFEM3D_GLOBE ======
  
-==== Running SPECFEM3D_GLOBE on the Strasbourg HPC cluster with gnu 4.8 (from the system) ​and cuda 7.5 ====+ 
 +==== Running SPECFEM3D_GLOBE on the Strasbourg HPC cluster with gnu 4.8 and cuda 7.5 ====
  
 === Setup the environment === === Setup the environment ===
Line 9: Line 10:
 module load batch/slurm module load batch/slurm
 module load compilers/​cuda-7.5 module load compilers/​cuda-7.5
-module load mpi/openmpi-basic+export CUDA_INC=/usr/​local/​cuda/​cuda-7.5/include
 export CUDA_LIB=/​usr/​local/​cuda/​cuda-7.5/​lib64 export CUDA_LIB=/​usr/​local/​cuda/​cuda-7.5/​lib64
-export ​CUDA_INC=/usr/local/cuda/cuda-7.5/include+export ​PATH=/rpriv/ipgs/zac/openmpi-1.10.7/​bin:​$PATH 
 +export LD_LIBRARY_PATH=/​rpriv/​ipgs/​zac/​openmpi-1.10.7/lib:​$LD_LIBRARY_PATH
 </​code>​ </​code>​
-Notice that we use gnu 4.8 compiler ​directly from the system.  +Notice that we use default ​gnu compiler ​of the operating ​system: ​
- +
-=== Compilation === +
-Before compilation,​ make sure that you are logged in hpc-n523 (the frontal node is not yet migrated on the new OS):+
 <​code>​ <​code>​
-ssh hpc-n523+$ gfortran ​--version 
 +GNU Fortran (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
 </​code>​ </​code>​
  
-First make sure that required modules are loaded and CUDA_LIB, CUDA_INC environment variables are declared (see previous section). Create a run directory including directories ''​DATABASE_MPI'',​ ''​OUTPUT_FILES'',​ ''​bin''​ and ''​DATA''​.+=== Compilation === 
 +Before compilation, ​make sure that required modules are loaded and CUDA_LIB, CUDA_INC environment variables are declared (see previous section). Create a run directory including directories ''​DATABASE_MPI'',​ ''​OUTPUT_FILES'',​ ''​bin''​ and ''​DATA''​.
 In the directory ''​DATA'',​ create ''​CMTSOLUTION'',​ ''​Par_file''​ and ''​STATIONS''​ file (cf., SPECFEM3D_GLOBE documentation). In the directory ''​DATA'',​ create ''​CMTSOLUTION'',​ ''​Par_file''​ and ''​STATIONS''​ file (cf., SPECFEM3D_GLOBE documentation).
  
Line 55: Line 56:
 <​code>​ <​code>​
 #!/bin/bash #!/bin/bash
 +
 +# Load modules ​
 +module purge
 +module load batch/slurm
 +module load compilers/​cuda-7.5
 +module load mpi/​openmpi-basic
 +export CUDA_LIB=/​usr/​local/​cuda/​cuda-7.5/​lib64
 +export CUDA_INC=/​usr/​local/​cuda/​cuda-7.5/​include
  
 # source directory # source directory
Line 72: Line 81:
  
 # configure # configure
-./configure -with-gpu=cuda5+./configure -with-cuda=cuda5
  
 # compiles for a forward simulation # compiles for a forward simulation
Line 112: Line 121:
 #SBATCH -t 23:​00:​00 ​           # Wall time #SBATCH -t 23:​00:​00 ​           # Wall time
  
 +# Load modules ​
 +module purge
 +module load batch/slurm
 +module load mpi/​openmpi-basic
 +
 +
 echo Master on host `hostname` echo Master on host `hostname`
 echo Time is `date` echo Time is `date`
Line 192: Line 207:
 #SBATCH -t 12:​00:​00 ​           # Wall time #SBATCH -t 12:​00:​00 ​           # Wall time
 #SBATCH --cpu_bind=verbose ​ #SBATCH --cpu_bind=verbose ​
 +
 +# Load modules ​
 +module purge
 +module load batch/slurm
 +module load compilers/​cuda-7.5
 +module load mpi/​openmpi-basic
  
 # ID of each GPU (should be adapted if using a different number of GPUs) # ID of each GPU (should be adapted if using a different number of GPUs)
Line 260: Line 281:
 echo CPUtime ​ : $(squeue -j $SLURM_JOBID -o "​%M"​ -h)   # HH:MM:SS echo CPUtime ​ : $(squeue -j $SLURM_JOBID -o "​%M"​ -h)   # HH:MM:SS
 </​code>​ </​code>​
 +
 +
 +
 +==== Running simulations in parallel ====
 +
 +Some instructions to use custom scripts enabling parallel SEM simulations on the HPC cluster
 +
 +=== Preparing the input files ===
 +
 +
 +First, create an event list "​Events.txt"​ with 3 collumns:
 +  * 1st column: event_id (will also be the name of the run directory)
 +  * 2nd column: path to ''​CMTSOLUTION''​ file for this event
 +  * 3nd column: path to ''​STATION''​ file for this event (can be the same for all events)
 +
 +Then you must setup a ''​Par_file''​ (be careful to use a version of ''​Par_file''​ that is compatible with your SEM version)
 +
 +Finally, you must setup hostfiles named ''​nodelistN''​ files where N=0,​...,​Np-1 (Np, the number of parallel SEM simulations). These files must specify host names and number of slots per node. Here is an example:
 +<​code>​
 +$ cat nodelist0 ​
 +hpc-n443 slots=8
 +hpc-n444 slots=8
 +hpc-n445 slots=8
 +</​code>​
 +(see ''/​b/​home/​eost/​zac/​jobs/​specfem/​parallelSEM/​nodelist0''​)
 +
 +=== Running the simulations in parallel ===
 +
 +Parallel SEM simulations are handled using 3 scripts:
 +  * ''​parallelSEM.sh'':​ is the main script, that compiles the code and run the simulations
 +  * ''​run_gpu_nodelist.sh''​ is the script used to run the mesher and solver
 +  * ''​sleep.slurm''​ is a script to reserve the GPU nodes
 +All these scripts are available in ''/​b/​home/​eost/​zac/​jobs/​specfem/​parallelSEM''​
 +
 +Before running your job, make sure that the input parameters in ''​parallelSEM.sh''​ are consistent with the input parameters stated above (see ''​INPUT PARAMS''​ in the main script). Specifically:​
 +  * ''​SPECFEMDIR'':​ path to SPECFEM3D_GLOBE directory
 +  * ''​Par_file'':​ path to the Par_file used in simulations
 +  * ''​Nparallel'':​ Number of SEM simulations in parallel (make sure enough GPUs are available)
 +  * ''​event_list'':​ List of events with the format given above
 + 
 +Then run your simulations:​
 +<​code>​
 +./​parallelSEM.sh
 +</​code>​
 +The script will make sure that the GPU nodes are available before launching SPECFEM3D_GLOBE.
 +
 +
 +
 +
software/specfem.txt · Last modified: 2018/03/05 16:06 by wphase