IMEX SDC for SWE on the sphere
This tutorial describes how to run the the IMEX Spectral Deferred Correction (SDC) time-stepping method implemented in SWEET. For theoretical content on SDC, you can check this project (in construction) or any literature on the subject. This tutorial suppose that the reader is aware of the algebraic formulation of SDC using $Q$ and $Q_\Delta$ matrices, and just present how to set those coefficients for the IMEX SDC solver in SWEET.
If you are not familiar with the whole theory on SDC, you can always follow this tutorials using the provided default parameters …
1. Compilation and basic run
The up-to-date IMEX-SDC implementation is currently only implemented in the parallel_sdc
branch, that you can obtain like this :
$ git checkout parallel_sdc
Then before compiling with scons
, here is the list of libraries that should be installed locally from the local_software
folder :
$ cd local_software
$ ./install_miniconda.sh
$ ./install_scons.sh
$ ./install_numactl.sh
$ ./install_fftw3.sh
$ ./install_shtns.sh
$ ./install_shtns_python.sh
$ ./install_lapack.sh
To run the SWEET program with GUI (recommended for first tests), those installations are required :
$ cd local_software
$ ./install_sdl2.sh
$ ./install_libfreetype.sh
$ # For Ubuntu systems :
$ sudo apt install pkg-config libgl-dev libxext-dev
Then you can compile the SWE Sphere program with
$ cd .. # if you were before in local_software
$ make clean
$ scons --program=programs/PDE_SWESphere2D --gui=enable
Finally, you can run the simulation with default settings for IMEX SDC using the following command :
$ ./build/programs/PDE_SWESphere2D* --benchmark-name=galewsky -M 128 --dt=1200 --timestepping-method=ln_imex_sdc
This runs default IMEX-SDC, with Backward Euler (BE) for the linear terms and Forward Euler (FE) for the non-linear terms. In particular, it is stable with a time step $\Delta t=1200s$, which is four times larger that the maximum stable time step for ERK(4,4) with the same space discretization.
2. Changing SDC parameters
The simulation command shown in the previous section use IMEX SDC with the default settings implemented in SWEET, that is :
Current default settings for IMEX SDC are :
- 3 RADAU-RIGHT nodes
- 3 iterations (sweeps)
- initial sweep using COPY (of the initial time-step solution)
- no end update with the quadrature formula
- BE for implicit sweep
- FE for explicit sweep
You can change those parameters by providing a parameter file (SWEETFileDict format) with
the --sdc-file=[...]
program argument.
Parameter files can be generated using the mule.sdc.generateSetup
utility script.
For instance,
$ mule.sdc.generateSetup
generates a output_params_SDC.sweet
file containing all the defaults parameters of IMEX SDC described above.
Don’t hesitate to use the
mule.displaySWEETFileDict
script to display all parameters contained in the SDC parameters file, e.g :$ mule.displaySWEETFileDict output_params_SDC.sweet
You can change the SDC parameter to be used by providing arguments to mule.sdc.generateSetup
, for instance
$ mule.sdc.generateSetup --nIter=4 --nNodes=4 --nodeType=LOBATTO --qDeltaExplicit=PIC --useEndUpdate=ouiiii
change the default configuration to 4 sweeps, 4 LOBATTO nodes and a Picard iteration for the explicit sweep, adding after all sweep the end update with collocation.
For boolean type arguments (like --useEndUpdate
), you have to put an empty argument to set it to false, for instance
$ mule.sdc.generateSetup --useEndUpdate= --nIter=4 # ...
Don’t hesitate to use the -h
or --help
option of the mule.sdc.generateSetup
script.
In particular, you can find there some preset configuration for sdc, that you can use like this
$ mule.sdc.generateSetup --preset=P1
Here the P1
preset configuration use the following parameters :
P1 :
--nNodes 4
--nodeType RADAU-RIGHT
--nIter 3
--qDeltaImplicit BEPAR
--qDeltaExplicit PIC
--initialSweepType QDELTA
--useEndUpdate False
--diagQDeltaInit BEPAR
--diagonal True
You can see all the registered preset configurations using
$ mule.sdc.generateSetup --showPreset
3. Parallel SDC
Preliminary notes
This describe how to run diagonal SDC in parallel using OpenMP, combining eventually with space parallelization (also OpenMP).
Nested parallelization is not possible with gcc
, you must use a compiler allowing nested openMP parallelism. For instance, if llvm
compilers are installed, you can use a dedicated SWEET environment with :
$ source ./activate.sh default_llvm
Installation and parallel run
First, compile the programs/PDE_SWESphere2D
program (no GUI !)
$ make clean
$ scons --program=programs/PDE_SWESphere2D --parallel-sdc-par-model=omp
Then, generate the parameter file for diagonal SDC on 4 nodes :
$ mule.sdc.generateSetup --preset P1
And finally, program can be run on 100 time steps of size $\Delta{t}=300$ using :
OMP_NUM_THREADS=8 ./build/programs/PDE_SWESphere2D* --benchmark-name=galewsky -M 128 --dt=300 --timestepping-method=ln_imex_sdc --sdc-file=output_params_SDC.sweet -t $((300*100)) --num-threads-space=2
$\Rightarrow$ uses 8 processes in total, with 2 in space, and hopefully 4 in time