Advanced Scientific computing

Package Requirements¶

For this workshop, we’ll need a few packages. I create the list below.

Installing requirements¶

We recommend creating a new environment for your project. This will allow you to track dependencies a lot easier. There are several different ways to do this (virtualenv, pipenv).

You can copy the list of packages above into a file (e.g. requirements.txt) to install them with pip.

python3 -m pip install -r requirements.txt

Creating a virtual environment using venv (built-in Python 3.8+)¶

It’s the officially recommended way to create virtual environments since Python 3.5.

See Documentation > The Python Tutorial > 12. Virtual Environments and Packages for more information and a complete tutorial on venv.

The module used to create and manage virtual environments is called venv. venv will usually install the most recent version of Python that you have available. If you have multiple versions of Python on your system, you can select a specific Python version by running python3 or whichever version you want.

To create a virtual environment, decide upon a directory where you want to place it, and run the venv module as a script with the directory path:

python3 -m venv tutorial-env
source tutorial-env/bin/activate

python3 -m venv <dir> creates a directory (and sub-directories) if it doesn’t exist, inside which you have a copy of the Python interpreter and various supporting files. The last line activates the environment.

Using conda is also an option.

Outline of the workshop¶

• Researchers and scientific software developers write software daily. But only a few of us have gone through a specific training

• Good programming practices make a HUGE difference!

• We can learn a lot from open source software and open science practices.

📝 Requirements for scientific programming¶

• ⚙️ Main requirement: scientific code must be error free
• 🧑‍🔬 Scientist(s) time is the bottleneck, not computer time
  • code optimized to explore many different models (or statistical analyses) rather than a very fast single approach
• 💡 Reproducibility and re-usability, i.e., open science!
  • Every scientific result should be independently reproduced at least internally before publication
  • No need for somebody else to re-implement your algorithm
  • Increasing pressure for making the source code used in publications available online (especially for theoretical papers)

There are three interconnected primary aspects of advanced scientific computing:

1. Code optimization: the code should be error free and runs within contraints (memory limits, cpu limits, project deadline)
2. Scaling up & parallelizing: Overcoming some limitations by adding more CPUs/GPUs
3. Workflow design: complex project have successive tasks

Code optimization¶

I will not get into the details of this part. This could be a lengthy workshop on its own.

Unit tests in scientific computing¶

I will not cover unit testing in details but only briefly give some reminders.

The basics¶

• What to test, and how?
• At first testing seems cumbersome
  • It’s obvious that this code works
  • The test scripts are longer than the code
  • The test code is partially a duplicate of the real code
• What does a good test looks like?
• What should I test?
• Anything specific to scientific code?

A good test is divided in three parts:¶

1. Put your system in the right state for testing
   • Create objects, initialize parameters, define constants…
   • Define the expected result of the test
2. Execute the feature that you are testing
   • Typically one or two lines of code
3. Compare outcomes with the expected ones
   • Set of assertions regarding the new state of your system

Tips¶

• Test simple but general cases
  • Take a realistic scenario for your code; try to reduce it to a simple example
• Test special cases and boundary conditions
  • Code often breaks because of empty lists, None, NaN, 0.0, duplicated elements, non-existing file, and what not...

Numerical traps to avoid¶

• Use deterministic test cases when possible
  • set a seed for any randomness
• For most numerical algorithm, tests cover only oversimplified situations; sometimes it is impossible
• “fuzz testing”, i.e. generated random input is mostly used to stress-test error handling, memory leaks, safety, etc

Testing learning algorithm¶

• Learning algorithms can get stuck in local optima. The solution for general cases might not be known (e.g., unsupervised learning)
• Turn your validation cases into tests (with a fixed random seed)
• Stability tests:
  • Start from final solution; verify that the algorithm stays there
  • Start from solution and add noise to the parameters; verify that the algorithm converges back
• Generate mock data from the model with known parameters
  • E.g., linear regression: generate data as y = a*x + b + noise for random a, b, and x, and test

Good practice: Test-driven development (TDD)¶

• TDD = write your tests before your code
  • Choose what is the next feature you’d like to implement
  • Write a test for that feature
  • Write the simplest code that will make the test pass

def construct_stellar_binary_system(
    primary: Star, 
    other: Star, 
    separation_pc: float) -> BinaryStar:
    """ combine two stars into a binary system """
    pass

def test_construct_stellar_binary_system():
    """ testing binary creation"""
    age_myr = 100
    mass_msun = 1.2
    separation_pc = 5
    primary = Star(age=age_myr, mass=mass_msun)
    secondary = Star(age=age_myr, mass=mass_msun)
    binary = construct_stellar_binary_system(primary, secondary, separation_pc)
    assert(binary_center_of_mass = separation_pc / (1 + mass_msun/mass_msun)

• Pros

  • Forces you to think about the design of your code before writing it. How would you like to interact with it? Functionality oriented
  • The result is a code, with functions that can be tested individually.
  • If the results are bad, you will always write tests to find a bug. Would you if it works fine?

• Cons

  • The tests may be difficult to write, especially beyond the level of unit testing.
  • In the beginning, it may slow down development, but in the long run, it actually speeds up development.
  • The whole team needs to buy into Unit testing for it to work well.
  • can be tiresome, but it pays off big time in the end.
  • Early stage refactoring requires refactoring test classes as well.

Just remember, code quality is not just testing: “Trying to improve the quality of software by doing more testing is like trying to lose weight by weighing yourself more often” (Steve McConnell, Code Complete)

Debugging¶

• The best way to debug is to avoid bugs; In the TDD approach, you anticipate the bugs.

• Your test cases should already exclude a big portion of the possible causes

• Debugging: you can stop the execution of your application at the bug, look at the state of the variables, and execute the code step by step

• Avoid littering your code with print statements

pdb, the Python debugger¶

Official documentation

• Command-line based debugger

  • pdb opens an interactive shell, in which one can interact with the code

• examine and change value of variables

  • execute code line by line
  • set up breakpoints
  • examine calls stack

• Enter debugger at the start of a file:

python –m pdb script.py

• Enter at a specific point in the code (alternative to print):

# some code here
# the debugger starts here
import pdb;		pdb.set_trace()
# rest of the code

• Entering the debugger from ipython
  • %pdb – preventive
  • %debug – post-mortem

%xmode Verbose

Optimization & Profiling¶

• Python or <insert code name here > is slower than C, but not prohibitively so. (For python progress is happening right now!)
• In scientific applications, the difference is even more complicated as you often call C/Fortan libraries without knowing it (e.g., numpy, scipy, ...).
• “It is better to have a working code than being very fast to spill garbage” - MF

So never rush into writing optimizations

What is the strategy to optimize codes?¶

It is important to first say that optimizing means updating the code, not changing where it runs.

• Usually, a small percentage of your code takes up most of the time

1. Identify time-consuming parts of the code (use a profiler)
2. Only optimize those parts of the code
3. Keep running the tests to make sure that code is not broken

• Finally, stop optimizing as soon as possible

Preferred optimization order¶

Below, I give what in my opinion the priority of optimization strategies.

1. Vectorize code (e.g. using numpy)
2. Use a specific optimization toolkits (e.g., numexpr, numba, jax)
3. Check if compiler change could make sense (e.g., Intel’s MKL optimized libraries)
4. Use Cython to create compiled functions
5. Change approach
6. approximate calculations

(Again: we do not touch on parallelization of change of hardware here)

Python examples¶

Using timeit¶

• Precise timing of a function/expression
• Test different versions of a small amount of code, often used in interactive Python shell
• In ipython, you can use the %timeit magic command

Sadly, timeit does not have convenient calls for multiple tests at once.

The above example demonstrates that the join method is a bit more efficient than the others.

See more examples in the timeit documentation

Profiling code, cProfile¶

It’s since Python 2.5 that cProfile is a part of the Python package. It brings a nice set of profiling features to isolate bottlenecks in the code. You can tie it in many ways with your code. Like, wrap a function inside its run method to measure the performance. Or, run the whole script from the command line.

• standard Python module to profile an entire application (profile is an old, slow profiling module)
• Running the profiler from command line: python -m cProfile myscript.py
• Options

-o output_file
-s sort_mode (calls, cumulative,name, …)

A convenient way of visualizing results: RunSnakeRun

Tip: you can make your life easier sometimes Use decorators, context managers, ... without much details, an example below

turn this code

prof = cProfile.Profile()
retval = prof.runcall(myfunction, a, b, c, alpha=1)
prof.print_stats()

into this code

cProfile_decorator(myfunction)(na, b, c, alpha=1)

Example usage

How to interpret cProfile results?

1. <ncalls>: It is the number of calls made.
2. <tottime>: It is the aggregate time spent in the given function.
3. <percall>: Represents the quotient of <tottime> divided by <ncalls>.
4. <cumtime>: The cumulative time in executing functions and its subfunctions.
5. <percall>: Signifies the quotient of <cumtime> divided by primitive calls.
6. <filename_lineno(function)>: Point of action in a program. It could be a line no. or a function at some place in a file.

Now, you have all elements of profiling report under check. So you can go on hunting the possible sections of your program creating bottlenecks in code.

First of all, start checking the <tottime> and <cumtime> which matters the most. The <ncalls> could also be relevant at times. For rest of the items, you need to practice it yourself.

Optimizing loops¶

Most programming language stress upon the need to optimize loops.

In Python, you’ll see a couple of building blocks that support looping. Out of these few, the use of “for” loop is prevalent. While you might be fond of using loops but they come at a cost. Python loops are slow for instance. Although not as slow as other languages, the Python engine spends substantial efforts in interpreting the for loop construction. Hence, it’s always preferable to replace them with built-in constructs, such as maps, or generators.

Next, the level of code optimization also depends on your knowledge of Python built-in features.

The above examples are showing that using built-in generators could speed up your code instead of using some for-loops

You can also use external libraries (e.g. numpy)

Check out also our FAQ: e.g. NumbaFun

Python vs. Numpy vs. Cython¶

NumPy is approximately 2000 times faster than naive Python implementation of dot product.

Although, to be fair, NumPy uses an optimized BLAS library when possible, i.e., it calls fortran routines to achieve such performance.

Below the -a option indicates annotate, which let us look at the C code generated by cython. Note that annotations may also slow down the execution as it generates some additional code sometimes.

The c_dot_raw function is as slow as the pure python implementation, but as soon as we help the compiler with typing the variables, c_dot becomes 400 times faster, and still 10 times faster than numpy.

Why is that faster than numpy?? (think overhead and checks)

Probably unintuitively, removing the GIL does not give us a speed up. But note that we did not set any parallelization rule, so python does not further optimize the code.

The last example is the perfect queue to the next topic: parallelization!

Scaling up & parallelization¶

References:

• Introduction to High-Performance Computing
• Pratical Datascience

Research problems these days can outgrow the capabilities of the desktop/laptop computer.

• If you want to cross-validate a model, you need to run multiple times a training/testing workflow (e.g. K-fold or leave-one-out) -- each could take hours. Since each of the runs is independent of all others, and given enough computers, it’s theoretically possible to run them all at once (in parallel).

• When the dataset does not fit into memory. Take for instance Gaia data, it could easily fill your memory before any manipulation. The calculations required might be impossible to parallelize, but a computer with more memory would be required. Imagine analyzing the Gaia spectra for 1.2 Bn sources, it would be foolish to load it all in memory, but each source analysis can be independent from each other. This is call a massively parallel problem.

• More complex calultations moving away from “toy” examples. For instance going from 2D-LTE stellar atmospheres to 3D-NLTE simulations. Or large N-Body simulations (e.g. TNG). In these research problems, the calculations in each sub-region of the simulation are largely independent of other regions of the simulation (apart from these edges). It’s possible to run each region’s calculations simultaneously (in parallel), and communicate selected results to adjacent regions as needed. both the amount of data and the amount of calculations increases greatly, and it’s theoretically possible to distribute the calculations across multiple computers communicating over a shared network.

In all these cases, having bigger computers would help.

What is Parallelism?¶

Parallelism is the process of:

1. taking a single problem,
2. breaking it into lots of smaller problems,
3. assigning those smaller problems to a number of processing cores that are able to operate independently, and
4. recombining the results.

Parallelism is not easy, and so not only does it take substantial developer time (the time it takes you to implement it), but there are computer-time costs to breaking down problems, distributing them, and recombining them, often limiting the returns you will see to parallelism.

Limitations of parallelisation: Amdahl’s law¶

Amdahl’s Law describes the theoretical limits of parallelization. If $P$ is the proportion of your algorithm that can be parallelized, and $N$ is the number of cores available, the fundamental limit to the speed-up you can get from parallelization is given by:

This analysis neglects other potential bottlenecks such as memory bandwidth and I/O bandwidth. If these resources do not scale with the number of processors, then merely adding processors provides even lower returns.

Strategies: what to parallelize?¶

It is important to identify the bottleneck of your analysis, but also which steps can run independently from one-another.

A general rule of thumb: you’ll achieve a larger parallel fraction if you distributed the most outer loops.

Backends: Multi-Threading vs. Multi-Processing¶

There are fundamentally only 2 approaches to parallelize codes.

• Multi-Threading: running tasks on a set of cores sharing information by accessing the same memory (multi-core computations)
• Multi-Processing: running tasks across multiple processings which have to communicate data to share information. E.g. multiple physical units CPUs or computers.

Confusingly, though, the way hyperthreading manifests is by telling your operating system that you have two cores for every physical core on your computer.

So, how should you think about this?

• In most contexts (if you’re not shoping for hardware), expect the terms “cores” and “processors” to both be used to refer to independent processing cores.

• Don’t worry about whether they’re distributed across two different physical chips or not – all that matters is the number of cores on your computer.

• On most computers, expect the operating system to think that you have twice as many cores (sometimes referred to as “logical cores”") as you have actual “physical cores”, but expect your performance to reflect the number of physical cores you actually have.

A first example with joblib¶

First, let’s write a little weather temperature simulation. We assume that from day to day we cannot have arbitrarily large variations, but that the temperature at $t+1$ is randomly distribution around the temperature at $t$ (with some spread) -- if it’s hot today, it’s likely to be hot tomorrow.

So let’s try parallelizing these simulations using joblib.

So this shows there’s a lot of fixed costs to getting this independent processes running and passing things back and forth, as a result of which the benefits of parallelism are almost always sub-linear (i.e. using 10 processors instead of 1 will result in <10x speedups).

Multi-processing versus Multi-threading¶

Multiprocessing is nice because (a) you can use it on big server clusters where different processes may be running on different computers, and (b) it’s quite safe. As a result, most parallelism you’re like to encounter will be multi-processing.

In multi-threading, all the code being run (each sequence execution of which is called a “thread”) exists within a single process, meaning that all the different threads have access to the same objects in memory. This massively reduces duplication because you don’t have to copy your code and data and pass it around – all the threads can see the same parts of memory.

But multi-threading has three major shortcomings:

• It is very easy to run into very subtle but profound problems that lead to corruption (the biggest of which is something called Race Conditions).

• Multi-threading can only distribute a job over the cores on a single computer, meaning it can’t be used to distribute a job over hundreds of cores in a large computing cluster.

• You generally can’t use multi-thread parallelism in Python because of a fundamental component of its architecture (the GIL).

GPU Parallelism¶

GPU parallelism is the practice of using Graphical Processing Units (GPUs) to do extremely parallel computing.

GPUs are basically designed for the sole purpose of processing graphics, i.e, lots of matrix algebra. And so scientists initially hacked their way to using their capacities, and we now even have double precision GPU units.

GPU processors aren’t “general purpose”, they only do tensor operations, and as a result they do it super fast. They are massively parallel: while your CPU (your Intel chip) may have ~4-8 cores, a modern GPU has either hundreds or thousands of cores.

But because GPUs are basically computers onto themselves, to use them you have to write special code to both manage all those cores and also manage the movement of data from your regular computer to the GPU. However, you probably will never write your own GPU parallel algorithm, but if you end up in an area that uses them a lot (i.e. training neural networks), the libraries you use will (e.g. Tensorflow, pyTorch, Jax, Numpyro, ...).

Illustration of GPU parallel processing¶

Parallelism and Distributed Computing¶

We’ll discuss some packages that are designed to make parallelism as easy as possible in situations where you really need parallelism. In particular, we’ll focus on tools for distributed computer: situations where you don’t just want to parallelize across the cores in your computer, but across many computers (e.g. in the cloud). But when we get to these tools – like pyspark and dask – try to keep in mind the lessons we’ve learned here, because everything you’ve learned here applies to those packages as well!

Most of the times when you are using distributed computing, you will be using a scheduler and a pool of workers. Tasks are submitted to the scheduler and the latter takes care of scattering to the workers and gathering the results.

Below an illustration from Dask.org.

Example of calulating $\pi$¶

Let’s try running a Python code that makes use of parallel computing.

The following implements a classic example in computational science: estimating the numerical value of pi via Monte Carlo sampling. Imagine you’re throwing darts, and you’re not very accurate. You are trying to hit a spot somewhere within a circular target, but you can’t manage to do much better than throw it somewhere within the square that bounds the target, hitting any point within the square with equal probability. The red dots are those darts that manage to hit the circle, and the blue dots are those darts that don’t.

With python multiprocessing¶

with Dask¶

Direct adaptation: map/reduce¶

Dask magic: Delayed and graph calculations¶

We see that Dask built a graph of operations, for each CPU unit, call the monte carlo function and gather the results for the sum value.

Again, the cost differs, and one may approach the problem one way or the other depending on the constraints.

MPIA, MPCDF, and other places¶

Being at MPIA, we have access to varied computation facilities.

1. MPIA (in-house): hosted, managed, and funded by MPIA.
   • Only MPIA employees (after requesting account) can use the clusters
2. MPCDF (garching): hosted, managed by MPCDF; funded by MPG (and in part by MPIA)
   • All MPIs employees (after requesting MPCDF account) can use some clusters
   • Access to some machines may be restricted by who funded those.
3. Germany/EU HPC facilities: Hosted and managed by various supercomputing centers
   • Anyone can apply through computation proposals -- Competitive access.

Different machines/configurations are suitable differently for different tasks. At MPIA, we have a diversity of needs and resources.

MPIA¶

MPIA has 9 clusters of non-homogeneous platforms, funded by and accessible to individual groups. This represents a total of ~ 5 400 CPUs.

Two main categories of clusters

• Interactive Data Processing: e.g., ASTRO nodes -- basically a pool of “desktop” computers.
• Batch/non-interactive Computing Jobs: e.g. BACHELOR_NEW -- submit jobs through a scheduler.

Status of the clusters: ganglia (internal/vpn accessible)

Pros/Cons for the users

• [+] directly accessible, no (or virtually-no) queues and waiting times
• [+] very low utilization: ~30% over the year
• [-] individually accessible only to selected group members
• [-] non-homogeneous platforms = harder to optimize codes

Things will start to change at MPIA. Systems will slowly merge into a gigantic (single) cluster. This will help both the users and the IT maintenance load.

MPCDF (The Max Planck Computing and Data Facility)¶

The MPCDF (formerly RZG) is a cross-institutional center of MPG to support computational and data sciences.

Any MPG employee can get an account (via online forms and approval from members of computing committee).

List of clusters

Supercomputers¶

• RAVEN (2021)

  • 1592 nodes
  • 114,624 Intel Xeon IceLake-SP CPU-cores, 375 TB RAM (DDR4),
  • 768 Nvidia A100 GPUs (192 nodes), 30 TB GPU RAM (HBM2)

• COBRA (2018)

  • 3424 nodes
  • 136,960 Intel Xeon Skylake-SP CPU-cores, 529 TB CPU RAM (DDR4),
  • 128 Tesla V100-32 GPUs, 240 Quadro RTX 5000 GPUs, 7.9 TB GPU RAM HBM2

• Some facts about Raven & Cobra

  • Hardware renewed every ~5-6 years
  • Idle ~5% on average per year (MPIA > 70%)
  • About 150M Core-hours per month
    • across all 25 MPIs as main users
    • MPIA’s share ~4 to 8% per month

MPIA dedicated cluster¶

• VERA (2022)

  • login nodes vera[01-02] (500 GB RAM each)
  • 72 execution nodes vera[001-072] (250 GB RAM each)
  • 36 execution nodes vera[101-136] (500 GB RAM each)
  • 2 execution nodes vera[201-202] (2 TB RAM each)
  • 3 execution nodes verag[001-003] (500 GB RAM and 4 Nvidia A100-40GB GPUs each)
  • node interconnect is based on Mellanox/Nvidia Infiniband HDR-100 technology (Speed: 100 Gb/s)
  • Largest possible single batch job: 1680 parallel tasks (48hrs max)
  • 400 TB filesystem (for the GC and PSF, separately, with quotas)

• Some facts:

  • Hardware renewed every ~5-6 years
  • Idle ~10-15% on average per year (MPIA > 70%)
  • About 2.3M Core-hours per month
  • 15-20 main users (average per month)

What about cloud Computing?¶

Cloud computing is the on-demand availability of computer system resources. Large clouds often have functions distributed over multiple locations, each location being a data center.

The cloud is a generic term commonly used to refer to remote computing resources of any kind -- that is, any computers that you use but are not right in front of you. Cloud can refer to machines serving websites, providing shared storage, providing webservices (such as e-mail or social media platforms), as well as more traditional “compute” resources.

Cloud computing is like a virtual cluster, capable of instanciating nodes on-demand to provide services and run tasks.

Turns-out, MPCDF has a cloud computing service that we are exploring as beta testers at the moment.

Working on a remote HPC system¶

What is an HPC system?¶

An HPC system is a term used to describe a network of computers dedicated to “high performance computing”. The computers in a cluster typically share a common purpose, and are used to accomplish tasks that might otherwise be too big for any one computer.

MPCDF Logging in¶

The gateway machines gatezero.mpcdf.mpg.de and gateafs.mpcdf.mpg.de provide ssh access to MPCDF computing resources. One should note that gatezero has no access to AFS, but the home directory $HOME is local to that machine and very limited in size. Note that all MPCDF gateway machines enforce 2 factor authentication (2FA).

See details on the gateways online documentation.

Go ahead and log in to the MPCDF gateway:

> ssh -XY <your_username>@gate.mpcdf.mpg.de

This should prompt you for your 2FA code as well.

The options -XY ensure that any Unix graphical output (e.g. if you run a GUI, look at an image or plot a graph) are sent back to your own laptop. You may need to specify these options explicitly.

From there you need to reach final destination, the cluster of your choice: e.g. raven:

> ssh raven
[...]
fmorg@raven01:~>

Tip: sshconfig¶

You can set your computer to directly connect you to the machines by add some lines to your ~/.ssh/config file.

Example:

Host raven
        User <your_username>
        Hostname raven.mpcdf.mpg.de
        ProxyCommand ssh -W %h:%p fmorg@gate.mpcdf.mpg.de 

Host cobra
        User <your_username>
        Hostname cobra.mpcdf.mpg.de
        ProxyCommand ssh -W %h:%p fmorg@gate.mpcdf.mpg.de 

Host vera01
        User <your_username>
        Hostname vera01.bc.rzg.mpg.de
        ProxyCommand ssh -W %h:%p fmorg@gate.mpcdf.mpg.de 

• sshkeys
• ssh remember passwords time.

Let’s have a look at the computing documentation

Cheatsheet¶

HPC clusters at MPCDF use Slurm job scheduler for batch job management and execution.

MPCDF scheduler cheatsheet

Software environment (modules)¶

The MPCDF uses module system to adapt the user environment for working with software installed at various locations in the file system or for switching between different software versions. Users need to explicitly specify the full version for compiler and MPI modules during compilation and in batch scripts to ensure compatibility of the MPI library.

Due to the hierarchical module environment, many libraries and applications only appear after loading a compiler, and subsequently also a MPI module (the order is important here: first, load compiler, then load the MPI). These modules provide libraries and software consistently compiled with/for the user-selected combination of compiler and MPI library.

To search the full hierarchy, the find-module command can be used. All fftw-mpi modules, e.g., can be found using find-module fftw-mpi.

• module avail to list the available software packages on the HPC system. Note that you can search for a certain module by using the find-module tool (see below).
• module load package_name/version to actually load a software package at a specific version.

Information on the software packages provided by the MPCDF is available here.

The module library is also on the MPIA clusters (but the list of libraries differs from MPCDF)

SLURM: Scheduler / batch system¶

The batch system on the MPCDF & MPIA clusters is the open-source workload manager Slurm (Simple Linux Utility for Resource management). To run test or production jobs, submit a job script (see below) to Slurm, which will find and allocate the resources required for your job (e.g. the compute nodes to run your job on).

By default, their is a limited number of jobs you can run at once. For instance, it is set to 8 on Raven, the default job submit limit is 300. If your batch jobs can’t run independently from each other, you need to use job steps.

There are mainly two types of batch jobs:

• Exclusive, where all resources on the nodes are allocated to the job

• Shared, where several jobs share the resources of one node. In this case it is necessary that the number of CPUs and the amount of memory are specified for each job.

MPCDF introduction

Important key points¶

• Job script structure: scripts are relatively always the same: written in bash with a sweep of #SBATCH option declarations, then loading necessary modules and libraries, and finally what the job commands are.

• Bash shebang option -l (#!/bin/bash -l): The -l option makes “bash act as if it had been invoked as a login shell”. Login shells read certain initialization files from your home directory, such as . bash_profile . Since you set the value of TEST in your .

• Wall clock limit (--time): Always set how long your code will run instead of using the largest value. This not only makes sure you have an idea, but also can be used to get your job up the queue!

• srun for parallel processing: as soon as your code will use multiple cores, run the code with srun so the system uses optimally the resources.

Examples¶

• MPI batch job without hyperthreading

#!/bin/bash -l
# Standard output and error:
#SBATCH -o ./job.out.%j
#SBATCH -e ./job.err.%j
# Initial working directory:
#SBATCH -D ./
# Job Name:
#SBATCH -J test_slurm
#
# Number of nodes and MPI tasks per node:
#SBATCH --nodes=16
#SBATCH --ntasks-per-node=72
#
#SBATCH --mail-type=none
#SBATCH --mail-user=userid@example.mpg.de
#
# Wall clock limit (max. is 24 hours):
#SBATCH --time=12:00:00

# Load compiler and MPI modules (must be the same as used for compiling the code)
module purge
module load intel/21.2.0 impi/2021.2

# Run the program:
srun ./myprog > prog.out

• Using SLURM variables to set openMP

#!/bin/bash -l
# Standard output and error:
#SBATCH -o ./job_hybrid.out.%j
#SBATCH -e ./job_hybrid.err.%j
# Initial working directory:
#SBATCH -D ./
# Job Name:
#SBATCH -J test_slurm
#
# Number of nodes and MPI tasks per node:
#SBATCH --nodes=16
#SBATCH --ntasks-per-node=4
# for OpenMP:
#SBATCH --cpus-per-task=18
#
#SBATCH --mail-type=none
#SBATCH --mail-user=userid@example.mpg.de
#
# Wall clock limit (max. is 24 hours):
#SBATCH --time=12:00:00

# Load compiler and MPI modules (must be the same as used for compiling the code)
module purge
module load intel/21.2.0 impi/2021.2

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# For pinning threads correctly:
export OMP_PLACES=cores

# Run the program:
srun ./myprog > prog.out

• an example from my work

#!/bin/bash -l
#
# This script submit to a SLURM queue
#
# Example Usage:
#
# 	> sbatch --array=1-10:1 slurm.script.array
#
# This should submit (in principle) <CMD> 1 then <CMD> 2 ...
#
#------------------- SLURM --------------------
#
#SBATCH --job-name=bob_template
#SBATCH --chdir=./
#SBATCH --export=ALL
# output: {jobname}_{jobid}_{arraytask}.stdout
#SBATCH --output=logs/%x_%A_%a.stdout
#SBATCH --partition=p.48h.share
#SBATCH -t 48:0:0
#SBATCH --get-user-env
#SBATCH --exclusive=user
#SBATCH --mem=1G
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mail-user do-not-exist-user@mpia.de
#SBATCH --mail-type=ALL
#
# -------------- Configuration ---------------

module purge
module load anaconda/3_2019.10
module load git gcc/7.2 hdf5-serial/gcc/1.8.18 mkl/2019

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKL_HOME/lib/intel64/:${HDF5_HOME}/lib

# Set the number of cores available per processif the $SLURM_CPUS_PER_TASK is
# set
if [ ! -z $SLURM_CPUS_PER_TASK ] ; then
	export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
else
	export OMP_NUM_THREADS=1
fi

# -------------- Commands ---------------

# Checking python distribution
function python_info(){
        printf "\033[1;32m * Python Distribution:\033[0m "
	python -c "import sys; print('Python %s' % sys.version.split('\n')[0])"
	}
# python_info

# get the configuration slice
n_cpus=${SLURM_CPUS_PER_TASK}
n_per_process=1000
slice_index=$((${SLURM_ARRAY_TASK_ID} -1))
processing_start=$((${slice_index} * ${n_per_process}))
processing_end=$((${slice_index} * ${n_per_process} + ${n_per_process}))

CMD="./main_mars"
RUN="testsample"
# RUN="orionsample"
MOD="emulators/oldgrid_with_r0_w_adjacent.emulator.hdf5"
INF="data/testsample.vaex.hdf5"
# INF="data/orionsample.vaex.hdf5"
OUT="./results/${RUN}"
min=$((processing_start))
max=$((processing_end + 1))
NBURN=4000
NKEEP=500
NWALKERS=40
SAMPLES="samples/${RUN}"

CMDLINE="${CMD} -r ${RUN} -m ${MOD} -i ${INF} -o ${OUT} --from ${min} --to ${max} --nburn ${NBURN} --nkeep ${NKEEP} --nwalkers ${NWALKERS}"
echo ${CMDLINE}
${CMDLINE}

Codes will often run longer than the queues allow them. For instance, an N-Body simulation like TNG or Aquarius, runs over millions of CPU hours for many months. It is critical that you have checkpoints in your codes, i.e. ways to resume from a mid-point, not restart from scratch. (see workflow design)

Workflow design¶

A workflow defines the phases (or steps) in a science project. Using a well-defined workflow is useful in that it provides a simple way to remind all team members (which includes a scheduler for instance) of the work to be done.

One way to think about the benefit of having a well-defined data workflow is that itsets guardrails to help you plan, organize, and implement your runs.

You have been using workflows before:

echo "number of physical cpus"  $(cat /proc/cpuinfo | grep "physical id" | uniq | wc -l)
echo "number of logical cores"  $(cat /proc/cpuinfo | grep "processor" | uniq | wc -l)

You could rapidly see which parts could be parallelized. In addition, one could imagine analyzing a lot of independent chunks of data.

But you do not want to restart any of this from scratch.

What about having a flow that would be something like

download data -> checkpoint -> Analyze chunk = {C1 -> checkpoint, C2 -> checkpoint} -> post processing

where your code would know automatically that if the result of C1 did not change, there is not need to redo.

Have you noticed the resemblance with Dask’s graph?

Make¶

Gnu Make is a / “the” tool of reference which controls the generation of executables and other non-source files of a program from the program’s source files.

Make gets its knowledge of how to build your program from a file called the makefile, which lists each of the non-source files and how to compute it from other files. When you write a program, you should write a makefile for it, so that it is possible to use Make to build and install the program.

Of course, even if initially meant to construct executables, we (users) have (trivially) extended the use-cases to generating anything.

Snakemake (“the pythonic make”)¶

Snakemake is workflow management system, i.e. it is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment.

Created in