HPC Workflow @ENCCB

15 Feb 2022, 08:00 → 16 Feb 2022, 17:00 Europe/Brussels

                                          

                                    Main event website | Videos  

Overview

Whenever one has to deal with multiple jobs on a HPC system, the idea of automating parts or all of the job management process involves describing and implementing so-called 'workflows'. Options for managing workflows are numerous and range from using basic scheduler features such as job arrays and job dependencies, up to using a complex system managed by a central, multi-user, database. Moreover, workflows tools are also available from the software development and deployment ecosystem and the whole "devops" movement. This online workshop aims at guiding participants towards the right tool for their use and help them reduce the time they spend managing their jobs by automating what can be automated and follow best practices.

After attending the workshop, you will

• know about generally available GNU/Linux tools to manage multiple tasks and jobs
• learn about some easy-to-install scientific workflow tools for single-user usage
• learn how to automate packaging and deploying software with containers or git-based web sites
• see how some research groups used around workflow management system tools

Prerequisites

• General knowledge of Linux and the command line
• Knowing how to install software as a regular user
• Knowing about Slurm and job submission and management

Agenda

Day1: February 15th 2022

• 9h-9h15: Introduction/Welcome (Raphaël Leplae)
• 9h15-9h30: Introduction to Workflow (Damien François)
• 9h30-11h + 30 min Q/A+break: Workflows with basic GNU tools (and Maestro) (Damien François)
• 11h30-12h30 + 15min Q/A: Checkpoint/Restart (Olivier Mattelaer)
• Lunch break
• 14h-15h + 30 min Q/A+break: Atools (Geert Jan Bex)
• 15h30-16h15 + 15 min Q/A+break: Makeflow (Damien François)

Day2: February 16th 2022

• 9h-10h + 30 min Q/A+break: Github (CI/CD)  (Olivier Mattelaer)
• 10h30-11h30 + 30 min Q/A+break: Singularity (Olivier Mattelaer)
• Lunch break:
• 14h-15h + 30 min Q/A+break: Snakemake (Orian Louant)
• 15h30-17h: Users testimonials
  • 15h30-15h50: NextFlow for bioinformatics by Luc Cornet
  • 15h50-16h10: Fireworks for material science by Guillaume Brunin
  • 16h10-16h30: Coral, a home made workflow system tool to manage numerical climate simulations by François Klein
  • 16h30-16h50: Examples of CI/CD in research codebases using git-based websites and HPC Denis-Gabriel Caprace

Tuesday, 15 February

1 Welcome & Introduction

To start the workshop, this talk will first present the ecosystem and environment related to HPC in Belgium and the contexts of PRACE and EuroCC. It will then move on to a short introduction on Workflows in HPC and a description of the different seminars to come in this workshop.

Speaker: Leplae Raphaël (ULB)

IntroductionToWorkflows.pdf

2 Workflows with basic GNU tools (and Maestro)

This presentation will present the two basic building blocks of workflows that are the job arrays and job dependencies. Job arrays allow creating parametrised jobs that all look identical except for one parameter that varies through the workflow, while job dependencies enable a fixed ordering of jobs and make sure the steps of the workflows are carried on only when their requirements (input data, software, output directory, etc.) are available. It will also discuss the concepts of micro-scheduling (running multiple small jobs steps inside of a single job allocation) and macro-scheduling (submitting multiple jobs at the same time with a single command). The presentation will also introduce the use of basic GNU/Linux commands that make micro- and macro-scheduling easier: xargs, seq, GNU Parallel, GNU Make, envsubst. The concepts will be illustrated with Slurm but should apply to any other scheduler. Finally, the session will present Maestro, a little workflow manager developed by the same lab as Slurm originated from, that focuses on documentation and organisation, and that makes it easy to build small workflows without the need to manually submit the jobs and is a nice complement to the Linux tools mentioned earlier.

Speaker: François Damien (UCLouvain)

WorkflowsWithBasicTools.pdf

3 Checkpoint/Restart

This session will discuss one specific type of workflows that is checkpoint/restart and how Linux signals can be leveraged to build self-resubmitting jobs that can run longer than the maximum wall time of the cluster.

Speaker: Mattelaer Olivier (UCLouvain)

22_02_checkpointing_PRACE.pdf

checkpointing_prace.tgz

checkpointing.tgz

4 atools

This presentation will present a collection of tools named atools that help building and managing large job arrays for parametrised studies. Such workflows can be referred to as "wide" workflows: many similar jobs siblings one to another, with no dependency among them.

Speaker: Bex Geert Jan (Uhasselt – KULeuven)

atools_managing_workflows_on_HPC.pptx

5 Makeflow

This session will discuss Makeflow, a tool that can be used to model workflows with many dependencies among jobs. Such workflows can be referred to as "deep" workflows by contrast with the "wide workflows" described earlier.

Speaker: François Damien (UCLouvain)

Makeflow.pdf

Wednesday, 16 February

6 CI/CD implementation with gitHub

This session will be about GitHub and its continuous integration/continuous deployment (CI/CD) features and how it can be used on clusters with a regular user to automatically compile software and even submit benchmark jobs whenever new features or improvements are added to the software you are writing.

Speaker: Mattelaer Olivier (UCLouvain)

7 Singularity

This presentation will be about Singularity and how to build containers and deploy them on clusters so as to install software in a uniform way, not being stopped by the Linux flavour or available software modules.

Speaker: Mattelaer Olivier (UCLouvain)

22_02_Singularity_Prace.pdf

8 Snakemake

We are back to scientific workflows and the seventh presentation will be a tutorial on SnakeMake, a tool that is a bit more complex to use than the other two but that can handle both wide and deep workflows, and can do more things like templating, containers, etc.

Speaker: Louant Orian (ULiège)

snakemake-enccb-22-02-16.pdf

9 User testimonials

15h30-15h50: NextFlow for bioinformatics by Luc Cornet
15h50-16h10: Fireworks for material science by Guillaume Brunin
16h10-16h30: Coral, a home made workflow system tool to manage numerical climate simulations by François Klein
16h30-16h50: Examples of CI/CD in research codebases using git-based websites and HPC by Denis-Gabriel Caprace

Brunin.pdf

Caprace.pdf

Cornet.pdf

Klein.pdf