Overview
Whenever one has to deal with multiple jobs on a HPC system, the idea of automating parts or all of the job management process involves describing and implementing so-called 'workflows'. Options for managing workflows are numerous and range from using basic scheduler features such as job arrays and job dependencies, up to using a complex system managed by a central, multi-user, database. Moreover, workflows tools are also available from the software development and deployment ecosystem and the whole "devops" movement. This online workshop aims at guiding participants towards the right tool for their use and help them reduce the time they spend managing their jobs by automating what can be automated and follow best practices.
After attending the workshop, you will
- know about generally available GNU/Linux tools to manage multiple tasks and jobs
- learn about some easy-to-install scientific workflow tools for single-user usage
- learn how to automate packaging and deploying software with containers or git-based web sites
- see how some research groups used around workflow management system tools
Prerequisites
- General knowledge of Linux and the command line
- Knowing how to install software as a regular user
- Knowing about Slurm and job submission and management
Agenda
Day1: February 15th 2022
- 9h-9h15: Introduction/Welcome (Raphaël Leplae)
- 9h15-9h30: Introduction to Workflow (Damien François)
- 9h30-11h + 30 min Q/A+break: Workflows with basic GNU tools (and Maestro) (Damien François)
- 11h30-12h30 + 15min Q/A: Checkpoint/Restart (Olivier Mattelaer)
- Lunch break
- 14h-15h + 30 min Q/A+break: Atools (Geert Jan Bex)
- 15h30-16h15 + 15 min Q/A+break: Makeflow (Damien François)
Day2: February 16th 2022
- 9h-10h + 30 min Q/A+break: Github (CI/CD) (Olivier Mattelaer)
- 10h30-11h30 + 30 min Q/A+break: Singularity (Olivier Mattelaer)
- Lunch break:
- 14h-15h + 30 min Q/A+break: Snakemake (Orian Louant)
- 15h30-17h: Users testimonials
- 15h30-15h50: NextFlow for bioinformatics by Luc Cornet
- 15h50-16h10: Fireworks for material science by Guillaume Brunin
- 16h10-16h30: Coral, a home made workflow system tool to manage numerical climate simulations by François Klein
- 16h30-16h50: Examples of CI/CD in research codebases using git-based websites and HPC Denis-Gabriel Caprace
