Learning how to use HPC infrastructure (part III: Submitting jobs, using containers, checkpointing, workflows)

Workflow management systems

19 Oct 2023, 14:00

1h

Maxwell/Shannon (first floor) (Louvain-La-Neuve)

Speaker

Damien François (UCLouvain/CISM)

Description

Whenever one has to deal with multiple jobs on a HPC system, the idea of automating parts or all of the job management process involves describing and implementing so-called 'workflows'. Options for managing workflows are numerous and range from using basic scheduler features such as job arrays and job dependencies, up to using a complex system managed by a central, multi-user, database. This session aims at guiding participants towards the right tool for their use and help them reduce the time they spend managing their jobs by automating what can be automated and follow best practices.

 

Contents: Introduction to workflows  Types of workflow management systems  Choosing a workflow management system  An example with Maestro	Prerequisite: Being able to use SSH with private keys  Being familiar with a text editor  Mastering the Linux command line and the GNU utilities Working knowledge of Slurm
Type: Hands-on Target audience: Everyone Must: This session is useful.

Presentation materials

Workflows-2023-1.pdf

Workflows-2023-2.pdf