Data Management (Data storage and transfer, filesystems)

Wednesday 27 Nov 2024, 09:30 → 15:00 Europe/Brussels

CYCL E349 (Louvain-La-Neuve)

All HPC program will produce or consume data at a point or another. Where to write/store those data during and after your program is finished is important both in term of efficiency and in term of data preservation (which is important due to the growing relevance of Open Science)
Contents: Introduction to data storage and access Efficient data storage on CECI clusters Open Science and Open Research Data / Data Management Plan	Prerequisite: None Type: presentation, discussions and hands on Must: This session is a must have for researchers concerned by the dissemination of research results and by their impact.

1 Introduction to data storage and access

Storing data in an efficient way is very important for many scientific applications. Yet, most of the time, a myriad of small files is used, imposing a large burdun on the file system, spending a lot of time in file access, making transfers very inefficients, etc. Other solutions exist and are presented in this session.
Contents: Storing in files vs in database Using an in-memory database Using HDF5 CLI tools and libraries	Prerequisite: Being able to use SSH with private keys  Being familiar with a text editor  Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.) Passive knowledge of either C, Fortran, Octave, Python or R Working knowledge of C or Fortran Familiarity with OpenMP and MPI
Type: Hands-on Target audience: Everyone Must: This session is a must-have for anyone who thinks generating a million small files is an optimal way of storing data.

Speaker: Damien François (UCLouvain/CISM)

data.pdf

2 Efficient data storage on CECI clusters

Speaker: Ariel Lozano (ULB)