Data on HPC: ( Storage/versioning/privacy and parralel IO: netCDF/HDF5)

Wednesday 10 Dec 2025, 09:30 → 16:00 Europe/Brussels

CYCL09b (Louvain-La-Neuve)

All HPC program will produce or consume data at a point or another. Where/how to write/store those data during and after your program is finished is important  in term of efficiency, in term of data preservation and in term of data privacy.
Contents: How to secure your data for maximal privacy. The netcdf/hdf5 file format (IO in parralel) How to version your data like you do for code?	Prerequisite: None Type: presentation, discussions and hands on Must: This session is a must have for researchers either handling large amount of data and/or concerned by the privacy of such data.

 

1 Netcdf and HDF5 file formats for HPC

Description:
This seminar provides a practical introduction to NetCDF and HDF5, two key structured data formats for scientific data management. Participants will learn to create, manipulate, and visualize these formats using Python, Fortran, and C, with a focus on their application in high-performance computing environments.

Contents:

• Overview of data formats in Sciences
• Comparing NetCDF vs HDF5
• Tools and libraries for working with NetCDF and HDF5
• Use cases in scientific research and HPC
• Hands-on exercises in creating, reading, and visualizing data

Prerequisite:

• Being able to connect to the clusters.
• Basic programming skills (Python, Fortran, or C).
• Familiarity with Linux (navigation, file creation and editing).

Target audience:
Researchers and data scientists using HPC systems.

Must for:
Essential for geoscientists, or anyone needing structured and efficient data format.

Speaker: Quentin Glaude (ULiège)

Netcdf and HDF5 file formats for HPC.pdf

2 Data Privacy

3 Versioning your data: Datalad

Speaker: Damien François (UCLouvain/CISM)