Introduction to data storage and access

by Damien François (UCLouvain/CISM)

Wednesday 17 Nov 2021, 09:00 → 11:00 Europe/Brussels

bibliotheque des sciences: salle pasteur (comodal (louvain-la-neuve or remote))

Storing data in an efficient way is very important for many scientific applications. Yet, most of the time, a myriad of small files is used, imposing a large burdun on the file system, spending a lot of time in file access, making transfers very inefficients, etc. Other solutions exist and are presented in this session.
Contents: Storing in files vs in database Using an in-memory database Using HDF5 CLI tools and libraries	Prerequisite: Being able to use SSH with private keys  Being familiar with a text editor  Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.) Passive knowledge of either C, Fortran, Octave, Python or R Working knowledge of C or Fortran Familiarity with OpenMP and MPI
	Type: Hands-on Target audience: Everyone Must: This session is a must-have for anyone who thinks generating a million small files is an optimal way of storing data.

datamanagement.pdf