Introduction to data storage and access

by Damien François (UCLouvain/CISM)

bibliotheque des sciences: salle pasteur (comodal (louvain-la-neuve or remote))

bibliotheque des sciences: salle pasteur

comodal (louvain-la-neuve or remote)

Join Zoom Meeting https://cern.zoom.us/j/68165517034?pwd=bllaeUZlNFlZZmh5RVYrVytudnJTQT09 Meeting ID: 681 6551 7034 Passcode: 257128 Join by SIP 68165517034@ 68165517034@ Join by H.323 Meeting ID: 681 6551 7034 Passcode: 257128

Storing data in an efficient way is very important for many scientific applications. Yet, most of the time, a myriad of small files is used, imposing a large burdun on the file system, spending a lot of time in file access, making transfers very inefficients, etc. Other solutions exist and are presented in this session.


  • Storing in files vs in database
  • Using an in-memory database
  • Using HDF5 CLI tools and libraries


  • Being able to use SSH with private keys 
  • Being familiar with a text editor 
  • Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.)
  • Passive knowledge of either C, Fortran, Octave, Python or R
  • Working knowledge of C or Fortran
  • Familiarity with OpenMP and MPI

Type: Hands-on
Target audience: Everyone
Must: This session is a must-have for anyone who thinks generating a million small files is an optimal way of storing data.

Organized by


Quality Survey
17 / 60