2020

Introduction to data storage and access

by Damien François (UCLouvain/CISM)

→ Europe/Brussels
Virtual Event

Virtual Event

Link to the event: https://teams.microsoft.com/l/meetup-join/19%3ameeting_OTg2NTk3ZWMtY2Q5YS00MTJlLWI2MjEtY2Y2M2U4NTZjOWY1%40thread.v2/0?context=%7b%22Tid%22%3a%227ab090d4-fa2e-4ecf-bc7c-4127b4d582ec%22%2c%22Oid%22%3a%226677b3f8-ce21-40ac-a3f2-4440ec78487e%22%2c%22IsBroadcastMeeting%22%3atrue%7d
Description

Storing data in an efficient way is very important for many scientific applications. Yet, most of the time, a myriad of small files is used, imposing a large burdun on the file system, spending a lot of time in file access, making transfers very inefficients, etc. Other solutions exist and are presented in this session.

Contents:

  • Storing in files vs in database
  • Using an in-memory database
  • Using HDF5 CLI tools and libraries

Prerequisite:

  • Being able to use SSH with private keys 
  • Being familiar with a text editor 
  • Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.)
  • Passive knowledge of either C, Fortran, Octave, Python or R
  • Working knowledge of C or Fortran
  • Familiarity with OpenMP and MPI

Type: Hands-on
Target audience: Everyone
Must: This session is a must-have for anyone who thinks generating a million small files is an optimal way of storing data.
datamanagement.pdf
Organised by

UCLouvain/CISM

Contact
Surveys
Quality Survey
Fill out