Introduction to data storage and access

• Damien François (UCLouvain/CISM)

Room: BST: Pasteur <table border="0" cellpadding="10px"> <tbody> <tr> <td colspan="2"> <p>Storing data in an efficient way is very important for many scientific applications. Yet, most of the time, a myriad of small files is used, imposing a large burdun on the file system, spending a lot of time in file access, making transfers very inefficients, etc. Other solutions exist and are presented in this session.</p> </td> </tr> <tr> <td rowspan="2"> <p><strong>Contents:</strong></p> <ul> <li>Storing in files vs in database</li> <li>Using an in-memory database</li> <li>Using HDF5 CLI tools and libraries</li> </ul> </td> <td> <p><strong>Prerequisite:</strong></p> <ul> <li>Being able to use SSH with private keys&nbsp;</li> <li>Being familiar with a text editor&nbsp;</li> <li>Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.)</li> <li>Passive knowledge of either C, Fortran, Octave, Python or R</li> <li>Working knowledge of C or Fortran</li> <li>Familiarity with OpenMP and MPI</li> </ul> </td> </tr> <tr> <td> <p><strong>Type:</strong>&nbsp;Hands-on<br /> <strong>Target audience</strong>: Everyone<br /> <strong>Must:&nbsp;</strong>This session is a must-have for anyone who thinks generating a million small files is an optimal way of storing data.</p> </td> </tr> </tbody> </table>

Efficient data storage on CECI clusters

• Ariel Lozano (ULB)

Room: BST: Pasteur <table border="0" cellpadding="10px"> <tbody> <tr> <td colspan="2"> <p>The CECI clusters are equipped with <a class="moz-txt-link-freetext" href="https://support.ceci-hpc.be/doc/_contents/ManagingFiles/Storage.html">different storage solutions</a> that you can use for managing your data.<br /> Each of them have different properties such as capacity, I/O performance, accessibility and data longevity as they are meant for different usages.<br /> In this presentation we will go through the different options we have on the clusters and explain how to organize your workflows to make an efficient and practical use of them.</p> </td> </tr> <tr> <td rowspan="2"> <p><strong>Contents:</strong></p> <ul> <li>Storage solutions on the CECI clusters<br /> &nbsp;</li> <li>CECI environment variables for data location</li> <li>Data operations inside Slurm batch scripts</li> </ul> </td> <td> <p><strong>Prerequisite:</strong></p> <ul> <li>Being able to login to cluster&nbsp;</li> <li>Being familiar with a text editor&nbsp;</li> <li>Being able to submit jobs with Slurm</li> <li>Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.)</li> </ul> </td> </tr> <tr> <td> <p><strong>Type:</strong>&nbsp;Hands-on<br /> <strong>Target audience</strong>: Everyone<br /> <strong>Must:&nbsp;</strong>This session is a must-have for anyone who doesn't know where $HOME, $WORKDIR, $GLOBALSCRATCH, $LOCALSCRATCH or $CECIHOME points to.</p> </td> </tr> </tbody> </table>

Open Science and Open Research Data / Data Management Plan

• Jonathan Dedonder (IACCHOS)

Room: BST: Pasteur The growing relevance of Open Science poses challenges to research practices. Open Research Data, which aims to provide free access to research data in order to ensure the reproducibility of scientific results, is one important aspect of Open Science. Research Data Management (RDM), on its side, addresses the entire life cycle of data, covering planning, collection, management, storage, publication, referencing, preservation and sharing of research data, as well as access and reuse rights. This seminar addresses concerns of openness, covers the integration of open Data/FAIR Data into research data management principles as well as practical aspects such as the publication of data in repositories.

Data versioning

• Damien François (UCLouvain/CISM)

Room: BST: Pasteur <table border="0" cellpadding="10px"> <tbody> <tr> <td colspan="2"> <p>Everyone is familiar with code versioning, that allows recalling what modification was implementer in the code, by whom, when, and why. The same idea can be transposed to data, but requires a specific set of tools, and while Git is the <i>de facto</i> standard tool for code, it is not really suitable for data. Other options exist, either as a Git plugin, a standalone CLI tool, or a full-featured data management website. The landscape for data versioning will be presented in this session, with a focus on a simple to use and simple to install CLI tool: Datalad.</p> </td> </tr> <tr> <td rowspan="2"> <p><strong>Contents:</strong></p> <ul> <li>Specific aspects of data versioning vs code versioning</li> <li>The landscape of tools for data versioning</li> <li>Tutorial using Datalad</li> </ul> </td><tr> <td> <p><strong>Prerequisite:</strong></p> <ul> <li>Being able to use SSH with private keys&nbsp;</li> <li>Being familiar with a text editor&nbsp;</li> <li>Mastering the Linux command line and the GNU utilities (mkdir, cp, scp, etc.)</li> <li>Familiarity with code versioning</li> </ul> </td> </tr> </tr> <tr> <td> <p><strong>Type:</strong>&nbsp;Hands-on<br /> <strong>Target audience</strong>: Everyone<br /> <strong>Must:&nbsp;</strong>This session is interesting for users who must process data and recall what was done to which data piece.</p> </td> </tr> </tbody> </table>