The EasyBuild (https://easybuilders.github.io/easybuild/) User Meeting is an open and highly interactive event that provides a great opportunity to meet fellow EasyBuild enthusiasts, discuss related topics and learn about new aspects of the tool.
It is intended for people that are already familiar with EasyBuild, ranging from occasional users to Easybuild core developers and experts, and topics will be less introductory in nature than during EasyBuild hackathons/workshops that have been organised in the past.
The program includes presentations by both EasyBuild users and developers, next to hands-on sessions.
The presentations will cover a wide variety of EasyBuild related topics, including how EasyBuild has been integrated into existing infrastructures, how it can be combined with other tools, (new) EasyBuild features that are considered particularly useful, or other tools that are somehow (loosely) related to EasyBuild and could be of interest to attendees.
The hands-on sessions are intended for becoming more familiar with the practical aspects of EasyBuild, resolving problems with the help of others, and/or working together to further enhance EasyBuild. Participants are expected to decide for themselves what they work on during the hands-on sessions.
Note that registration is free BUT mandatory.
President of the Institute of Condensed Matter and Nanosciences
Report on the survey
Status of easybuild
Future of easybuild
Site report
Overview of scientific software management at CSCS, including EasyBuild setup, recipe review, automated deployment and Continuous Integration with Jenkins pipelines + Github
We would like to present our site, the simple way we use the EasyBuild and our own extensions. It can be an inspiration for starting EasyBuild users.
Fred Hutch Cancer Research Center has about 230 scientific faculties and
just over 3,000 staff. Located in Seattle, Washington on a 15 Acre
Campus (60,702 M^2). We have an HPC group of 6 people and support 6
thousand core cluster on campus.
The Fred Hutch has hundreds of R users. Robert Gentleman who is one of
the co-developers of R worked at the Hutch while still actively
developing the language. Gentleman now works at 23andMe. Bioconductor
originated at the Fred Hutchinson Cancer Research Center in the Fall of
2001. Martin Morgan led Bioconductor development since 2008 joined the
Fred Hutch in 2005. The BioConductor group has since moved to Roswell
Park Cancer Center. R is the predominant language used by BioInformatics
staff at the Hutch. Our local R build has over 700 modules. Our HPC
group (me) has to devote considerable effort to maintain R for the center.
Our users want the latest version of R built as quickly as possible.
Using the foss-2018b toolchain, we will deploy the standard EB release
of R with a day of its Release. The Fred Hutch release of R will be
built as Bundle using the community release as a base. The Fred Hutch R
build has over 800 modules. Our local version of R has a version suffix
of "-fhX."
I am the author of easy_update. Easy_update is used to update package
versions for R and Python easyconfig recipes. Version information is
obtained from the respected package authority CRAN, PyPi, and
BioConductor. Easy_update resolves package dependencies.
Easy_Annotate documents our R and Python builds at the package level.
Fred Hutch uses GitHub pages to document our Easybuild scientific
software: https://urldefense.proofpoint.com/v2/url?u=https-3A__fredhutch.github.io_easybuild-2Dlife-2Dsciences_&d=DwIDaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=3TR-iteG1SyRqQ5yubQg-_2KIAToz9bj5dZrRdW36Hc&m=aKY87YplHcgIjHlZTGEyDTp23LoyK5aKhezrP87D0Y0&s=HDoEsgVW8-k4qfXW0pL4xEB3uL0wj73fJ6kVKN8rOho&e=
I plan to add the easy_update features to easybuild. Easy_update has
been under much change due to the move to PyPi and needing to support
Bioconductor packages for R., And I keep finding horrible bugs which I
would not want to in EasyBuild.
Validating experimental results from articles has finally become a norm at many HPC and systems conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.
While overviewing more than 100 papers during artifact evaluation at PPoPP, CGO, PACT and other conferences, I noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated me to develop Collective Knowledge (CK), an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing time-consuming experiments across computing resources provided by volunteers similar to SETI@home, applying statistical analysis and machine learning, validating results and plotting them on a common scoreboard for open and fair comparison, automatically generating interactive articles, and so on: http://cKnowledge.org.
In this talk I will introduce CK concepts and present several real world use cases from General Motors, Amazon and Arm on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for deep learning. I will also present results and reusable CK components from the 1st ACM ReQuEST optimization tournament: http://cKnowledge.org/request. Finally, I will introduce our latest initiative to create an open repository of reusable research components and workflows, and conclude with an open discussion on how to possibly connect it with the EasyBuild package manager.
High performance computing (HPC) - the aggregation of computers into clusters to increase computing speed and power- relies heavily on the software that connects and manages the various nodes in the cluster. Linux is the dominant HPC operating system, and many HPC sites expand upon the operating system's capabilities with different scientific applications, libraries, and other tools.
To avoid duplication of the necessary steps to run an HPC site the OpenHPC project was created in response to these issues. OpenHPC is a collaborative, community-based effort under the auspices of the Linux Foundation to solve common tasks in HPC environments by providing documentation and building blocks that can be combined by HPC sites according to their needs.
This talk gives an introduction in OpenHPC and how it tries to help to set up HPC systems.
Regression testing of HPC systems is of crucial importance when it comes to ensure the quality of service offered to the end users. At the same time, it poses a great challenge to the systems and application engineers to continuously maintain regression tests that cover as many aspects as possible of the user experience. In this presentation, we present ReFrame, a new framework for writing regression tests for HPC systems. ReFrame is designed to abstract away the complexity of the interactions with the system and separate the logic of a regression test from the low-level details, which pertain to the system configuration and setup. Regression tests in ReFrame are simple Python classes that specify the basic parameters of the test plus any additional logic. The framework will load the test and send it down a wel-defined pipeline which will take care of its execution. All the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment, are performed by the different pipeline stages. Thanks to its high-level abstractions and modular design, ReFrame can also serve as a tool for continuous integration (CI) of scientific software, complementary to other well-known CI solutions. Finally, we present the use cases of two large HPC centers that have adopted or are now adopting ReFrame for regression testing of their computing facilities.
Site report
EasyBuild is very helpful to researchers using electronic-structure codes by progressively replacing a lot of quick-and-dirty installation scripts and hastily set-up directory tree structures by a well-organised and systematic deployment frameowrk for HPC clusters.
One of the major challenges this field is currently facing is the so-called "dependency hell" arising from an increasing and necessary modularization of the numerous software components developed within larger and larger collaborative efforts. The geographical and cultural dispersion of the various developer teams is another one.
Through an overview of EasyBuild-related efforts undertaken for ABINIT and SIESTA since 2017, as well as the provision of various EasyConfigs in 2019 for the Electronic Structure Library, we will describe what the different stakeholders are experiencing within these endeavours and what could be done to further support them.
I will explain our Python setup: we compile it using the "dummy " toolchain with a very limited set of packages. We ask users to use virtual environments for anything beyond the Scientific Python stack. We also developed and are contributing a method for a single module to work with multiple versions of Python instead of being bound to a specific version. See also https://docs.computecanada.ca/wiki/Python
Capabilities to consistently handle dependencies between modulefiles have been improved recently on Modules. This talk will introduce the new automated module handling mechanisms and will also shed lights on further ongoing work on that field.
These features offer new perspectives for build and installation framework like EasyBuild when it comes to provide a way to consistently access the numerous software installed.
Some ideas will be shared with the audience on how to design modulefiles to leverage the automated module handling mechanisms in a multi build-toolchain context.
The European High-Perfomance Computing Joint Undertaking (EuroHPC JU) will pool European resources to develop exascale supercomputers for European researchers and industry. The amount of funding involved is very large (in the billions) and there may well be some opportunities to fund development of EasyBuild through some of the funding calls that are expected to emerge in 2019. I would like to look at some of the scope indicated by the EU to date and also discuss this in the context of HPC Centres of Excellence.