Designing machine learning experiments using SLURM within a Cloud Trusted Research Environment

Description

Trusted Research Environments (TRE) are secure platforms which enable researchers to access and analyse personal data within the ‘Five Safes’ framework. TREs have been in use for many years to enable the safe use of sensitive data in research yet are not fully capable of supporting researcher needs including big data and high computational demands for machine learning (ML) workloads. Researchers are increasingly applying a range of machine learning (ML) algorithms on de-identified personal datasets derived from healthcare (for example: electronic health records, routinely collected medical scans and diagnosis information). TREs are looking to deploy on-demand high-performance computing (HPC) environments using third-party cloud computing providers that enable batch processing pipelines such as SLURM (Simple Linux Utility for Resource Management). Slurm is a open source, scalable scheduling tool for HPC environments which can be used to launch and monitor jobs on assigned nodes. This ‘on-demand’ approach allows for cost optimisation of the TRE resources in comparison to the ‘always on’ physical computing environment. Also, each researcher can be provisioned their own HPC environment, providing full data and network isolation from other projects, and able to grow dynamically depending on compute requirements.

Period	5 Sept 2023
Event title	Research Software Engineering Conference 2023
Event type	Conference
Conference number	7
Location	Swansea, United KingdomShow on map