By integrating the interTwin DTE core module Infrastructure Manager (IM) with Chameleon, a transatlantic testbed composed of OSCAR clusters in Europe and the USA was deployed. Using Common Workflow Language (CWL), we achieved seamless execution of scientific workflows for flood assessment.
The Challenge
In this project, the objective was to address the computational and infrastructure challenges of combining resources from two OSCAR clusters located in different distributed Cloud infrastructures and regions—EGI Federated Cloud in Europe and Chameleon in the USA. By uniting these clusters, the project aimed to achieve three key goals:
- Optimise Resource Utilisation: Allocate tasks to the cluster with available capacity to maximise efficiency and streamline operations.
- Ensure Data Locality Compliance: Assign tasks to the cluster closest to the required data, reducing latency and improving processing speed.
- Increase System Resilience: Create a robust infrastructure capable of executing workloads between clusters during maintenance or unforeseen disruptions, ensuring uninterrupted operations.
These goals reflect the importance of efficiently harnessing distributed computational resources while overcoming geographic and operational barriers.
Solution
The project utilised Infrastructure Manager (IM) to deploy OSCAR clusters in Europe (EGI federated cloud) and the USA (Chameleon), enabling seamless setup across both sites. OSCAR, an open-source Kubernetes-based serverless platform, was used for event-driven computation to efficiently run the services required for the scientific workflow.
The Common Workflow Language (CWL) was applied to develop and define the scientific workflows, with the FloodAdapt Digital Twin (DT-Flood) as the use case. CWL integrated a Python script that leveraged the oscar-python library, enabling seamless connection and interaction with the OSCAR clusters across regions. This streamlined approach ensured clarity and efficiency in executing distributed workflows.
DT-Flood
+
Python
+
A Python script, using the oscar-python library, allows users to decide where to offload computation at the step level of the CWL-defined workflow
Looking Forward
We plan to further enhance the integration of the Infrastructure Manager with Chameleon to facilitate the deployment of customized virtualized infrastructure for Chameleon users using the easy-to-used IM Dashboard. This will unlock access to the wide catalog of curated recipes for the deployment of popular applications (e.g. Kubernetes, SLURM-based Clusters, MLFlow, etc.). The advanced interoperability that provides the IM as an orchestrator of Cloud-based infrastructures will facilitate the deployment of additional transatlantic computational testbeds required to aggregate disparate computing Cloud-based resources from large-scale distributed infrastructures.
