Users can set up an OSCAR cluster themselves on any available cloud infrastructure. The automatically scalable cluster can be used to scale file-based on-demand processing (e.g. automatically when a file is uploaded to an object store such as MinIO). Furthermore, it allows communication via HTTP-based calls for programmatic interaction with auto-scaled, stateless, user-defined services.
Release Notes
OSCAR has been adopted by interTwin in order to implement a generic framework for real-time data acquisition and processing that builds on event-triggered execution of workflows, to i) improve response times and minimise data transfers, and to ii) support new event sources.
For the first release, OSCAR has been extended to provide support to Apache Nifi, an open-source system that supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. The goal is to use Apache Nifi as an intermediate event-ingestion platform (buffering) to decouple the rate at which files are uploaded to an object/file, from the rate at which these data files are processed. In order to facilitate this integration, the component dCNiOS was created (see next subsection). This required i) an adaptation to the internal event structure that is used by Nifi and ii) an adaptation to perform dynamic data fetching from the FaaS supervisor component in OSCAR, to prevent data movements from the data storage system into Nifi (and then into OSCAR).
Future Plans
OSCAR will be evolved, during the interTwin project, to accommodate new sources of events. We will add support for incoming events from Amazon S3, a widely used data lake and object storage system from Amazon Web Services (AWS), the leading public Cloud provider. We will also include support for Kafka an open-source distributed event streaming platform for high-performance data pipelines, streaming analytics and data integration. Also, an integration between OSCAR and the interLink component will be performed to achieve event-driven support for data processing delegation into HPC supercomputers.