Users can set up an OSCAR cluster themselves on any available cloud infrastructure. The automatically scalable cluster can be used to scale file-based on-demand processing (e.g. automatically when a file is uploaded to an object store such as MinIO). Furthermore, it allows communication via HTTP-based calls for programmatic interaction with auto-scaled, stateless, user-defined services.
OSCAR has been adopted by interTwin in order to implement a generic framework for real-time data acquisition and processing that builds on event-triggered execution of workflows, to i) improve response times and minimise data transfers, and to ii) support new event sources.
For the first release, OSCAR has been extended to provide support to Apache Nifi, an open-source system that supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. The goal is to use Apache Nifi as an intermediate event-ingestion platform (buffering) to decouple the rate at which files are uploaded to an object/file, from the rate at which these data files are processed. In order to facilitate this integration, the component dCNiOS was created (see next subsection). This required i) an adaptation to the internal event structure that is used by Nifi and ii) an adaptation to perform dynamic data fetching from the FaaS supervisor component in OSCAR, to prevent data movements from the data storage system into Nifi (and then into OSCAR).
OSCAR will be evolved, during the interTwin project, to accommodate new sources of events. It has been explored the usage of TriggerMesh to support incoming events from Amazon S3, a widely used data lake and object storage system from Amazon Web Services (AWS), the leading public Cloud provider. TriggerMesh will allow us to create a gateway between public data lakes and thus to support long-term data persistence. It also enables the dynamic deployment of OSCAR clusters on any cloud infrastructure – such as the EGI’s federated cloud or as any public cloud provider – on which the project-defined workflows can be triggered. Also, an integration between OSCAR and the interLink component will be performed in order to achieve event-driven support for data processing delegation into HPC supercomputers.