A new open-source command-line tool to easily manage the creation of event-driven data processing flows. DCNiOS, Data Connector through Apache NiFi for OSCAR, facilitates the creation of event-driven processes connecting a Storage System like dCache to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi.
Release Notes
DCNiOS was developed within the interTwin project. The current release facilitates the integration between dCache and OSCAR. DCNiOS creates a dataflow that captures events in dCache, queues them up in a Nifi dataflow, and ingests them in an OSCAR cluster at a customized rate, where an OSCAR service is run based on a user-defined application (containerized in a Docker image). The DCNiOS command-line application is available in the Source Code repository. Additionally, the corresponding TOSCA templates and the ansible roles that are required to deploy an Apache Nifi cluster via the Infrastructure Manager (IM) have been provided. Any user can self-deploy such a cluster via the IM Dashboard.
Future Plans
DCNiOS will be further extended to improve security, efficiency, and compatibility. Some future implementation plans may include support for OIDC authentication and support for additional data sources and event types. For instance, we aim to incorporate event handling similar to what Apache Kafka offers. This enhancement will allow our tool to seamlessly integrate with a broader range of systems and provide more robust event-driven capabilities.