Core DTE Modules

dCNiOS

Updated 13/02/2024

Description

A new open-source command-line tool to easily manage the creation of event-driven data processing flows.

When files are uploaded to dCache, events are ingested by Apache NiFi, which can queue them up depending on the (modifiable at runtime) ingestion rate. Then, they are delegated for processing into a scalable OSCAR cluster, where a user-defined application, based on a Docker image, can process the data file.

Release Notes

This is a new development performed in interTwin, facilitating the integration of dCache within OSCAR. Apart from the command-line application available in the GitHub repository above, this development has also been supplemented i) by the corresponding TOSCA templates and ii) by ansible roles that are required to deploy an Apache Nifi cluster via the Infrastructure Manager (IM) described in section 2.4.1 and section 2.4.2. Any user can self-deploy such a cluster via the IM Dashboard – https://im.egi.eu.

Future Plans

dCNiOS will be further extended according to the use case requirements. It currently supports user / password authentication to connect to the components. OIDC support may be integrated if it is deemed necessary.

Target Audience
+

Users of the Component Technology Integrators

License
+

Apache 2.0

Created by
+