A Python library for High Performance Data Analytics at scale

Value proposition PyOphidia provides the Python bindings for the Ophidia framework, an open-source solution for the analysis of scientific multi-dimensional data, joining HPC paradigms and Big Data approaches. It provides an environment targeting High Performance Data Analytics through parallel and in-memory data processing, data-driven task scheduling and server-side analysis. The framework supports the execution of complex analytics workflows in the form of DAGs of Ophidia operators. The PyOphidia module brings HPDA capabilities within the scientific Python ecosystem through a high-level interface that allows handling distributed datasets, running parallel analysis and interacting with High Performance Computing (HPC) infrastructures.

Release Notes

The Ophidia framework, and in particular its Python bindings, PyOphidia, have been extended in order to provide a preliminary support for Ophidia workflows coded in Common Workflow Language (CWL): https://github.com/OphidiaBigData/PyOphidia/tree/feature-cwl/PyOphidia/utils. In this preliminary development, new Python modules are provided for translating workflows from CWL format to the native Ophidia one. The new capability is based on the CWLtool command line interface, which performs this translation. The new capabilities also integrate direct submission of the translated workflow to the Ophidia server.

Future Plans

A stronger integration of the capabilities for supporting CWL format is envisioned in future releases. Moreover, support for workflows including non-Ophidia tasks will be explored. Besides, full documentation of the new capabilities, including examples of usage will be added. Furthermore, integration with the yProv service will be performed to support fine-grained provenance tracking applied to the workflows.