The tool is a seamless integration with the esgpull db API, which optimizes the workflow of downloading, cataloguing and appending to RUCIO of ESGF and CMIP data.
Release Notes
This initial release of the tool allows for synchronization between a local depot of ESGF datasets and RUCIO catalogues. It relies on an intermediary JSON catalogue that represents a targeted subset of data from the esgpull database.
Another key point about this release is that it does not handle the authentication side of things automatically, an hourly RUCIO login prompt would require using a browser and going through the EGI SSO check-in process.
Future Plans
The next steps include:
- bypassing the JSON catalogue intermediary and syncing directly with esgpull’s database
- automating the EGI SSO check-in process and generating on-the-fly RUCIO tokens for authentication and authorization
- improving error handling and message logging to future proof the tool.
