Experiments at neutron or photon source inevitable generate a substantial amount of scientific data and usually not in a format particular suited for the tasks involved. Some of these experiments, or rather the detectors recording the data, produce extremely high data rates in a rather erratic pattern. Coping with these data streams requires an alignment of the capabilities of detectors, scientific file formats and storage devices. To put discussions, issues and possible solutions on the broadest possible basis we initiated - in cooperation with people from HDFgroup - two groups at the Research Data Alliance (RDA). The first one is an Interest Group on Scientific File Formats, the second one is a Working Group on HDF5 filter extensions.
The Interest Group on Scientific File Formats (SFF) is intended as a long-term activity to initiate, coordinate and support SFF working groups targeting specific issues. The IG will primarily - but not exclusively - consider open formats and standards like HDF5. The IG intends to promote active involvement of developers of open standards, applications and scientific instruments as well as scientific communities. We made already a good start with HDFgroup, IUCr, NIAC as well as with detector and software vendors. Read the full IG-proposal here!
The Working Group on HDF5 filter extensions will focus on a very specific, technical implementation of a rather new enhancement to HDF5. This enhancement is essential for upcoming scientific instruments and has been funded jointly by scientific research facilities and detector companies, and been implemented by HDFgroup. The WG will also serve as a kind of "proof of concept" for the SFF-IG and is designed to produce concrete results within 12-18 month. Both the IG and WG drafts have been co-authored by members of HDFgroup and the NeXus NIAC. Read the full proposal here!
The proposed IG and WG will be presented at the RDA 4th plenary. Whoever is interested in these topics is very welcome to join the initiatives. Just drop a message to pan-data@desy.de and we'll keep you posted about the group endorsement and upcoming activities at the RDA 4th plenary.
