HDF5 data format

In the beginning there was the EDF format...

INFO: Problems caused by the "ESRF data format" are unfortunately far too common. The orginal software available from the ESRF is far too bugged to be usable. It is not available for many of the operating systems on which FIT2D is required to run, and users use. Therefore I (and others) have written their own input routines. However, this is only a partial solution. This is because the "format" is totally inadequately defined, and even where defined there are huge differences between the specification and the files actually produced. It is also an unnecessarily complicated format with certain data compression schemes almost totally undefined. Therefore this code has only been written to input a small subset of the possible file formats which could be produced. In particular all header information must be found before the image data, and data compression is not supported. Message when importing EDF files in Fit2D

...now we have a NeXus standard

NeXus is an effort by an international group of scientists to define a common data exchange and archival format for neutron, X-ray and muon experiments. NeXus is built on top of the scientific data format HDF5 and adds domain-spe- cific rules for organizing data within HDF5 files, in addition to a dictiona- ry of well defined domain-specific field names. The NeXus data format has two purposes. First, it defines a format that can serve as a container for all relevant data associated with a beamline. This is a very important use case. Second, it defines standards in the form of application definitions for the exchange of data between applications. NeXus provides structures for raw experimental data as well as for processed data. J. Appl. Cryst. (2015). 48, 301-305

This sounds promising, but reality is as usual much, much more chaotic. In fact, it is nowadays rather straightforward to read or write data/error arrays together with important metadata in EDF format. Routines are available for Python (fabio library) or Matlab (see EDF data format).

NeXus on the other hand is a standard which leaves a lot of room for interpretation. As a result, there are unfortunately no routines readily available for Python/Matlab to read data/error arrays and metadata from HDF5 files as comfortably as it is the case for EDF files. In the following I list the search path for data and metadata for a selection of known interpretations of the NeXus standard. The main problem in many cases is to programmatically determine the present data type, as there is no common identifier foreseen. Once known, it is rather easy to extract data as high level routines are available in programming languages. The correct interpretation of metadata needs to be taken care in each case individually.

List of important HDF 'formats' using NeXus standard - RAW data

Lima written raw data at ESRF from 2020

entry	further HDF5 groups				interpretation
entry_{number}					* default attribute at root level which points to relevant entry number * default attribute at entry level which points to relevant plot (NXdata)
	instrument
		{detector_name}
			plot		NX_class=NXdata
				data	data array
			header		group with static metadata - at same level as plot

Lima written raw data at ESRF before 2020

entry	further HDF5 groups				interpretation
entry_{number}					* default attribute at root level which points to relevant entry number * default attribute at entry level which points to relevant NXdata group
	measurement
		{detector_name}
			data		NX_class=NXdata
				array	data array
				header	group with static metadata

Dynamic metadata for both types of raw data (scalers file)

entry	further HDF5 groups				interpretation
entry_{number}					* default attribute at root level which points to relevant entry number
	id02
		MCS			multi channel scalers
		TFG			time frame generator
		parameters			group with copy of static metadata

Dectris written raw data

entry	further HDF5 groups			interpretation
entry
	data				NXclass: NXdata
		data			data array
					no metadata available

List of important HDF 'formats' using NeXus standard - reduced data

Written by online datareduction at ESRF (pyFAI/DAHU) - SAXS

entry	further HDF5 groups				interpretation

Written by online datareduction at ESRF (pyFAI/DAHU) - XPCS

entry	further HDF5 groups			interpretation

Written by SAXSutilites2 package

entry	further HDF5 groups			interpretation

Written by saxs programs package

entry	further HDF5 groups			interpretation