| In the beginning there was the EDF format...
|
INFO: Problems caused by the "ESRF data format" are unfortunately far too
common. The orginal software available from the ESRF is far too
bugged to be usable. It is not available for many of the operating
systems on which FIT2D is required to run, and users use. Therefore
I (and others) have written their own input routines. However, this
is only a partial solution. This is because the "format" is totally
inadequately defined, and even where defined there are huge
differences between the specification and the files actually produced.
It is also an unnecessarily complicated format with certain data
compression schemes almost totally undefined. Therefore this code has
only been written to input a small subset of the possible file formats
which could be produced. In particular all header information must be
found before the image data, and data compression is not supported.
Message when importing EDF files in Fit2D
|
| ...now we have a NeXus standard
|
NeXus is an effort by an international group of scientists to define a common
data exchange and archival format for neutron, X-ray and muon experiments.
NeXus is built on top of the scientific data format HDF5 and adds domain-spe-
cific rules for organizing data within HDF5 files, in addition to a dictiona-
ry of well defined domain-specific field names. The NeXus data format has two
purposes. First, it defines a format that can serve as a container for all
relevant data associated with a beamline. This is a very important use case.
Second, it defines standards in the form of application definitions for the
exchange of data between applications. NeXus provides structures for raw
experimental data as well as for processed data.
J. Appl. Cryst. (2015). 48, 301-305
|
This sounds promising, but reality is as usual much, much more chaotic. In fact, it is nowadays rather straightforward to read or write data/error arrays together with important metadata in EDF format. Routines are available for Python (fabio library) or Matlab (see EDF data format).
NeXus on the other hand is a standard which leaves a lot of room for interpretation. As a result, there are unfortunately no routines readily available for Python/Matlab to read data/error arrays and metadata from HDF5 files as comfortably as it is the case for EDF files.
In the following I list the search path for data and metadata for a selection of known interpretations of the NeXus standard. The main problem in many cases is to programmatically determine the present data type, as there is no common identifier foreseen. Once known, it is rather easy to extract data as high level routines are available in programming languages. The correct interpretation of metadata needs to be taken care in each case individually.
List of important HDF 'formats' using NeXus standard - RAW data
Lima written raw data at ESRF from 2020
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
Lima written raw data at ESRF before 2020
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
Dectris written raw data
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
List of important HDF 'formats' using NeXus standard - reduced data
Written by online datareduction at ESRF (pyFAI/DAHU) - SAXS
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
Written by online datareduction at ESRF (pyFAI/DAHU) - XPCS
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
Written by SAXSutilites2 package
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
Written by saxs programs package
| entry
|
further HDF5 groups
|
interpretation
|
| entry
|
|
|
|
|
|
|
data
|
|
|
NXclass: NXdata
|
|
|
|
data
|
|
data array
|
|
|
|
|
|
no header information available
|
