h5py example writing a simple NeXus data file with links

Building on the previous example, we wish to identify our measured data with the detector on the instrument where it was generated. In this hypothetical case, since the detector was positioned at some angle two_theta, we choose to store both datasets, two_theta and counts, in a NeXus group. One appropriate NeXus group is NXdetector. This group is placed in a NXinstrument group which is placed in a NXentry group. Still, NeXus requires a NXdata group. Rather than duplicate the same data already placed in the detector group, we choose to link to those datasets from the NXdata group. (Compare the next figure with Linking in a NeXus file in the NeXus Design chapter of the NeXus User Manual.) The NeXus Design chapter provides a figure (Linking in a NeXus file) with a small variation from our previous example, placing the measured data within the /entry/instrument/detector group. Links are made from that data to the /entry/data group.

h5py example showing linking in a NeXus file

The Python code to build an HDF5 data file with that structure (using numerical data from the previous example) is shown below.

#!/usr/bin/env python
'''
Writes a simple NeXus HDF5 file using h5py with links
according to the example from Figure 2.1 in the Design chapter
'''

import my_lib

INPUT_FILE = 'input.dat'
HDF5_FILE = 'writer_2_1.hdf5'

#---------------------------

tthData, countsData = my_lib.get2ColumnData(INPUT_FILE)

f = my_lib.makeFile(HDF5_FILE)  # create the HDF5 NeXus file

nxentry = my_lib.makeGroup(f, 'entry', 'NXentry')
nxinstrument = my_lib.makeGroup(nxentry, 'instrument', 'NXinstrument')
nxdetector = my_lib.makeGroup(nxinstrument, 'detector', 'NXdetector')

tth = my_lib.makeDataset(nxdetector, "two_theta", tthData, units='degrees')
counts = my_lib.makeDataset(nxdetector, "counts", countsData, 
                   units='counts', signal=1, axes='two_theta')

nxdata = my_lib.makeGroup(nxentry, 'data', 'NXdata')
my_lib.makeLink(nxdetector, tth, nxdata.name+'/two_theta')
my_lib.makeLink(nxdetector, counts, nxdata.name+'/counts')

f.close()   # be CERTAIN to close the file

It is interesting to compare the output of the h5dump of the data file writer_2_1.hdf5 with our Python instructions.

HDF5 "writer_2_1.hdf5" {
GROUP "/" {
   GROUP "entry" {
      ATTRIBUTE "NX_class" {
         DATATYPE  H5T_STRING {
               STRSIZE 7;
               STRPAD H5T_STR_NULLPAD;
               CSET H5T_CSET_ASCII;
               CTYPE H5T_C_S1;
            }
         DATASPACE  SCALAR
         DATA {
         (0): "NXentry"
         }
      }
      GROUP "data" {
         ATTRIBUTE "NX_class" {
            DATATYPE  H5T_STRING {
                  STRSIZE 6;
                  STRPAD H5T_STR_NULLPAD;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
            DATASPACE  SCALAR
            DATA {
            (0): "NXdata"
            }
         }
         DATASET "counts" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 31 ) / ( 31 ) }
            DATA {
            (0): 1037, 1318, 1704, 2857, 4516, 9998, 23819, 31662, 40458,
            (9): 49087, 56514, 63499, 66802, 66863, 66599, 66206, 65747,
            (17): 65250, 64129, 63044, 60796, 56795, 51550, 43710, 29315,
            (25): 19782, 12992, 6622, 4198, 2248, 1321
            }
            ATTRIBUTE "units" {
               DATATYPE  H5T_STRING {
                     STRSIZE 6;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "counts"
               }
            }
            ATTRIBUTE "signal" {
               DATATYPE  H5T_STRING {
                     STRSIZE 1;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "1"
               }
            }
            ATTRIBUTE "axes" {
               DATATYPE  H5T_STRING {
                     STRSIZE 9;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "two_theta"
               }
            }
            ATTRIBUTE "target" {
               DATATYPE  H5T_STRING {
                     STRSIZE 33;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "/entry/instrument/detector/counts"
               }
            }
         }
         DATASET "two_theta" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 31 ) / ( 31 ) }
            DATA {
            (0): 17.9261, 17.9259, 17.9258, 17.9256, 17.9254, 17.9252,
            (6): 17.9251, 17.9249, 17.9247, 17.9246, 17.9244, 17.9243,
            (12): 17.9241, 17.9239, 17.9237, 17.9236, 17.9234, 17.9232,
            (18): 17.9231, 17.9229, 17.9228, 17.9226, 17.9224, 17.9222,
            (24): 17.9221, 17.9219, 17.9217, 17.9216, 17.9214, 17.9213,
            (30): 17.9211
            }
            ATTRIBUTE "units" {
               DATATYPE  H5T_STRING {
                     STRSIZE 7;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "degrees"
               }
            }
            ATTRIBUTE "target" {
               DATATYPE  H5T_STRING {
                     STRSIZE 36;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "/entry/instrument/detector/two_theta"
               }
            }
         }
      }
      GROUP "instrument" {
         ATTRIBUTE "NX_class" {
            DATATYPE  H5T_STRING {
                  STRSIZE 12;
                  STRPAD H5T_STR_NULLPAD;
                  CSET H5T_CSET_ASCII;
                  CTYPE H5T_C_S1;
               }
            DATASPACE  SCALAR
            DATA {
            (0): "NXinstrument"
            }
         }
         GROUP "detector" {
            ATTRIBUTE "NX_class" {
               DATATYPE  H5T_STRING {
                     STRSIZE 10;
                     STRPAD H5T_STR_NULLPAD;
                     CSET H5T_CSET_ASCII;
                     CTYPE H5T_C_S1;
                  }
               DATASPACE  SCALAR
               DATA {
               (0): "NXdetector"
               }
            }
            DATASET "counts" {
               HARDLINK "/entry/data/counts"
            }
            DATASET "two_theta" {
               HARDLINK "/entry/data/two_theta"
            }
         }
      }
   }
}
}

Look carefully! It appears from the output of h5dump that the actual data for two_theta and counts has moved into the NXdata group at HDF5 path /entry/data! But we stored that data in the NXdetector group at /entry/instrument/detector. This is normal for h5dump output.

A bit of explanation is necessary at this point. The data is not stored in either HDF5 group directly. Instead, HDF5 creates a DATA storage element in the file and posts a reference to that DATA storage element as needed. An HDF5 hard link requests another reference to that same DATA storage element. The h5dump tool describes in full that DATA storage element the first time (alphabetically) it is called. In our case, that is within the NXdata group. The next time it is called, within the NXdetector group, h5dump reports that a hard link has been made and shows the HDF5 path to the description.

NeXus recognizes this behavior of the HDF5 library and adds an additional structure when building hard links, the target attribute, to preserve the original location of the data. Not that it actually matters. The h5toText.py tool knows about the additional NeXus target attribute and shows the data to appear in its original location, in the NXdetector group.

writer_2_1.hdf5:NeXus data file
  entry:NXentry
    @NX_class = NXentry
    data:NXdata
      @NX_class = NXdata
      counts --> /entry/instrument/detector/counts
      two_theta --> /entry/instrument/detector/two_theta
    instrument:NXinstrument
      @NX_class = NXinstrument
      detector:NXdetector
        @NX_class = NXdetector
        counts:NX_INT32[31] = __array
          @units = counts
          @signal = 1
          @axes = two_theta
          @target = /entry/instrument/detector/counts
          __array = [1037, 1318, 1704, '...', 1321]
        two_theta:NX_FLOAT64[31] = __array
          @units = degrees
          @target = /entry/instrument/detector/two_theta
          __array = [17.926079999999999, 17.925909999999998, 
                     17.925750000000001, '...', 17.92108]