HDF5 Utilities

The logging framework uses the HDF5 standard to store log information. You can find more documentation on HDF5 at http://www.h5py.org/

HDF5 Container Class

class HDF5LogContainer(file_handle, name=None, compression=None)[source]

Class to define an HDF5 log container.

Parameters:
  • file_handle (h5py.File() – Handle of the HDF5 file
  • name (str, optional) – Name of the HDF5 group of the log container
  • compression (bool, optional) – HDF5 compression setting on the log container

When an HDF5LogContainer is created, the underlying HDF5 file will not be modified unless one of the write_* methods are called.

is_valid()[source]

Check that the HDF5 Log Container is valid.

Returns:is_valid – * True –> This is a valid HDF5 log file * False –> This is NOT a valid HDF5 log file
Return type:bool
write_log_data(log_data, append=True)[source]

Write the log data to the log container.

Parameters:
  • log_data (bytes) – Binary data from a WlanExpNode log
  • append (bool, optional) – Append to (True) or Overwrite (False) the current log data
write_log_index(log_index=None)[source]

Write the log index to the log container.

Parameters:log_index (dict) – Log index generated from wlan_exp log data

If the log index currently exists in the HDF5 file, that log index will be replaced with this new log index. If log_index is provided then that log index will be written to the log container. Otherwise, a raw log index will be generated and added to the log container.

write_attr_dict(attr_dict)[source]

Add the given attribute dictionary to the opened log container.

Parameters:attr_dict (dict) – A dictionary of user provided attributes that will be added to the HDF5 group.
get_log_data_size()[source]

Get the current size of the log data in the log container.

Returns:size – Number of bytes of log data in the log container
Return type:int
get_log_data()[source]

Get the log data from the log container.

Returns:log_data – Bytes object of the log data in the container
Return type:bytes
get_log_index(gen_index=True)[source]

Get the raw log index from the log container.

Parameters:gen_index (bool, optional) – Generate the raw log index if the log index does not exist in the log container.
Returns:log_index – Log index from the log container
Return type:dict
get_attr_dict()[source]

Get the attribute dictionary from the log container.

Returns:attr_dict – The dictionary of user provided attributes in the log container.
Return type:dict

HDF5 Utility Functions

These utility functions wrap the interactions with the HDF5 Container class.

hdf5_open_file(filename, readonly=False, append=False, print_warnings=True)[source]

Open an HDF5 file.

Parameters:
  • filename (str) – Filename of the HDF5 file to open
  • readonly (bool, optional) – Open the file in read-only mode
  • append (bool, optional) – Append to the data in the current file
  • print_warnings (bool, optional) – Print warning messages
Returns:

file_handle – Handle for the HDF5 file

Return type:

h5py.File

Behavior of input attributes:
readonly append Behavior
True T/F File opened in read-only mode
False True File opened in append mode; created if it does not exist
False False If file with filename exists, then a new filename is generated using the log utilities. The new file is then created by the h5py File method (DEFAULT)
hdf5_close_file(file_handle)[source]

Close an HDF5 file.

Parameters:file_handle (h5py.File) – Handle for the HDF5 file
log_data_to_hdf5(log_data, filename, attr_dict=None, gen_index=True, overwrite=False)[source]

Create an HDF5 file that contains the log_data, a raw_log_index, and any user attributes.

Parameters:
  • log_data (bytes) – Binary data from a WlanExpNode log
  • filename (str) – Filename of the HDF5 file to appear on disk
  • attr_dict (dict, optional) – A dictionary of user provided attributes that will be added to the HDF5 group.
  • gen_index (bool, optional) – Generate the raw_log_index from the log_data and store it in the file.
  • overwrite (bool, optional) – If True method will overwrite existing file with filename

If the requested filename already exists and overwrite==True this method will replace the existing file, destroying any data in the original file.

If the filename already exists and overwrite==False this method will print a warning, then create a new filename with a unique date-time suffix.

hdf5_to_log_data(filename, group_name=None)[source]

Extract the log_data from an HDF5 Log Container

Parameters:
  • filename (str) – Name of HDF5 file to open
  • group_name (str, optional) – Name of Group within the HDF5 file object (defaults to “”)
Returns:

log_data – Log data in the HDF5 file

Return type:

bytes

hdf5_to_log_index(filename, group_name=None, gen_index=True)[source]

Extract the log_index from an HDF5 Log Container

Parameters:
  • filename (str) – Name of HDF5 file to open
  • group_name (str, optional) – Name of Group within the HDF5 file object (defaults to “”)
  • gen_index (bool, optional) – Generate the raw_log_index from the log_data and store it in the file if the log_index is not in the file.
Returns:

log_index – Either the log_index from HDF5 file or a generated raw_log_index from log_data in HDF5 file

Return type:

dict

hdf5_to_attr_dict(filename=None, group_name=None)[source]

Extract the attribute dictionary from an HDF5 Log Container.

Parameters:
  • filename (str) – Name of HDF5 file to open
  • group_name (str, optional) – Name of Group within the HDF5 file object (defaults to “”)
Returns:

attr_dict – The dictionary of user provided attributes in the HDF5 file

Return type:

dict

np_arrays_to_hdf5(filename, np_log_dict, attr_dict=None, compression=None)[source]

Generate an HDF5 file from numpy arrays.

Parameters:
  • filename (str) – Name of HDF5 file to open
  • np_log_dict (Numpy Array) – Numpy array to add to the HDF5 file
  • attr_dict (dict, optional) – A dictionary of user provided attributes that will be added to the HDF5 file.
  • compression (bool, optional) – HDF5 compression setting on the log container
The np_log_dict input must be either:
  1. A dictionary with numpy record arrays as values; each array will be a dataset in the HDF5 file root group
  2. A dictionary of dictionaries like (1); each top-level value will be a group in the root HDF5 group, each numpy array will be a dataset in the group.

attr_dict is optional. If provied, values in attr_dict will be copied to HDF5 group and dataset attributes. attr_dict values with keys matching np_log_dict keys will be used as dataset attributes named '<the_key>_INFO'. Attribute dictionary entries may have an extra value with key '/', which will be used as the value for a group attribute named 'INFO'.

Examples:

# No groups - all datasets in root group
np_log_dict = {
    'RX_OFDM':  np_array_of_rx_etries,
    'TX_HIGH':  np_array_of_tx_entries
}

attr_dict = {
    '/':        'Data from some_log_file.bin, node serial number W3-a-00001, written on 2014-03-18',
    'RX_OFDM':  'Filtered Rx OFDM events, only good FCS receptions',
    'TX_HIGH':  'Filtered Tx events, only DATA packets'
}

# Two groups, with two datasets in each group
np_log_dict = {
    'Log_Node_A': {
        'RX_OFDM':  np_array_of_rx_etries_A,
        'TX_HIGH':  np_array_of_tx_entries_A
    },
    'Log_Node_B': {
        'RX_OFDM':  np_array_of_rx_etries_B,
        'TX_HIGH':  np_array_of_tx_entries_B
    }
}

attr_dict = {
    '/':        'Written on 2014-03-18',
    'Log_Node_A': {
        '/':        'Data from node_A_log_file.bin, node serial number W3-a-00001',
        'RX_OFDM':  'Filtered Rx OFDM events, only good FCS receptions',
        'TX_HIGH':  'Filtered Tx events, only DATA packets'
    }
    'Log_Node_B': {
        '/':        'Data from node_B_log_file.bin, node serial number W3-a-00002',
        'RX_OFDM':  'Filtered Rx OFDM events, only good FCS receptions',
        'TX_HIGH':  'Filtered Tx events, only DATA packets'
    }
}