DQA internal data format#

The DQA frameworks works with an internal data representation format, which is implemented in the module dqa.data_representation.data_representation. The main purpose of this data format is a standardized interface to pass data between processing units (“Tasks”, implemented in the dqa.tasks.tasks.Task class).

The data structure follows a hierarchy with multiple levels. For each of these levels, the Task class provides a corresponding method modify_*. By overriding this method, one can implement a task by modifying the data for each unit on the particular level separately.

The following table summarizes the hierarchy of the data format along with the corresponding methods in the task class.

DQA data structure level

Method in dqa.tasks.tasks.Task

dataset_dict: Dict[str, Dataset]

modify_dataset_dict(datasets)

    ↓   str key

    ↓

dataset: Dataset

modify_dataset(dataset)

    ↓   str key (in dataset.machines)

    ↓

machine: Machine

modify_machine(machine)

    ↓   int index (in machine.measurements)

    ↓

measurement: Dict[str, Dict[str, Any]]

modify_measurement(measurement)

    ↓     ↓   'metadata' key

    ↓

    ↓   metadata: Dict[str, Any]

    ↓

    ↓   'data' key

    ↓

data: Dict[str, np.ndarray]

    ↓

    ↓   str key

    ↓

data_row: np.ndarray

modify_data_row(data_row)

The total data processed by DQA is contained in a dictionary of dqa.data_representation.data_representation.Dataset objects whose keys are strings. Each such Dataset contains a dictionary machines mapping string identifiers to dqa.data_representation.data_representation.Machine objects.

The Dataset and Machine class both implement the [] operator, allowing to access the includes machines or measurements, respectively. For example,

Access operator example#
dataset_dict['Data']['M0'][0]['data']['my_data_row']

accesses the data row my_data_row in the measurement with index 0 of the machine M0 in the dataset Data.