The integrity of the raw ShapeArray data collected from a Campbell Scientific CR series data logger is very important for long term monitoring and analysis. Many things can affect this data which can lower its integrity and skew deformation results once it is converted and visualized. Measurand provides the DataChecker utility to validate the integrity of raw data, filter invalid readings from the raw data files, and visualize changes in the raw data returned from the ShapeArray. This guide will cover the key methods for validating and filtering raw ShapeArray data files with DataChecker. 

Step 1 - Open a Raw ShapeArray Data File

In SAASuite 3.0 and higher, the DataChecker utility is accessed from the Advanced menu by selecting the Check Logger Data option. 

Figure 1: DataChecker is accessed from the Advanced menu by selecting Check Logger Data

Once you have launched SAACR_DataChecker, click on the Open Data File(s) button to select a raw data file. Select one or more SAA#_DATA.dat files and click the Open button.

Figure 2: SAACR_DataChecker

Step 2 - Examine Raw Data Counts

ShapeArray data is recorded as a number of counts. Valid ranges for these counts are between 10000 and 60000 for the acceleration sensors. Values outside of this range indicate a bad or zero reading from the instrument. 

NOTE: Valid raw data counts from magnetometer sensors may exceed the range defined above for acceleration sensors. Raw data for segments with magnetometers typically appear as horizontal black lines. You can disable the visualization of magnetometer data by selecting the Hide Magnetometer Data checkbox.

When you first open a raw data file in SAACR_DataChecker, the counts for each segment's X-, Y-, and Z-axis readings within the data set are graphically presented. Each reading is presented as a coloured block matching the legend shown in the right side of the window. Any black blocks within the graph represent a reading outside of the valid range for data counts. This could indicate a problem with a sensor within a segment. 

Figure 3: A raw ShapeArray data file open in SAACR_DataChecker

Vertical black lines typically indicate a bad or zero reading from the ShapeArray for specific samples. This is often caused by the power source not providing enough voltage to sufficiently power the instrument when the reading was being recorded by the data logger. 

NOTE: Vertical grey lines (absence of coloured blocks) indicate missing data samples. SAACR_DataChecker will scale the graph of displayed readings based on the time interval over which the data was collected. A grey area occurs when there is a gap without samples that exceeds 10 times the average time between samples. 

Figure 4: A raw ShapeArray data file open in SAACR_DataChecker - Note the grey and black vertical bars indicating missing or bad samples

Step 3 - Filter Bad Data Samples

If you encounter vertical black lines in your raw data, as shown in the figure above, it is a good idea to filter those results from the raw data file before you process it with SAACR_raw2data. To do this, click on the Filter button near the top, center of the window. This will open the dialog shown in the figure below. In this dialog, click on the Remove Bad / Invalid Data button. Once the bad data has been removed, you can save the data file by clicking on the Save Data File(s) button.

Figure 5: The Filter Data dialog

Step 4 - Examine Raw Data Counts in Differential Mode

Select the Differential checkbox near the top right side of the SAACR_DataChecker window to enable Differential mode. You will be prompted to select a maximum range for the differential data plot. The range of counts specified will define the colour scale for the plotted data in this mode. Typically a setting of 100 counts is appropriate for troubleshooting data integrity.

Figure 6: When switching to Differential mode, you will be prompted to configure the differential range in counts

In Differential mode, SAACR_DataChecker displays coloured blocks for each data reading that represent the difference between the counts recorded in the initial data reading and each other reading in the data set. It is primarily used to identify shifts in sensor output that can sometimes occur in MEMS acceleration sensors. A sensor shift is identified by a sudden change in the counts returned by a sensor in a single axis. The count changes that occur in sensor shifts are consistent in magnitude and typically remain stable at a new level after they occur. Graphically, they appear as a sudden change in the colour at a specific elevation, occurring only in a single axis. Once identified, a single-axis sensor shift of 40 counts or more can typically be corrected with SAACR_raw2data's Bias Shift adjustment (formerly known as eXYZ adjustment).

Figure 7: A raw data file open in Differential mode - The sudden change in counts that appears in only the X-axis at segment 22 indicates a sensor shift

Step 5 - Optional, Examine Temperature Data

SAACR_DataChecker can also visualize the temperature data recorded in raw ShapeArray data files. To do this, select the Temperature checkbox near the top, center of the window. You will then be prompted to confirm the minimum and maximum temperatures. The values selected will determine the colour scale used to visualize the temperature data. By default, SAACR_DataChecker will suggest the minimum and maximum temperatures based on temperatures found in the data file. Click the OK button to save the range and visualize the temperature data.

Figure 8: Configuring the minimum and maximum temperatures will determine the colour scale used to visualize the temperature data

Figure 9: Visualized temperature data in SAACR_DataChecker

Step 6 - Optional, Select Data to Include or Exclude

You can also filter a raw data file to include or exclude records by manually selecting data points or specifying a range of time to include or exclude. This is done by clicking on the Filter button and then selecting which method you would like to use to choose the data for inclusion or exclusion.

Figure 10. Choose data for inclusion or exclusion

If you select the Choose Individual Points option, you will be presented with a listing of records with checkboxes for selecting the records you wish to keep in the data file.

WARNING: For data files with a large number of records, selecting individual points is a very resource intensive operation. Performance of the program will degrade for very large data sets filtered in this manner.

Figure 11. Choosing individual records to include

If you select the Choose Range of Data to Include or Choose Range of Data to Exclude options, you will be presented with options for selecting the beginning and end of the range you wish to include or exclude.

Figure 12. Selecting a range of data to include