In very long running ShapeArray™ monitoring projects or projects with very fast collection intervals, the ever increasing size of the data set will cause the execution time and CPU and memory usage of the Raw2Data conversion software to increase. Each execution of Raw2Data from the command line will only convert new data; however, each execution must load up and parse the multi_saa_allcart.mat file in order to determine which readings are new and which have previously been converted. The only way to reduce the resource usage and time required for performing a conversion is to make the multi_saa_allcart.mat file smaller. 

To make the multi_saa_allcart.mat files smaller, the raw data files containing the historical readings must be sparsed (filtered down to contain fewer readings) and then a Reset conversion done to regenerate the multi_saa_allcart.mat file with fewer readings. During this process, it is vitally important that the first reading (by earliest timestamp in the file) be maintained in order for consistent results to be produced for the Reset conversion data and any prior conversions. This is due to how software adjustments, such as cyclical adjustment for vertical SAAVs, are implemented. It is also important for ShapeArrays that have been installed in an arc for convergence monitoring.

Sparsing the data files can most easily be done with our Data Checker software that is included in the SAASuite. The steps outlined below will help you to reduce the size of your raw data files with the Data Checker utility and ultimately reduce the conversion time and resource usage of the Raw2Data software.

Step 1 - Disable Existing Automations

Before working with any of ShapeArray project files, any automated conversions should be temporarily disabled to ensure Raw2Data is not triggered unexpectedly during these operations.

Step 2 - Sparse the Raw Data Files

Measurand's Data Checker utility can be used to sparse raw ShapeArray data (.DAT) files. Data Checker can be accessed from the SAASuite by selecting the Advanced menu and then the Check Logger Data option. 

Figure 1. Data Checker can be accessed from the Advanced menu in SAASuite

In the Data Checker window, click on the Open Data File(s) button, navigate to, and select your ShapeArray raw data (.DAT) file. If the file is sufficiently large, then you will be prompted to reduce the file size and save a backup. Click on the Yes button to continue.

Figure 2. Click Yes to reduce the file size when prompted after opening a large raw data file in Data Checker

You will then be prompted to select the interval for which samples in the raw data file will be saved. By default, this will be every 10th sample; however, any interval from 1 to 100 can be selected. 

NOTE: Data Checker will automatically keep the first and last reading in the data file regardless of the sampling interval selected. This is extremely important for conversions using software adjustments that rely on the first reading in the data set, such as cyclical adjustment for vertical SAAVs, and for ShapeArrays that have been installed in an arc for convergence monitoring.

Figure 3. Select the sampling interval to continue

This will save a copy of the existing .DAT file with a .BAK extension and then save the .DAT file with only every Nth record included (where N is the sampling rate selected). Once the sparsing process has completed and the remaining data is loaded into Data Checker, you will see both the backup file and the sparsed file in the project folder. 

Figure 4. When the file has been sparsed, the remaining data is displayed in Data Checker

Repeat this step for any raw data (.DAT) files containing ShapeArray data readings in the project folder.

Step 3 - Run a Reset Conversion with Raw2Data

After the raw data files have been sparsed, a Reset conversion with Raw2Data must be executed while keeping all of the same settings as the prior conversion. Those settings should be loaded and selected automatically from the pref_project.txt file in the project folder (created when the last Reset conversion was executed). Clicking through the Reset conversion will create smaller multi_saa_allcart.mat files containing only the remaining readings. 

Step 4 - Re-Enable Automated Conversions

From this point, the automation of command lines calls to the Raw2Data software can be resumed. Subsequent conversions should take much less time and require fewer computer hardware resources. 

NOTE: This entire process will need to be repeated again periodically as more data is collected and converted in order to keep the conversion times and resource usage low.