Removing 'Garbage' Data from your RAW

Files
By Joe Burnett

Recently, here in HYPACK Tech Support, we have had numerous files come in about, well,
'garbage' data that was recorded in the Raw files.
In this case, the ‘garbage’ was corrupt, garbled ASCII strings instead of the standard
HYPACK® MSG data strings. MSG strings are used only for troubleshooting. They are not
used by the processing programs so they can be removed without losing data.
Here are some steps that you can take to remove this 'garbage' data and, hopefully, a
solution to keep it from happening again.
First, you will need to find the location of the 'garbage' data in your file. This can done fairly
quickly, by loading the file in the SINGLE BEAM EDITOR (SBMax), and looking at the Time
stamp where the editor stopped reading the data.
1. Start SBMAX and load the file that is not displaying all of its data.
FIGURE 1. Finding the Time of the Last Good Data

2. Place the cursor on the last data point in the Profile window.
3. Note the time of this data point. In Figure 1, the time of the last data point displayed, is
10:53:00.91.
4. Convert the time to ‘Seconds Past Midnight’—the format in which HYPACK® time
stamps all of its data.
• Use the following equation:
(3600 x Hours ) + (60 x Minutes) + (Seconds)
In this example:
(3600 x 10) + (60 x 53) + (00.91) = 39180.91 Seconds Past Midnight

July / 2012 1
 • Alternatively, you could setup the Spreadsheet to automatically display the time
in 'Seconds Past Midnight'.
i. Click [Export].
ii. Select the 'Alternate Time Format: Seconds Past Midnight'’ option and click
[OK].
FIGURE 2. Setting the Time Format

iii. Reload the raw data in SBMAX to

apply the change in the time display.

Note: The time setting will now default

to 'Seconds Past Midnight' until
you change the export option
again.

FIGURE 3. SBMax Displays Time

in Seconds Past Midnight

5. Open the ASCII raw file in

any text editing program
(Notepad, Wordpad,
WORD, TextPad, etc) and
search for the erroneous
or 'garbage' data.

Note: In this example, I used

Textpad to locate the
'garbage' data, and
modify the Raw file.

a. Start your text editor

and open the raw file.

2
 FIGURE 4. Sample Raw Data File in TextPad

b. Search for the time stamp of the last data displayed in SBMax (39180.91). You
should find the first time the 'garbage' data appeared in the file, shortly after that time.
FIGURE 5. Searching for the Bad Data in TextPad

c. Scroll-down thru the file until you locate the 'garbage' data.
HCP 1 39182.959 0.100 1.230 2.270
HCP 1 39182.985 0.110 1.100 2.260
MSG 2 39181.924
uu*Š)¢ª‚’¢r‚‚bš²‚’r‚š¢š²šºbr±ÁºªšÊršÊ‚Â’šÊ‚bº±ÑbÊb‚rÊbj‚r’Ê‚bj±µšªrÂ’Êbj±År
‚b‚‚’ÊR²5)":Ai ±Å¢ª‚’¢r‚’ºb’¢b‚ºb’‚Š’b‚‚b‚‚RªªjRü$GPGGA,145025.00,3602.08286
273,N,07539.39152383,W,4,9,0.9,-0.185,M,-35.829,M,1.0,0029*68
EC1 0 39183.010 35.900
HCP 1 39183.011 0.110 0.980 2.250
We can see that the 'MSG 2' records have the 'garbage' data in them.
d. Use the 'Find' feature to locate and mark all of the MSG 2 records.

July / 2012 3
 FIGURE 6. Marking all MSG 2 Records in TextEditor

e. Select EDIT-DELETE-BOOKMARKED LINES. All of the 'MSG 2' records will now be
deleted from the file.
FIGURE 7. Deleting the Marked MSG
2 Records

f. Save the modified file

using the FILE-SAVE AS
option.

Beware! Do NOT save the

file to its original name. If
you do, it will permanently
overwrite the original RAW
data file.

4
 FIGURE 8. Saving the Raw File without the MSG 2 Records

6. Load the 'modified' raw file into SBMax to display and edit all of the data that was
collected in that file.
f

July / 2012 5
 FIGURE 9. Modified Raw Data in SBMax

WHAT CAUSED THIS 'GARBAGE' DATA TO APPEAR IN THE

RAW FILES ??
Most likely, bad data is caused by a baud rate too low for the amount of data output by a
particular device. If your device was set to a 9600 baud rate, try increasing it to 19200 and
see if the problem persists. If that doesn't solve it, try 38400.

Note: It could simply be that your device, is outputting 'garbage' data, randomly and
sporadically.