Admin History Service

DISTRIBUTED NETWORK ARCHITECTURE
History Service 3.x

Administrator Guide
PROPRIETARY INFORMATION
THE CONTENT OF THIS DOCUMENT IS PROPRIETARY INFORMATION OF INSTEP SOFTWARE, LLC (ISS)
AND SHALL NOT BE RELEASED IN ANY FORM TO ANY THIRD PARTY WITHOUT
THE EXPRESS WRITTEN CONSENT OF ISS
200 W. Jackson Blvd.

www.instepsoftware.com 28th Floor
Version 1.1.1.0 Chicago, IL 60606
T: 312.894.7837 / F: 312.894.7840
User Welcome
Dear eDNA User,
Thank you for selecting the eDNA History Service from InStep Software, LLC to fulfill
your real-time computing needs.
We appreciate your business and are confident you will be fully satisfied. Should you
have any comments on ways we could improve eDNA History Service, we would
appreciate hearing from you.
Thank you once again.
The ISS eDNA Team
How to Contact Us
Please feel free to contact your ISS representative at any time using contact information
provided to you.
General contact information:
Phone (312) 894-7837 InStep Main

(312) 894-7870 Support Line
Fax (312) 894-7840
Internet support@instepsoftware.com
WWW www.instepsoftware.com
US Mail 200 W. Jackson Blvd., 28th Floor

Chicago, IL 60606
Rights to Use
eDNA History Service
The software described in this document is furnished under a license agreement and
shall be used only in accordance with the terms of the agreement.
This document, whether in electronic or hard copy format, shall not, in whole, or in
part, be copied, photocopied, reproduced, translated, or reduced without prior
consent in writing from InStep Software, LLC.
Every effort has been made to ensure the accuracy of this document. However,
InStep Software, LLC makes no warranties with respect to this documentation and
disclaims any implied warranties of merchantability and fitness for a particular
purpose. InStep Software, LLC shall not be liable for any errors or for incidental or
consequential damages in connection with the furnishing, performance, or use of
this manual or the examples herein. The information in this document is subject to
change without notice.
Copyright 2012, InStep Software, LLC and its subsidiaries.

All rights are reserved.
Trademarks:
Windows, Windows 7, Windows 2008, Windows Vista, Windows XP, Windows
2003, Windows 2000, Windows NT, Windows 98, Windows 95, Microsoft Office,
Access, Excel, and Word are registered trademarks of the Microsoft Corporation.
Other product and brand names may be trademarks or registered trademarks of their
respective companies and remain the sole property of their respective manufacturer.
All rights are reserved.
For information regarding duplication and additional use of this document contact:
InStep Software, LLC

200 W. Jackson Blvd., 28th Floor
Chicago, Illinois 60606
(312) 894-7837
eDNA History Service Contents
CONTENTS
1. OVERVIEW.................................................................................................... 1
2. HISTORY SERVICE COMPRESSION TECHNIQUES .......................... 2
2.1. Overview ............................................................................................................. 2
2.2. Archiving eDNA Values ..................................................................................... 4
2.2.1. Status Values ..................................................................................................... 4
2.2.2. Time Values ...................................................................................................... 4
2.2.3. Digital Values ................................................................................................... 4
2.2.4. Floating-Point Values ....................................................................................... 4
3. HISTORY CONFIGURATION .................................................................... 6
3.1. Service Configuration File (History.cfg) ............................................................ 6
3.1.1. Age Out Setup ................................................................................................. 10
3.2. Service Mapping File ........................................................................................ 11
3.2.1. Reference Type ............................................................................................... 11
3.2.2. Reference Number .......................................................................................... 11
3.2.3. Reference Path ................................................................................................ 12
3.2.4. Limit Type ...................................................................................................... 12
3.2.5. Limit Constraint .............................................................................................. 12
4. EDNA HISTORY EXPLORER .................................................................. 14
4.1. Selecting a History Service ............................................................................... 14
4.2. History Service Point List and Point IDs .......................................................... 15
4.3. Point Details ...................................................................................................... 16
4.4. Point Index ........................................................................................................ 17
4.5. Point History ..................................................................................................... 18
4.5.1. Edit Data Record ............................................................................................. 18
4.6. Point Data Files ................................................................................................. 19
4.6.1. High Speed Point History ............................................................................... 19
4.7. All Index Files................................................................................................... 21
4.8. All Data Files .................................................................................................... 22
5. HISTORY BACKUP .................................................................................... 23
5.1. Overview ........................................................................................................... 23
5.2. Database Lockout Method ................................................................................ 23
5.2.1. Backing Up eDNA History ............................................................................. 25
5.3. Stopping and Starting History Using prBoss.exe ............................................. 27
5.4. Backup Recommendations................................................................................ 28
6. MAINTENANCE.......................................................................................... 29
InStep Software, LLC www.instepsoftware.com i

The content of this document is proprietary information of InStep Software, LLC (ISS) and shall not
be released in any form to any third party without the express written consent of ISS.
eDNA History Service Overview
1. Overview
The enterprise Distributed Network Architecture (eDNA) History provides for
long-term, online, real-time data storage, and retrieval.
eDNA History provides for high-ratio compression tailored specifically to real-

time data. It can also span multiple drives, allowing administrators to manage
large amounts of data in standard systems.
The compression techniques used by History achieve high compression ratios and
support decoding on the client end: all of the information needed to decode the
data is included in the data.
InStep Software, LLC www.instepsoftware.com 1

eDNA History Service Compression Techniques
2. History Service Compression Techniques

This section discusses History Service compression techniques.
2.1. Overview
In choosing the compression algorithms used by the eDNA History, a relatively
strict set of requirements is specified. The compression algorithms are required to
achieve high ratios of compression while maintaining the ability to be
uncompressed by separate applications; for example, eDNA Client Programs.
To represent a set of n symbols takes m bits, where n <+2m-1. For example, the
ASCII character set contains 127 ASCII characters (96 printable and 31
unprintable). Seven bits are required to represent the 127 characters (27 – 1 =
127). Another bit is added for parity, resulting in the familiar eight-bit
representation for characters.
Using all eight bits to store a set of characters is convenient; however, it does not
result in the most efficient use of space. A more efficient method for storing the
characters is to create a code where the most common letters are given short
codes, and the less common letters have longer codes. For example:
 If the character ―e‖ occurs more than the other characters in the set, the
fewest number of bits should represent it.
 If ―~‖ hardly occurs, it should be represented by the greatest number of
bits.
This efficient storage method is achieved by first collecting statistics on the data
set, and then by adding the data to a Huffman tree. For example, in the following
set of 1000 characters, the set contains only three different characters as shown:
VALUE COUNT
‗a‘ 950
‗b‘ 21
‗c‘ 29

To build a Huffman decoding tree, take the lowest two counts, and make a node
with the counts combined, with the two values attached on each leaf as shown:
Count = 50
b c
Repeat the process until the tree consumes the entire data set:
Count = 1000
Count = 50
b c
When encoding, branch down the tree to find the target value: branching to the
left codes a ―0‖, branching to the right codes a ―1‖. The result is as follows:
VALUE COUNT CODE BITS

‗a‘ 950 1 1
‗b‘ 21 00 2
‗c‘ 29 01 2
The table shows that representing an ―a‖ requires only one bit – 1. The ―b‖ and
―c‖ require two bits each – 00 and 01, respectively. Storing the 1000 characters as
eight-bit ASCII would have required 1000 bytes, but storing them using the
scheme described would require only 156 bytes as follows:
950 * 1 + 21 * 2 + 29 * 2 = 1050 bits = 132 bytes

plus 24 bytes for the table

Further compression can be achieved by taking advantage of repeated letters and

repeating patterns. For example, if ―a‖ occurs 250 times in a row, this can be
compressed even further by using a code plus repeat count.
This approach is similar to those used by ―zip‖ file compression tools.
2.2. Archiving eDNA Values

The basic data element saved to the archive appears as follows:
Long time 4-bytes

Unsigned short status 2-bytes
Char [16] value 16-bytes
2.2.1. Status Values

For a block of 100 points, there are typically only a handful of different values. It
is effectively encoded as in the Huffman code used in compression.
2.2.2. Time Values

Time is maintained as Universal Coordinated Time (UTC), the number of seconds
since 0:00 January 1, 1970, corrected back to Greenwich Mean Time. The time is
a large value, requiring nearly all of its four bytes to represent. However, because
of the generally regular intervals at which data is collected, the change in time
between two values is generally a small number requiring only a couple of bits to
represent.
The algorithm used to store the time stores the initial time and an average delta
time. Each of the time values are then stored as deviations from the average delta
time. The deviations are Huffman encoded.
2.2.3. Digital Values

Both the status and the value string are used to represent digital values. The
strings are encoded as seven-bit ASCII with all trailing spaces removed.
2.2.4. Floating-Point Values

Floating-point values are maintained in the current value table as a 16-character
string that is easily manipulated by a variety of computers and languages. For
efficiency of compression, they are converted to IEEE-754 floating-points (Intel
style). IEEE-754 floating points are represented using mantissa, exponent, and
sign. Their approximate range is 3.4E +/- 38.
Note that the range is very large; only a small portion is utilized by common data.
Effectively, the exponent can be much smaller, and it is very repetitive for data
from the same measurement device. It is Huffman encoded directly.

IEEE-754 has 23-bit mantissa, rendering floating-point values with nearly eight
decimal digits of precision. Delta encoding looks only at the differences in the
values typically reduces the mantissa. Further, trailing zeros need not be stored.
Further compression is achieved when taking into consideration that most

measurement devices cannot achieve eight decimal digits of precision; a typical
measurement device commonly achieves three decimal digits of precision. Three
decimal digits of precision requires only a 10-bit mantissa (lossy compression).

eDNA History Service History Configuration
3. History Configuration
Two files in History‘s root directory are used for configuration:
 Service configuration file
 History map file
3.1. Service Configuration File (History.cfg)

The service configuration file provides History with all the information required
to start. The service configuration file must be located in the same directory from
which the service process is launched. The administrator is responsible for
generating and maintaining the service configuration file.
Instructions for a service configuration file are given on a single line. These
instructions consist of a keyword, followed by a separator (space or equal sign),
and a list of zero or more parameters. Separators (spaces or commas) are used for
distinguishing parameters. The pound sign (#) keyword indicates a comment. To
get a configuration parameter list, open a DOS window, navigate to the
C:\Systems\History directory, and enter the following command:
History ?
The allowable keywords are as follows (* is required):
KEYWORD DESCRIPTION
AGEOUT_BATCH_SIZE Age Out batch size (20 to 5000 files, default 1000).
AGEOUT_DETAILED_TRACE Displays Age Out detailed trace.
AGEOUT_LEGACY_BA_FILES Required to Age Out legacy BA files on what is now
an HS system.
AGEOUT_LOOP_MSEC_DELAY Age Out loop mSec delay allowing other processes
to run (default 1000).
AGEOUT_REST_TIME Minutes between Age Out sessions (default = -1, do
not run).
AGEOUT_RESTART_COUNT Age Out returns to the oldest files after going this
deep into the file list (min 100, no max, default
10,000).
ALLOW_EARLY_TIMES Allow data with times earlier than two days ago to
be appended.
ALLOW_FUTURE_TIMES Allow data with times beyond two hours in the
future to be appended. This parameter must be
included if the user intends to record simulation data
that is in the future.
ALLOW_MODIFICATION Specifies that modification is allowed (edit, insert,
delete). Required for Redundancy.

KEYWORD (cont) DESCRIPTION (cont)

ALLOW_NON_REGISTRATION_ST Allow History to start when it fails SvcDir
ART registration.
AUTO_REMOVE_DUPLICATES Scan all points to remove duplicate values. This runs
once in the background and removes all duplicate
values from all points in all archive files.
CONSOLIDATE_RAW_FILES Forces a Check/Compression of Raw Data.
DELETE_BAD_FILES Obsolete.
DETAILED_TRACE Displays detailed trace messages.
DISABLE_ARCHIVE_TRACE Do NOT display early archive and on-demand
archive messages.
DO_NOT_SAVE_ALL_VALIDATION Write validation trace to disk only if user hits F9.
ENABLE_BACKUPS Obsolete.
ENABLE_IMPROPER_SHUTDOWN_ If power is cut suddenly, run recovery at the next
RECOVERY start up. Fixes dangling file references at end of data
chains.
ENABLE_POINT_COMPRESSION Enables point-name compression and compressed
point-name requests.
EVENT_LOG_SERVICE The site.service name of the eDNA Event Log
service.
FIREWALL_IP_ADDRESS The firewall IP address.
HS_EARLY_ARCHIVE_MIN HS only. The smallest input buffer record count that
can be sent to archive (500 to 1000, omit to disable).
Early Archive helps spread out disk activity in the
first hour after startup for very fast systems.
INCLUDE_HIGH_SPEED Include support for millisecond data requests.
INX_DATA_RECOVERY Run Index-Oriented Data Recovery on all points.
Saves the name of the last point scanned in
BackgroundProcess. CFG so it can resume after a
shutdown.
INX_DATA_RECOVERY_TRACE Produce a complete trace of Index-Oriented Data
Recovery.
MANUAL_REMOVE_DUPLICATES INX Data Recovery (which is triggered manually)
will include removing duplicate values from
archives.
*MAP_PATH The full path file name for the map file
(C:\Systems\Hist\MapFile.MAP), or a relative path
name (.\YourName.Extension).
MAX_BASIC_ARCH_OPEN_FILES Max Open BA files (500 default, 3000 max).
MAX_BASIC_DATA_OPEN_FILES Max Open BD files (1000 default, 5000 max).
MAX_HS_ARCH_OPEN_FILES Max Open HSA files (500 default, 3000 max).
MAX_HS_DATA_OPEN_FILES Max Open HSD files (1000 default, 5000 max).
MAX_INDEX_OPEN_FILES Max Index files (500 default, 3000 max).
MAX_METHOD_REQUEST_TIME Max time (seconds) to allow a method request to
process (default=5).


MAX_POINT_COUNT Maximum point count, 1000 to 1,000,000 (default:
10,000). Can be adjusted up or down at start up.
Putting in needlessly huge numbers wastes memory.
MAX_POINTS_PER_ARCHIVE Maximum number of points that are allowed to use a
given archive file (100 to 1000, default 1000).
Lowering the default conflicts with Age Out but
might improve data retrieval times for very populous
points.
NO_AUTO_DATA_RECOVERY Do not run data recovery when Point Trace reduces
value count.
NO_BACKGROUND_PROCESS Trace, Validation, Age Out, and file purge are all
disabled.
NO_FULL_VALIDATION Blocks Full File Validation from running during
startup.
NO_MEMORY_MAPPED_BUFFER Do not use memory map to read HSBuffers.DB.
NO_SEARCH_FOR_UNCOMPRESSE Obsolete. Use
D RUN_SEARCH_FOR_UNCOMPRESSED to run.
OPTIMIZE_FLUSHING Minimizes flushing for large point count systems.
Data is maintained in memory buffers for as long as
possible to reduce I/O. Refer to keyword
TARGET_LOOP_FLUSH.
PARTIAL_BACK_TRACE At startup perform a partial backwards trace from
the most recent end of the data chain. Provide a
count of the number of records to trace back. This is
an alternative to Improper Shutdown Recovery.
*POINT_FILE The module point file name with no path
(PointFile.PNT).
PROCESS_PRIORITY Sets the history process priority 1-3. 1 is normal and
default.
READ_BUFFERS Reads buffer count (default: 100). These are used by
clients such as Trend and eDNA Explorer.
RELEASE_VALIDATION_COUNT Momentarily unlocks point during Validation after
this loop count to allow other processes access to the
point.
RUN_SEARCH_FOR_UNCOMPRESS Runs a search for old data to compress. This should
ED be run ONLY if extra BD or HSD files have been
generated (as when the <Arch> directory becomes
unavailable).
SEARCH_FOR_STRANDED Run Trace + Index-Oriented Data Recovery on
every point, one by one. Saves the name of its
current point in BackgroundProcess. CFG so it can
resume after a shutdown.
SECURITY_APPLICATION The name of this service as it is known to Security
(default: HISTORY).
SECURITY_SERVICE The site.service name for the Security service.


*SERVICE The DNA site.service name of this service.
SERVICE_IP_ADDRESS The IP address.
SERVICE_PORT The IP port.
SFS_TOO_FUTURE_HOURS SFS validation too-future cutoff. Default: 5 hours
from current time.
SFS_TOO_OLD_YYYY_MM_DD SFS validation too-old cutoff. Default: 1990.01.01
During Search For Stranded, do not try to recover
data that is older than this.
TARGET_LOOP_FLUSH Considered the ideal count to flush each time
through the flush loop (roughly one second). Can be
used to balance I/O rates. Helps avoid constant
writing to disk, but it can cause a backlog if set too
low or excessive CPU/disk if set too high.
TRACE_ARCHIVE_CRCS Run the algorithm that traces archive file CRC lists
for BA files only.
TRACK_POINT_UPDATES Obsolete.
UNIQUE_TIMES_ONLY Only accept unique (non-duplicate) times when
appending.
VALIDATION_LOOP_DELAY mSec delay inside Point Validation (0 to 1000,
default 2).
WRITE_BUFFERS Write buffer count, size (default: 1000/100). Fast
systems that are under 50,000 points can use the
maximum setting of 1000 (if the box has sufficient
memory). In this case, make
MAX_POINT_COUNT the same value.
WRITE_TO_TRACE_FILE Writes to the trace file beginning at start up (same as
pressing F9).
The following is a typical example of a History Service Configuration File:
SERVICE DNA_DEMO.HISTORY
MAP_PATH MAPFILE.MAP
POINT_FILE POINTFILE.PNT
WRITE_BUFFERS 50000, 1000
READ_BUFFERS 200
MAX_POINT_COUNT 50000
INCLUDE_HIGH_SPEED
NO_FULL_VALIDATION
ALLOW_EARLY_TIMES
ENABLE_POINT_COMPRESSION
ALLOW_MODIFICATION
SERVICE_IP_ADDRESS 172.17.5.60
SERVICE_PORT 6335
OPTIMIZE_FLUSHING
TARGET_LOOP_FLUSH 120
AGEOUT_REST_TIME -1

Note: When tuning History, always set the width of the WRITE_BUFFERS (first
number) to match MAX_POINT_COUNT. If no MAX_POINT_COUNT is
present, set the width of the WRITE_BUFFERS to 10000. The width and depth
(first and second numbers) are multipliers and directly affect the memory
footprint of History.
3.1.1. Age Out Setup

History‘s Age Out routine deletes only complete files and never individual
records. It starts with the oldest files and looks at them as a batch or group. The
younger files in a batch become anchors for the data chains; the older files can
then be deleted. If a file contains any data that is still needed for a point, then the
entire file cannot be aged out. As a result, points with an Age Out setting of 9999
will interfere with aging out files for all points.
For a system with 5,000 low-speed points, the following settings would be
acceptable:
AGEOUT_REST_TIME 360
AGEOUT_BATCH_SIZE 100
AGEOUT_RESTART_COUNT 2000
These settings will insert six hours between Age Out runs, History will look at
100 files at a time, and it will return to the end of the file list after a total of 2,000
files have been examined (in roughly 20 runs). If half of the files in a batch are to
be deleted, then, approximately, 50 files would be deleted every six hours. With
1,000,000 values per HSA or BA file, this is 50 million values deleted every six
hours (or approximately 2,300 values per second). This would keep pace with
5,000 points updating once every five seconds.
A high-speed system with 5,000 points might need:
AGEOUT_REST_TIME 5
With 350,000 points, a larger number of files must be examined in each Age Out
run:
AGEOUT_REST_TIME 5

If the user has a high-speed system (as shown by INCLUDE_HIGH_SPEED in

the CFG file) that includes legacy low-speed archives (BA files), then include:
AGEOUT_LEGACY_BA_FILES
To see diagnostic messages on the F5 screen, include:
AGEOUT_DETAILED_TRACE
Note that the Age Out Days for a given point cannot be set in History; this value
needs to be set in the real-time service. For example, setting the entry within
UnivServ will propagate it to its CMCFG and, subsequently, propagate it to
History
3.2. Service Mapping File

Map files are text-based files that can be viewed or edited with almost any editor.
The History mapping file‘s primary purpose is to direct where historical data from
the History service is located.
A History map file requires the correct format in order to be usable by the
associated History service. Each History service is assigned a map file. This map
file must be located in the same folder/directory as the History service executable
file. The map file should always be named ―mapfile.map‖.
Each line in the map file contains from three to five tokens that have specific
meanings and are read at start-up, including the following:
 Reference Type
 Reference Number
 Reference Path
 Limit Type
 Limit Constraint
The following is an example of a line that uses all of the possible tokens:
1, 1, D:\eDNA\History\Data, 3, 5000
3.2.1. Reference Type

The Reference Type is a placeholder and must always be set to ―1‖.
3.2.2. Reference Number

Reference Numbers define the order of the storage locations. The first Reference
Number had to be set to ―0‖. Reference Number ―0‖ cannot include a Limit Type
or a Limit Constraint.
Reference Numbers should be listed in contiguous order. The numerical sequence

of Reference Numbers can be non-sequential if desired, though the recommended
practice is to follow a sequential and contiguous Reference Number
configuration.
The total number of Reference Numbers allowed is 100 (0-99). Reference

Numbers greater than 99 are ignored.
3.2.3. Reference Path

The Reference Path is either a fully qualified path or a relative path to a disk
location that is accessible to the History service. History will not create the
directory identified in the Reference Path. Each directory identified in the
Reference Path must be created and available before History can start up.
3.2.4. Limit Type

Inclusion of a Limit Type control identifies that a Limit Constraint will be
included in the lines. Default limits are applied if no Limit Type is provided. A
Limit Type is always a ―3‖. The use of any other limit type, other than no limit
type, will result in erroneous assignment of historical data.
3.2.5. Limit Constraint

A Limit Constraint should always follow a Limit Type. The Limit Constraint
identifies the amount of free disk space that must be available on the disk drive
that is associated with the Reference Path.
Limit Type and Limit Constraint controls are not used and do not apply to lines
that include the ―0‖ Reference Number. The data at that location is uncompressed,
keeps changing, and will cycle around a fixed data size once initial startup of the
History service is completed.
Lines that do not include a Limit Type and Limit Constraint are assigned default
size limits. The default size limit for data storage for this type of line is
approximately 95% of the storage space available. When the 95% level is
reached, data will be directed to the Reference Path location identified by the next
sequential Reference Number.

When a Limit Constraint is used, the constraint value represents the number of
megabytes of space that must be available at the identified storage location in
order for historical data to be written to the location. Consider the following
sample line:
1, 1, d:\eDNA\History\Data, 3, 5000
This specifies that 5000 MB (5.0 GB) of disk space must be available before
historical data can be written to the specified path on the volume labeled ―D:\‖.
The eDNA administrator must ensure that adequate storage space is always
available for storage of historical data. If no storage location is identified in
mapfile.map, or if there is less than 5% space available on the identified storage
device, historical data will be sent to the next location identified as having space
available. This can result in decreased performance of History as well as
decreased performance of the tools that access and make historical data available
for use throughout the system.
Example:
The following is an example of the contents in a mapfile.map, where archiving
moves to the ―e‖ drive when only 4.1 GB of free space is available on the ―d‖
drive:
1, 0, d:\eDNAhist\hist
1, 1, d:\eDNAhist\arch, 3, 4100
1, 2, e:\eDNAhist\arch

eDNA History Service eDNA History Explorer
4. eDNA History Explorer

History has one primary administrator configuration tool—eDNA Explorer
History—that provides administrators with a view of the database and all the data
contained in it.
4.1. Selecting a History Service

When eDNA Explorer is started and History is selected, it will display a list of
available History services as shown in the following example:
Services displayed in light gray are unavailable and cannot be selected. Click on
an available History service to select it. Right-click select Load Points to load the
service‘s available points.

4.2. History Service Point List and Point IDs

All of the History service‘s points are listed in a standard tree format. Click on a
Point ID to view information about it in the General, Point Index, Point Data
Files, All Index Files, and All Data Files tabs.
To delete the point, select the point, and right-click select Delete. Confirm the
deletion in the pop-up dialog. (Deleting a point requires that the
ALLOW_MODIFICATION keyword be present in the History configuration
file.)

4.3. Point Details

Point details are displayed in the General tab as shown in the following example:
This tab lists the point name, point status, and data availability, among other point
details. If data is available, ―HAS DATA‖ will display. If the point has no
available data, ―EMPTY‖ will display.
The Earliest Time and Latest Time fields indicate the earliest and latest times
for the point in History. The Hash value is an internal value indicating the hash
value assigned by History. The Hash value provides a quick search algorithm for
finding points in the list. The Record Count indicates the number of readings for
this point.
Administrators can configure the Age Out Days and view the Bits Precision
field. Age Out Days indicates how long History is to store the data. Bits
Precision defaults to 23 for lossless compression.
The user can trigger a QA audit of a point in History by clicking Force Trace.
This can be a time intensive operation and can cause some queries against the data
to take longer than expected. The information in the Indexes section displays the
location of the first and last index for the point.

4.4. Point Index

The Point Index tab references data records and also contains references to the
previous and next indexes. To navigate through indexes, click the Previous and
Next arrows in the Previous File Reference and Next File Reference groups. To
view a data record, double-click on the data record. To navigate to a specific
timestamp, click Jump to Time after entering the target time in the edit box next
to the button.

4.5. Point History

Point History data can be accessed from the Point Index tab by double-clicking
on a block of data. The Point History dialog displays archived and non-archived
data as well as high-speed and low-speed data.
The History Points dialog displays its reference in the File Reference group and
a reference back to its Index File Reference. It also displays the Previous File
Reference as well as the Next File Reference.
4.5.1. Edit Data Record

By selecting and double-clicking on a Point History data point, the Edit Data
Record dialog will display site.service.point name and the time, value, and status.
These values can then be edited and re-entered into history by clicking Update.
(Editing a data record requires that the ALLOW_MODIFICATION keyword be
present in the History configuration file.)

4.6. Point Data Files

The Point Data Files tab allows the user to view just those data files that contain
data for the selected point. In the example provided, point B0000002 has data in
five files, which are shown on the Data File List. File HSA00000002.DAT has
been selected, so its Data File Reference List on the right has been populated. In
the selected file, there are 10 records for the selected point. Record number 0632
has been highlighted in the example. Its CRC is 44588, its offset in the files is
2,007,240 bytes, and its compressed size is 3,640 bytes.
Clicking on the printer icon produces a comprehensive listing for this point. In the
example shown, the ―Index‖ option was selected and lists all of the Index File
records where this particular point appears.
4.6.1. High Speed Point History

Double-clicking a record in the Data File Reference List opens the High Speed
Point History dialogue. The High Speed Point History dialogue shows that the
Current Data File Reference is the highlighted item:
1:HSA00000002.DAT:632. The ―1:‖ at the head of the file name indicates the file
is in MapFile location 1. The offset and size are repeated in the dialogue. The data
in the list at the bottom, starting with value zero, has a date of March 10, 2011, at
16:04:00 and 942 milliseconds. The value is 20.524139 and the status OK, which
is 0x0003 in hexadecimal.

Clicking on the ―<<‖ or ―>>‖ button moves to the Previous Data File Reference
(1:HSA00000002.DAT:540) or the Next Data File Reference
(1:HSA00000002.DAT:815), respectively.

4.7. All Index Files

The All Index Files tab displays all the index files currently contained in History.
To select an individual index file from the Index File Reference List to view,
double-click on the file. This will display the Point Index tab.

4.8. All Data Files

The All Data Files tab displays all of the data files in History. Select a file in the
Data File List to view its internal records in the Data File Reference List.
The Data File Reference List displays record number, hash value, hex hash
value, file offset, and record size. The record size is static in raw data files but is
variable in archived files (different blocks of data compress differently).

eDNA History Service History Backup
5. History Backup
This section discusses backing up the History service.
5.1. Overview
Backing up highly dynamic databases poses challenges to most backup systems.
The systems either yank control of files out from under the database or make
inconsistent backups because the files continue to change. This section describes a
simple method administrators can use to facilitate backups.
The best way to ensure that History‘s database is flushed and unchanging is to
stop History. When History stops, it forces all outstanding data and all cached
files to flush. History then unlocks the files by closing them.
Stopping History does not cause a loss of data if the eDNA system is configured
properly. The real-time service sending data to History will cache the data in
memory and to disk while it is unable to communicate with History. It is
important to configure the real-time service to cache and to provide enough room
for the data to be cached.
Most eDNA real-time services provide for caching of data when its associated
History is inaccessible. Typically, an administrator adds the
MAX_HIST_QUEUE_SIZE configuration file parameter. For more information
on real-time service caching, refer to the specific documentation for that real-time
service.
The recommended method to stop and start History and other eDNA services is
through eDNA Explorer BOSS.
5.2. Database Lockout Method

Database lockout is intended to support third-party backup systems, though it
could be used elsewhere by administrators and users. Database lockout is
managed internally by History. History is instructed to put itself into—or out of—
a state of lockout through an eDNA request. The details of the request are not
described in this document but can be found in the Easy eDNA Application
Programmer’s Interface (API) Developer’s Guide.
There are two levels of database lockout: full lockout and read-only lockout. In
full lockout, History will not allow any kind of access to the database. Users can
neither read nor modify the database. In read-only lockout, History blocks updates
but allows reads. Most backup systems can coordinate with History in read-only
lockout; however, if either the backup system or History show signs of problems,
full lockout should be used.

Once the backup is accomplished, either of the lockout modes can be cancelled
using a restore command. The lockouts have a safety feature that automatically
cancels the lockouts after a specified period of time. This is known as the lockout
timeout. Timeouts are specified in seconds and can be set to infinite.
A console mode utility—LockHistory.exe—is provided to facilitate applying a

lockout to one or more History. Usage instructions for LockHistory are provided
when running the program without command line parameters. The usage
instructions appear as follows:
READONLY and LOCKDOWN commands cause History to lock its database

and block data input. RELEASE causes History to unlock its database and resume
normal operations. CHECK will output the current status of History‘s lockdown
mode.
The optional timeout applies only to READONLY and LOCKDOWN modes. It is

specified in seconds. The minimum timeout is 60 seconds. The default timeout is
three-hours.
The optional wait specifies how long to wait for a History to flush its buffers to
disk before locking. The default is set to 60 seconds, but this parameter can
specify a wait of up to 10 minutes.

5.2.1. Backing Up eDNA History

The eDNA History database contains all of the result information generated from
the eDNA real-time services. This History should be backed up as part of a
regularly scheduled backup process. The History holds open file locks for
improved performance. These files can be closed using the LockHistory utility so
that they may be backed up. While these files are closed for backup, the eDNA
services will continue to run as normal and results will be cached until the History
service returns to an unlocked state.
Complete the following steps to back up the eDNA History:
1. Prepare the eDNA real-time services to cache when History is locked.

When setting the RTS to cache, use the following formula:
Words: - Number of Points within RTS, divided by the scan frequency,

multiplied by 32 bytes, multiplied by the number of seconds each hour the
embedded History will be locked.
Calculation: - [Number of points in real-time service] / [Scan Frequency
of points] * 32 Bytes * (3600 * [Number of hours the History will be
locked] )
eDNA uses several data sources—both internal and external—to store data
to the History. In most cases, a real-time service (for example,
Calculation) is used to store the results to History. Another form or
method of storing data can be the eDNA specialty services (such as
Universal, OPCRTS, and DNAMDBus). All eDNA RTS services must
have the appropriate cache settings to write the data to disk during the lock
History period.
Note: Each RTS has its cache keyword to configure for service caching.
Cache for the Source eDNA Services:

Calculation Service (if applicable)
Keyword must be added to the Calculation Service Configuration file
(Calcserv.cfg).
Keyword: - MAX_HIST_QUEUE_SIZE [add calculated cache]
Universal Service
Keyword must be added to the Universal Service Configuration file
(univserv.cfg).
Keyword: - HIST_CACHE_MB [add calculated cache]
OPC Real Time Service (if applicable)
Keyword must be added to the OPC Real-Time Service Configuration file
(OPCRTS.cfg).
Keyword: - MAX_HIST_QUEUE_SIZE [add calculated cache]

The cache configuration sizes are in megabytes. All eDNA real-time

services will have to be restarted after adding the keywords for the cache
during the lock History period.
2. Stop and start the real-time services through the eDNA Explorer BOSS
Control.
A. Select the ―BOSS‖ service from the list and click OK.
B. Stop all source real-time services ([SITE].[RTS Service]). This
should include all real-time services (Calc, Universal, and so forth)
writing/storing data to the eDNA History.
C. Start all source real-time services after adding the Cache settings
from step 1, page 25.
Note: eDNA administrator privileges are required to stop and start eDNA
real-time services.
3. Lock the History.

Besides API commands, a console mode utility—LockHistory.exe—is
provided that facilitates applying a lockout to one or more Historians.
Usage instructions for LockHistory are provided when running the
program without command line parameters and appear as follows:
The maximum time allowed for the backup can be specified when issuing
the lock History command.

4. Execute the backup from the backup device or software.

The backup software should include the ―Hist‖ and ―Arch‖ directories
(.\History\Hist & Arch).
―Hist‖ - This folder contains all basic and high-speed map files, as well as,
the POINTFILE.PNT and HS/HD.DAT files. These files are contained
within the History service directory (Default: D:\Systems\History). Many
eDNA Systems services are installed on slightly different drives.
―arch‖ – This folder contains all high- and low-speed data index and data
files.
Note: Be sure to specify the folders mentioned in this step to back up the
eDNA Archive History files. The ―Hist‖ and ―Arch‖ folders should be
included for backup from the backup software.
5. Release eDNA History.

Release the History by using API commands or by using the LockHistory
executable. There is also a lockhistory.bat file that can be used to automate
the process.
6. Verify all cached data from the eDNA real-time services is stored in the
History.
Confirm that the data for the time the eDNA History was locked by
trending several of the eDNA RTS points using eDNA Trend.
InStep Software strongly recommends using the xcopy utility for users planning
to copy all historical files to a ―backup‖ folder before running the ntback utility
software.
5.3. Stopping and Starting History Using prBoss.exe

The ―BOSS‖ is a term frequently used among network administrators who deal
with eDNA. The term is an acronym describing an eDNA network control
facility: Boot, Operate, and Supervise Services. The BOSS is a powerful and
versatile tool for facilitating network operations from remote locations. Once a
remote service has been configured to operate under and communicate with the
BOSS, that service can be stopped and started remotely.
In order to use the recommended tools for backing up History, the History must
be running under a BOSS.

The prBOSS.exe application is capable of stopping and starting eDNA

applications through the BOSS. The format for calling prBOSS.exe is as follows:
prBOSS start|stop -nb<BOSS Service> -ss<eDNA

Service Site>
-sn<eDNA Service Name>
…where the stop and start specify the control command, –nb precedes the
name of the boss, in SITE.SERVICE format, –ss precedes the History‘s site,
and –sn precedes the History‘s name.
When attempting to stop MY.HISTORY running under MY.BOSS, issue the

following command:
prBOSS stop -nbMY.BOSS –ssMY -snHISTORY
To restart the History, issue the following command:
prBOSS start -nbMY.BOSS –ssMY -snHISTORY
5.4. Backup Recommendations

Different organizations have different backup philosophies. The purpose of this
section is to provide administrators with some place to start rather than a fixed,
inflexible routine.
The best way to know which directories to back up is to look in the Map File
(typically named MAPFILE.MAP). The Map File defines where the History is to
store its database. Be sure to back up all of the directories specified in the Map
File. If saving the configuration information, back up the executable directory as
well.
A typical periodicity for performing backups is to perform a full backup once a

week and an incremental/differential backup once a day. A full backup copies all
of the files in the database. An incremental/differential backup copies only files
that have changed since the previous backup. Full backups are performed in order
to reduce the number of backups that must be retrieved to recreate the existing
database.
Backups should occur at times when it is unlikely users will be viewing data.
History might allow viewing, but it will not allow user editing.

eDNA History Service Maintenance
6. Maintenance
eDNA History service requires no formal, routine maintenance. As with all eDNA
services, it is recommended that the administrator periodically check the service
screens for error messages and verify that the alarms are current.
An administrator should also retain copies of the application executables, service

configuration, and database files.


Admin History Service

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Admin History Service

Uploaded by

Copyright:

Available Formats

DISTRIBUTED NETWORK ARCHITECTURE

History Service 3.x

200 W. Jackson Blvd.

Dear eDNA User,

Thank you once again.

The ISS eDNA Team

General contact information:

Phone (312) 894-7837 InStep Main

Fax (312) 894-7840

US Mail 200 W. Jackson Blvd., 28th Floor

Copyright 2012, InStep Software, LLC and its subsidiaries.

InStep Software, LLC

InStep Software, LLC www.instepsoftware.com i

eDNA History provides for high-ratio compression tailored specifically to real-

InStep Software, LLC www.instepsoftware.com 1

2. History Service Compression Techniques

InStep Software, LLC www.instepsoftware.com 2

VALUE COUNT CODE BITS

950 * 1 + 21 * 2 + 29 * 2 = 1050 bits = 132 bytes

InStep Software, LLC www.instepsoftware.com 3

Further compression can be achieved by taking advantage of repeated letters and

This approach is similar to those used by ―zip‖ file compression tools.

2.2. Archiving eDNA Values

Long time 4-bytes

2.2.1. Status Values

2.2.2. Time Values

2.2.3. Digital Values

2.2.4. Floating-Point Values

InStep Software, LLC www.instepsoftware.com 4

Further compression is achieved when taking into consideration that most

InStep Software, LLC www.instepsoftware.com 5

3.1. Service Configuration File (History.cfg)

The allowable keywords are as follows (* is required):

InStep Software, LLC www.instepsoftware.com 6

KEYWORD (cont) DESCRIPTION (cont)

InStep Software, LLC www.instepsoftware.com 7

KEYWORD (cont) DESCRIPTION (cont)

InStep Software, LLC www.instepsoftware.com 8

KEYWORD (cont) DESCRIPTION (cont)

The following is a typical example of a History Service Configuration File:

InStep Software, LLC www.instepsoftware.com 9

3.1.1. Age Out Setup

A high-speed system with 5,000 points might need:

InStep Software, LLC www.instepsoftware.com 10

If the user has a high-speed system (as shown by INCLUDE_HIGH_SPEED in

To see diagnostic messages on the F5 screen, include:

3.2. Service Mapping File

3.2.1. Reference Type

3.2.2. Reference Number

Reference Numbers should be listed in contiguous order. The numerical sequence

The total number of Reference Numbers allowed is 100 (0-99). Reference

3.2.3. Reference Path

3.2.4. Limit Type

3.2.5. Limit Constraint

InStep Software, LLC www.instepsoftware.com 12

InStep Software, LLC www.instepsoftware.com 13

4. eDNA History Explorer

4.1. Selecting a History Service

InStep Software, LLC www.instepsoftware.com 14

4.2. History Service Point List and Point IDs

InStep Software, LLC www.instepsoftware.com 15

4.3. Point Details

InStep Software, LLC www.instepsoftware.com 16

4.4. Point Index