Compiling and Pre-Processing Geospatial Data Sets

Automated using Python

Benjamin Felton, EIT

MS Graduate Student
University of Virginia
Charlottesville, VA
brf2er@virginia.edu

Abstract—Data management is an important aspect of any project. identification techniques like ground surveying, which can be
This process can include the maintenance and formatting of raw costly and time consuming. This has caused a push in research
data. This study focuses on developing a Python scripted tool to towards using remote sensing techniques for wetlands
automate the acquisition and pre-processing steps of a project. delineation.
The particular project of interest involves the delineation of
wetlands using the following ancillary data: SSURGO, NWI, and
NHD. This tool utilizes the appropriate libraries to download the Remote sensing leverages technological hardware,
zipped data from federal FTP sites, extract the data into an particularly aerial sensors, to observe and record data about
appropriate folder structure, process this data by projecting and Earth’s surface, oceans, and atmosphere by means of
clipping it to a desired datum and spatial scope, and create a electromagnetic radiation. From remote sensing we are able to
compiled geodatabase of these geospatial data sets for utilization obtain useful data sets for wetlands delineation, like Digital
in ArcGIS. Elevation Models (DEM) and Light Detection and Ranging
(LiDAR), Multispectral Satellite Imagery, Radar, and
Keywords—data management; SSURGO; NWI; NHD; FTP; Normalized Difference Vegetation Index (NDVI). Other
ArcGIS
ancillary data sets are also required for the wetlands delineation
I. INTRODUCTION that are not necessarily obtained by means of remote sensing.
Some of these data sets include the National Wetlands
Civil Engineering encompasses a wide range of fields;
Inventory (NWI), Soil Survey Geographic (SSURGO) soils
spanning from roadway design in transportation to storm water
data, and National Hydrography Dataset (NHD).
control in hydrology. Irrespective of the particular field within
civil engineering, there will most likely be concerns about the
Of this data, there are a variety of formats utilized to
environmental impact within the design and planning phase of
represent this geospatial information, ranging from polylines
a particular project. There are multiple considerations to ensure
and polygons to rasters. Not only are there different formats,
that the ecology is maintained and projects are as non-invasive
but also different resolutions and they are packaged in different
as possible. A particular consideration in the interest of this
ways. Because each of these agencies use different schema to
paper is the preservation of wetlands.
organize and distribute this data, it would be extremely
beneficial to develop a script that will take all of this data and
Wetlands are an important natural feature inherently capable
preprocess it so that all data is formatted in an easily readable
of many beneficial hydrological and environmental processes.
and consistent way to facilitate quicker and easier wetland
Some of these benefits include storm water runoff control,
delineations.
effluent and sediment control, and providing habitats for
wildlife and plants. Historically, wetlands were thought to be II. OBJECTIVE
bastions for disease and were destroyed or repurposed for
The main objectives of this study are to develop a tool that
agricultural reasons. Because of this practice, approximately
will compile a wide range of data sets, preprocess data sets into
half of America’s original wetlands no longer exist. The need
a useable and consistent format, and organize these downloaded
for wetlands preservation quickly became apparent and a
and processed data sets in a systematic and readable hierarchy.
number of agencies showed interest in this matter.
This tool's primary purpose is to serve as a starting point for an
expanded project, which is for the development of an ArcGIS
In order to adequately prevent the destruction of wetlands,
tool to delineate wetlands. In the interest of time, the following
we need to be able to accurately and easily identify their
three data sets have been selected for inclusion in this study:
locations. The National Agriculture Imagery Program (NAIP)
NWI, SSURGO, and NHD.
maintains a current data base of wetlands called the National
Wetlands Inventory (NWI). However, this inventory has
proven to be insufficient at times, requiring further
 As these data sets have a number of different sources, the tool
will need to have the ability to utilize a source’s established web
application programming interfaces (API) or if none exist, have
the ability to almost completely automate the data acquisition
procedure to require as little user input as possible.
Preprocessing these data sets will require the appropriate
geospatial data processes to ensure that all datasets have
consistent attributes. Finally, the organization will play an
important role in separating the user from the tool's processes,
while making it easier for the user to give the tool the
appropriate required data in order to function.
III. METHODS
A. Language
Selecting the appropriate programming language was an
important step for this study. Considering that the intended use
of this study’s tool is to be used for another project to develop
a tool for ArcGIS, it seemed pertinent to utilize the same
programming language required by the system. Since Python
is the intended programming language for the development of
the wetlands delineation tool, this study’s tool also used Python.
The tool is segmented into two separate scripts, the main
script that queries the user for the appropriate preliminary
information and a separate module that contains all of the
procedures of the script. This segmentation between querying
and computation is to reduce overwhelming the user with
portions of code that should remain unchanged.
B. Preliminary Requirements
In order for the script to run without error, some preliminary
information and a file is required from the user. The user first
must be able to provide the state and county of interest. Both
of these inputs must be in the appropriate format. To indicate
the desired state, it must be represented by using the state’s
abbreviation, i.e. NC, NY, etc. To indicate the desired county,
it must be represented by typing out the full name including the
term county while maintaining capitalization, i.e Hyde County,
Lexington County, etc. The user must also be able to provide a
Comma Separated Values (.csv) file containing the Federal
Information Processing Standard (FIPS) codes for each
particular county within a state which can be downloaded from
the following site:
https://www.census.gov/geo/reference/codes/cou.html
C. Data Acquisition
To accomplish the data acquisition portion of this project,
the first step was to identify any web API’s used by the agencies
in charge of the desired data sets. These groups include the
United States Geologic Survey, United States Fish and Wildlife
Service, and the Natural Resource Conservation Service. Each
of these entities maintained File Transfer Protocol (FTP) sites
at the following locations for the indicated data sets:
NHD:
ftp://nhdftp.usgs.gov/DataSets/Staged/States/FileGDB/
SSURGO:
http://websoilsurvey.sc.egov.usda.gov/App/WebSoilSurvey.as
px
NWI:
http://www.fws.gov/wetlands/data/State-Downloads.html
Using these designated sites, the developed Python script
calls these data sets using the urllib2 library. By using this
library, the script establishes a link to the particular site
location, at which point, the script opens the indicated file for
reading. The script will then read the file in defined blocks and
write these blocks one after another to the same directory as the
script using built in libraries with the .read() and .write()
functions. The script also contains a file transfer progress
update that will print the status to the Integrated Development
Environment (IDE) console using built in libraries with the
.info() and .getheaders() functions. The block size was defined
as 250000 bytes, which was tested empirically to find an
appropriate block size, as a block too small would cause a
remote termination of the connection with the host server and a
block too large would not provide the user with frequent enough
updates on the file transfer progress.
The NHD and NWI sites are capable of extracting the
required data for a particular site using only the user defined
state, which also means that these data sets span the entirety of
the state. However, the SSURGO data also requires the county
of interest in order to determine the county’s FIPS code in order
to call the appropriate url. In order to accomplish this, a python
definition is used to unpack and generate a python dictionary
containing the county names as the keys and the FIPS codes
associate with those counties using the csv library. The
dictionary is then used to grab the FIPS code for the user
defined county.
Each of these data sets are downloaded as a zip file. In order
to utilize these data sets for the pre-processing step, they must
be extracted. To accomplish this, the zipfile library is used to
extract the data while the os library is used to specify the path
location for the extracted data sets.
D. Pre-Processing
The pre-processing portion of the script was initially
developed using ArcGIS’s Modelbuilder tool. Modelbuilder
was used to outline the operations used to process the data. The
particular tools of interest are Project, Clip, Create Personal
Geodatabase, Feature to Geodatabase, and Table to
Geodatabase. The Projection tool was used to convert all data
sets to the North American Datum 1983 (NAD83) datum. This
projection was selected since the intended use for this study’s
tool was for the download and pre-processing of any county
within any state in North America. The Clip tool was used to
reduce the NHD and NWI data sets to the particular county of
interest. Create Personal Geodatabase initializes the
geodatabase where Feature to Geodatabase and Table to
Geodatabase append the data sets to the newly created
geodatabase.
This model developed in Modelbuilder was exported to a
Python script, which was altered and pasted into the main script.
This portion of the script utilizes the ArcGIS Python library,
called arcpy, to access and run the Project, Clip, Create Personal
Geodatabase, Feature to Geodatabase, and Table to
Geodatabase geo-processing tools. This will output a
geodatabase file (.gdb) containing the NWI, NHD, and
SSURGO data in the scope of the user defined county.
IV. CONCLUSION
The development of a tool to acquire and pre-process data
sets utilized for water delineation was successful. However,
due to time limitations, only certain data sets were able to be
included in the tool. This tool shows that by using Python to
automate the tedious process of data acquisition and pre-
processing, time and effort can be saved by limiting redundant
tasks using minimal user input.
With any automated tasks, there are a number of assumptions
that are made that could prove to be problematic in the future.
For example, the website urls used in the script to download the
data are hard coded into the definitions. If one of the sources
decides to change its web location, the script will break and not
function properly. Similarly, if a source were to change the
naming convention of their zip files, the script would also break.
The task of including an intelligent algorithm that would search
the potential files for the most likely desired set of data is not
impossible, but it is outside the scope of this study and can be a
rigorous procedure to make this script more robust.
Even with the successful development of this script, there is
room for improvement that will need to be implemented before
the full potential of this script is reached. First, other data sets,
like DEM, Multispectral satellite imagery, radar, and NDVI
will need to be incorporated. Second, as the spatial area of
interest for the use of this tool does not expand beyond the
United States, creating a comprehensive list containing all FIPS
would be beneficial as the user would not have to provide a .csv
file. Third, implementing any catches for error codes would be
more beneficial to users without programing experience. These
tasks would improve the functionality and ease of use of the
script in its current condition. Finally, enveloping this code in
a loop will allow the user to define multiple states and counties
of interest.
This tool provides a useful functionality for compiling
multiple datasets from federal FTP sites and remote sensing
technologies that facilitate quicker and easier acquisition and
pre-processing of data for use within a wetlands delineation
model. This tool utilizes the appropriate libraries in order to
access, download, and extract zip files from the appropriate
sites followed by projecting and clipping the data sets so that
they have a consistent datum and spatial span. The benefit of
this tool is to automate the data acquisition and pre-processing
steps by allowing the user to define the area of interest and
allow the script to run over a period of time, freeing the user
from navigation through multiple sites demanding much more
user input and time.
REFERENCES
[1] Soil Survey Staff, Natural Resources Conservation Service, United States
Department of Agriculture. Web Soil Survey. Available online at
http://websoilsurvey.nrcs.usda.gov/. Accessed [12/9/2014].
[2] National Wetlands Inventory Staff, U.S. Fish and Wildlife Services, U.S.
Department of Agriculture. National Wetlands Inventory. Available
online at http://www.fws.gov/wetlands/data/Data-Download.html.
Accessed [12/9/2014]
[3] Python Software Foundation. Python Language Reference, version 2.7.
Available online at http://www.python.org.
[4] ESRI 2011. ArcGIS Desktop: Release 10. Redlands, CA: Environmental
Systems Research Institute.
[5] U.S. Census Bureau. (2014). 2010 FIPS Codes for Counties and County
Equivalent Entities. Available online at
http://www.census.gov/hhes/www/income/statemedfaminc.html.