LSGI547 – Advanced Geographic Information Systems

Tutorial 1 - Data Acquisition

Weiwei Tan
Department of Land Surveying and Geo-Informatics,
The Hong Kong Polytechnic University

acknowledge the contribution of Fan Shi and Mahzabin Abbasi

Discussion: ?
Where do you collect spatial data?
 Esri Open Data Hub
The Esri Open Data Hub is a hidden gold mine of free GIS data.
For example, it now houses over 250,000+ open data
sets from 5,000+ organizations worldwide. For this reason, we
have it at the top of our list of free GIS data.
Esri Open Data Hub

API (Application Programming Interface)

 Esri Open Data Hub
XML: represent any type of structured data. Data storage and exchange.
purpose KML: geospatial data and visualization.

XML: creating custom tags and defining the structure of the document according to
the specific data.
Structure KML: a predefined structure with specific elements and attributes tailored for
representing geographic features (points, lines, polygons, and overlays).
XML VS KML
XML: domains like web services, data storage, configuration files, and data exchange
Usage between systems.
KML: visualizing geographic data and creating interactive maps.

XML: defined by the World Wide Web Consortium (W3C) and has a wide range of
tools and libraries available for processing.
Standard KML: created by Keyhole, Inc. (later acquired by Google) for their mapping products,
supported by Google Earth and related applications.
 Esri Open Data Hub
<?xml version="1.0" encoding="UTF-8"?> XML declaration;
<kml xmlns="http://www.opengis.net/kml/2.2"> root element ;
<Document> the main container element for the KML data;
<name>My Example KML Document</name> name of the document or feature;
<open>1</open> whether the document is open or closed by default;
<Placemark> a point or set of points on the Earth's surface;
<name>Location 1</name> the name of the placemark ("Location 1" and "Location
<Point> 2". );
An example <coordinates>-118.4045,34.0551</coordinates> element that represents a point on the Earth's surface;
of XML </Point> coordinates of the point;
</Placemark>
<Placemark>
<name>Location 2</name>
<Point>
<coordinates>-117.1574,34.0702</coordinates>
</Point>
</Placemark>
</Document>
</kml>
 Esri Open Data Hub

https://pypi.org/project/simplekml/ https://pythonhosted.org/pykml/tutorial.html
 Esri Open Data Hub
OGC WMS stands for Open Geospatial Consortium Web Map Service. It is a standard protocol
for serving geospatial map images over the internet. WMS services allow users to access and
visualize geospatial data in the form of maps through a standardized interface.
Interoperability: adhere to a standard protocol, easier for different systems and software to communicate with
each other, can be accessed and displayed consistently across different platforms.

Integration: be easily integrated into various applications and websites, allow users to incorporate maps into their
own projects without extensive coding or data acquisition.

Customization: flexibility to customize the appearance and content of the map (specific layers, adjusting
transparency, and applying different styles and symbology).
benefits
Data Access: provide access to large amounts of geospatial data that may be stored on remote servers. Users can
access and visualize this data without the need to download or store it locally.

Collaboration: allow users to share maps and geospatial data with others. multiple users view and interact with the
same map simultaneously, promoting teamwork and information sharing.

Scalability: handle large amounts of geospatial data and can serve maps to a large number of users
simultaneously, suitable for applications with high traffic and large user bases.
 Esri Open Data Hub

What are JSON

and GeoJSON?

• JSON (JavaScript Object Notation) is a lightweight data interchange format that is used to store and
transmit data between a server and a web application. It is a text-based format that uses simple and
familiar syntax from JavaScript to define data structures.
• GeoJSON is a subset of JSON specifically designed to represent geographic data. GeoJSON includes
additional properties and structures to represent geographic features such as points, lines, and polygons,
along with their associated attributes. It provides a standardized way to encode geographic information.
 Natural Earth Data
• Natural Earth Data is number 2 on the list because it best suits the needs of cartographers. By
and large, all the key cultural and physical vector GIS datasets are on a global scale conveniently
for you to use. The raster datasets also provide beautiful hill shade relief for your map.
• The best part is Natural Earth Data is in public domain. So, this means that you have the right to
use, modify, and disseminate the data in any manner.
Natural Earth Data
 USGS Earth Explorer
 For satellite and aerial imagery, the USGS Earth Explorer is one of the
largest free sources of data.
 A friendly user interface makes accessing remote sensing data simple.
 In fact, it even has a bulk download application if you need to download
more than one data set.
 Registration is free.
 USGS Earth Explorer
Categories

• Aerial Photography
• Digitized Maps
• Elevation Products
• Land Cover
• Satellite Imagery
• Image Gallery

https://eros.usgs.gov

EarthExplorer Bulk Download Web Application (BDWA):

https://www.youtube.com/watch?v=yUMreXIzptc
 OpenStreetMap
 OpenStreetMap (OSM) is a free, editable map of the whole world
that is being built by volunteers largely from scratch and released
with an open-content license.
 The OpenStreetMap License allows free (or almost free) access to
OSM map images and all of OSM underlying map data.
 The project aims to promote new and interesting uses of this data.
 While OSM is open to the public, it’s also created by the public. So,
this means that accuracy varies based on the creator.
 From my experience, accuracy is very good. And people like you and
me are working on completeness every day.
 OpenStreetMap

One of the advantages of OSM is its flexibility to store any type of feature, given the
many tags that already exist and the community-based tag proposal and voting
process. In some cases, specialized thematic maps have been created around a
subset of feature types. Examples of these include:
 OpenCycleMap, specializing in drawing bicycle trails that people have submitted
to OSM
 OpenSkiMap, showing ski lifts and trails
 Wheelmap, allowing the browsing and marking of wheelchair-accessible
locations
 UNEP Environmental Data Explorer

United Nations Environment Programme (UNEP) is the

authoritative source for UN data. For example, it holds more
than 500 variables such as freshwater, climate and health.
I must admit. I found it a bit difficult to explore the GIS data
because the interface takes a while to get used to. If you click
advanced, you can filter “Geospatial Data Sets”. From here, you
can download free GIS data like climate, disasters, and
ecosystems.
UNEP Environmental Data Explorer
 EARTH DATA BY NASA
https://www.earthdata.nasa.gov
Earth Science Data Systems (ESDS) Program
provides full and open access to NASA’s
collection of Earth science data for
understanding and protecting our home
planet..

Data Type
 Atmosphere
 Biosphere
 Cryosphere
 Human Dimensions
 Land Surface
 Ocean Earth Data Home page
 Solid Earth
 Sun-Earth Interactions
 Terrestrial Hydrosphere
Downloading Data from Earth Data
Data Searching Link
https://search.earthdata.nasa.gov/search
 Should create an account to download data
 Select your area of interest
 Find your desired data in the search box

 Download Data Files from HTTPS Service with Wget

https://disc.gsfc.nasa.gov/information/howto?title=How%20to%20Download%20Data%20Files%20from%20HTTPS
%20Service%20with%20wget
 Google Earth Engine API
What is GEE?
• Cloud-based platform for geospatial analysis
• Access over 40 years of satellite imagery
• Upload own data sets to integrate with publicly available data
• Apply range of common algorithms to data
• Export images, tables, charts, map outputs

GlobCover Elevation Night-time lights

 What can you do in the API
• Run algorithms on image data:
– Simple Band combinations such as NDVI
– Masking bands using existing bands
– Classifying satellite data
– Subsetting data spatially using vector data
– Long list of options – most of the actions would appear in ArcMap, ENVI, Erdas
Imagine, R.
 Home page
• https://earthengine.google.com/
• https://explorer.earthengine.google.com/#workspace
Earth Engine Data Catalog
•https://developers.google.com/earth-engine/datasets/catalog
 How to Download Data from GEE
The Code of different products has also been provided….
ArcGIS online Data Source
• In ArcGIS Online, it is possible to download publicly shared data as
shapefiles to the local machine. The owner of the data must configure
the hosted feature layer to allow others to export the data. Refer to
ArcGIS Online: Export data from hosted feature layers for more
information.
Data downloading steps from ArcMap
 What we have learnt……
• Most popular portals for acquiring free data
• Advantage of using those websites
• Data format that can acquired from those websites

Links For Video tutorials

https://youtu.be/LqjWCFQ2AlE
https://youtu.be/UuE8tFYz6q0
https://youtu.be/oAElakLgCdA
https://youtu.be/X56vtqutv9o
https://youtu.be/KFuihGDHE-s
Useful Spatial Data Sources- GIS Portal
Open Government Data
• HK GeoData Store https://geodata.gov.hk/gs/ Hong Kong CSDI Portal
• NYC (New York City) Open Data https://opendata.cityofnewyork.us/
• Open Data for Europe https://data.europa.eu/en/
Useful Spatial Data Sources
Online Map Resources
• VGI websites
• Direct download
• APIs
• Digital Map Services
• APIs
• Web crawlers
Useful Spatial Data Sources
Social Media Data
• Twitter/Weibo
• APIs
• Web
• crawlers

https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api
Useful Spatial Data Sources
Data Competitions
• Kaggle
• IEEE GRSS
• ISPRS
• ACM SIGSPATIAL GIS Cup
Two useful coding tools
• R studio
- Accessing data through API
- Data processing and analysis

• Python
- Accessing data through API
- Data processing and analysis

IDE (Integrated Development Environment)

How to download Rstudio
• Installing R on Windows OS
• To install R on Windows OS:
1.Go to the CRAN website.
2.Click on "Download R for Windows".
3.Click on the “install R for the first time” link to download the R
executable (.exe) file.
4.Run the R executable file to start the installation and allow the app to
make changes to your device.
5.Select the installation language.
How to download Rstudio
• To install RStudio Desktop on your computer, do the following:
1.Go to the RStudio website.
2.Click on "DOWNLOAD" in the top-right corner.
3.Click on "DOWNLOAD" under the "RStudio Open Source
License".
4.Download RStudio Desktop recommended for your computer.
5.Run the RStudio Executable file (.exe) for Windows OS or the Apple
Image Disk file (.dmg) for macOS X.
How to download Anaconda
• Go to the Anaconda Website and choose a Python 3.x graphical
installer (A) or a Python 2.x graphical installer (B). If you aren't sure
which Python version you want to install, choose Python 3. Do not
choose both.
How to download Anaconda
• Run the downloaded exe file and start the installation.
How to import/download data using APIs
• Accessing vector data by R studio
How to import/download data using APIs
• Accessing vector data by Python
How to import/download data using APIs
• Accessing weather data using R studio
Summary

• A numerous data sources were presented and described.

• Several basic but pivotal terms were explained in detail.
• Two coding tools (R and Python) and their IDEs (Integrated Development Environments)
were introduced.
• Several cases by using Web Scraping were demonstrated.
 10 Free GIS Data source
• Esri Open Data Hub
• Natural Earth Data
• USGS Earth Explorer
• OpenStreetMap
• NASA's Socioeconomic Data and Applications Center (SEDAC)
• Open Topography
• UNEP Environmental Data Explorer
• Earth Data by NASA
• Sentinel Satellite Data
• NASA Earth Observations (NEO)