Geospatial Data Quality and Accuracy: The State-of-the-Practice

Dr. Qassim Abdullah

Chief Scientist and Senior Associate
Woolpert, Inc.
MSS 2019 Spring Technical Conference, April 4 & 5, 2019
Maritime Conference Center
692 Maritime Boulevard
Linthicum Heights, MD 21090-1952
 Agenda
1) The state of Geospatial Technologies:
1) Latest lidar technologies (mobile, aerial and UAS-based lidar) in the market
and its products quality and accuracy;
2) Latest digital camera technologies in the market and its products quality
and accuracy;
2) Unmanned Aerial System (UAS) for Geospatial Mapping
3) Key features of the new “ASPRS Positional Accuracy Standards for Geospatial
Digital Data” as it relates to products from the latest digital cameras and lidar
sensors;

2
• Aerial photography
• Lidar / Remote sensing
• Surveying
• Land agent services
• GIS
• Environmental /
Permitting
• Engineering
– Civil, structural,
mechanical,
electrical
• Architecture &
landscape
• Construction observation
• Asset management

3
 Geospatial Services

Aerial Mapping Mobile Lidar Terrestrial Lidar

Surveying GIS Bathymetry

UAS Operations 4
The State of Lidar Technologies
• Since early 90’s, lidar served the geospatial, engineering and
construction communities relentlessly and successfully

• Lidar became an integral part of the sensors group for

collecting 3D geospatial data

• Lidar technologies is ever evolving and manufacturers

surprises us every year with new innovations
Whether it is…

Mobile Lidar

Mobile Lidar
or…
Static Lidar
or…
Aerial
Aerial
Lidar
Lidar
Or…. Aerial Bathymetric Lidar

Courtesy: Geomatics Data Solutions

And now…

UAS-based Lidar
Lidar Gave Us Everything We Needed
Lidar Gave Us Everything We Needed
Lidar Gave Us Everything We Needed
Stationary or Mobile Lidar Systems

Pipeline Construction
Image courtesy: 14
Teledyne Optech
 3D Laser Scanning Report by IDOT
3.1 BENEFITS
The major benefits realized to date include the following:

1. An overall reduction in the total time required for a given

surveying project;

2. An approximate 35% time savings for personnel

conducting tasks in the field, leading to a significant
reduction in the exposure to traffic hazards;

3. An increase in the accuracy of the data obtained;

4. A decrease in the required survey crew size for certain

project tasks;

5. Innovative and alternate uses for the equipment were

discovered, such as high-accuracy surface topography
mapping, bridge and structure scanning, and forensic
investigations of the structure components and other
points of interest. 15
The State of Aerial Lidar Topo
Systems

16
 PulseTRAK
• Truly continuous operating envelope for maximum efficiency
No data gaps or density loss across PIA/MTA zones
•
PRIME
SwathTRAK
• Dynamic FOV: fewer flight lines and constant point distribution
• Increased range of motion: survey the steepest mountains

More Range, More Density

• 30% increase in range performance and/or increased point density
• “Night mode” increases range performance further still

17
Latest RIEGL Waveform-LiDAR Technology

• atmospheric clutter suppression

• acquisition of up to
300 km²/h at 8 pts/m² • field of view optimized for
corridor mapping
• 100 pts/m² for typical
RIEGL VQ-780i multi-rotor copters

RIEGL miniVUX-1DL

• 2 million measurements/sec
• flexible camera system with
up to 9 cameras

• up 60 scan positions/hour
• up to 2,500 m range
RIEGL VMX-2HA
RIEGL VZ-2000i
www.riegl.com
 Linear mode: TerrainMapper
- 2.0 MHz pulse rate
- Advanced scanning optics for 5500 m AGL
- Gateless MPiA (to 35 zones)

Single-photon mode: SPL100 Processing: RealTerrain

- Automatic range gate - Auto calibration
- Direct intensity measurement - Registration
- Hi-res scan angle encoding - Line-line, intra-line QC
- Distributed processing
 Advances in Geiger Mode LiDAR – Harris Contribution
Continuous improvement through experience, customer feedback, and IR&D over 3+ years
HARDWARE:
EAR Certification under DOC
• System design design yielding improved SNR and data quality No longer ITAR Controlled
• Increased reliability via hardware improvements – Zero downtime over past year
• Characterization atmospherics/altitudes/landcovers – Validation/Calibration- sensor model
SOFTWARE:
• Mission planner improvement yielding:
• Higher collection efficiency
• Pilot navigation – reduced error
• Registration – Improved automation, throughput, and accuracy
• Aggregation – Improved product uniformity and overall quality
• Automated QA/QC tools and reporting
• Noise fitter
PROJECTS PAST YEAR:
• Utilities
• Federal - USGS and DoD
• State & County
• Forestry
• Foreign Government
 New Aerial System
Integrated Oblique and Lidar system

21
 Leica CityMapper
Hybrid Airborne Sensor
• Oblique and Multispectral nadir imagery
• LiDAR data
• 2D and 3D Geospatial data

Industry’s Fastest Data Throughput

• High-performance multi-sensor workflow
• Generate comprehensive geospatial base layer
• Scalable and upgradable to your needs

Make Smart Decisions

• Adapt to rapidly changing urban environments
• The fastest way to create geospatial information layers
• Highly accurate 2D products and 3D models

Courtesy of Leica and Geomni 22

 CityMapper Specifications

• 1 x Leica RCD30 CH82 multispectral camera in nadir (4 Band)

• 4 x Leica RCD30 CH81m oblique camera, viewing angle 45° (3 Band)

• 1 x Leica Hyperion LiDAR unit (Pulse repetition up to 700KHz)

• SPAN IMU CNUS5-H, no export license required US ECCN 7A994

• CCD Size (80 Mega Pixels) 10,320 x 7,752 pixels / Camera

Courtesy of Leica and Geomni 23
Dual Oblique
Imagery and
Lidar

City Mapper

Courtesy of Leica and Geomni

24
The State of Aerial Bathymetric
Lidar Systems

25
26
27
28
The State of Mobil Mapping Systems

29
 Mobile Mapping System Technology (MMS)

2,000 pts/m2 to 6,000 pts/m2 Accuracy@1.8 cm

30
Aerial and MMS Lidar for Road Design and Construction

Platforms for Lidar

Survey

• Interstate – Ramps –
Local Roads
The “Twin Tunnels”

31
 MMS for Sign Inventory - City of Indianapolis

+ 3200 drive miles

+ 400 square mile
area
+ Acquisition
+ LiDAR
+ Imagery

32
 Woolpert’s Smart Feature Extraction Tool

• Imagery (Visual Information) • LiDAR (Spatial Information)

– Sign Color
– Sign Position/Location
– Sign Text
– Condition – Sign Dimensions
• Functional – Cardinal Direction
• Needs Attention
– Mounting Type 33
• Needs Replacement
MMS for Surveyor

34
Feature
Extraction

35
Contours from MMS

36
Contours from MMS at 0.1’ Intervals

37
 UAS-based Lidar

Points Density: 600-900 pts/m2

Accuracy: 2 cm
Altitude: 70 to 300’ AMT
Powerful Lidar offered for UAS
 What is Next for lidar?

Increased Efficiency and Productivity

40
The birth of: Flash, Geiger Mode, and Single Photon Lidar

41
 Linear mode Lidar, Geiger-Mode Lidar (GML), Single-Photon Lidar (SPL):

It is all about harvesting the laser energy…

In Linear mode Lidar, a laser pulse received by single pixel receiver

42
The difference between current Lidar and
Single Photon/Geiger Mode Lidar

source laser pulse Linear Lidar

One-to-one

spot Image received at the single anode

Single Photon
Lidar

One-to-many

43
Single Photon Lidar Samples
25 points/sq m

44
 Single Photon Lidar
Mecklenburg County, NC Pilot,
1.6 Km swath, 15 points/sq m, 180 Knots, single pass

Acquired using HRQLS Lidar System

45
 Single Photon Lidar
Mecklenburg County, NC Pilot,
1.6 Km swath, 15 points/sq m, 180 Knots, single pass

Acquired using HRQLS Lidar System 46

The New Lidar Technology: Our
Experience

The USGS Task Orders to

Evaluate the New Lidar
Technologies
 The New Lidar Technology: Woolpert Experience

South Dakota Missouri River Lidar Task

Order
• Woolpert was tasked by the USGS to use the Single
Photon Lidar to collect QL1 data for 5 counties in South
Dakota

• Data was acquired and delivered during 2017/2018

SPL Acquisition in
Intensity
South DakotaSample
Missouri River,
Chamberlain SD

This task includes 3,726 square miles of high

density USGSv1.2 specification, QL1 or better

Single Photon Lidar acquisition and processing.

The AOI is located in southeast South Dakota and

consists of Aurora, Buffalo, Brule, Jerauld, and
Charles Mix Counties.
 Positional Accuracy

Meta Data for the Delivery of SPL for South Dakota Counties

Bare-Earth DEM Non-Vegetated Vertical Accuracy (NVA) Tested 0.139

Meters Non-Vegetated vertical accuracy at a 95 percent confidence
level, derived according to NSSDA, in open terrain using (RMSEz) 0.071 x
1.96000 as defined by the National Standards for Spatial Data Accuracy
(NSSDA); assessed and reported using National Digital Elevation Program
(NDEP)/ASPRS Guidelines and tested against the DEM against 108 NVA
points.
The New Lidar Technology is the
new tool in the tool’s box for the
Geospatial industry
 CHALLENGES IN LIDAR ACQUISITION OVER HAWAII BIG ISLAND

CHALLENGE #1

Dense
Vegetation

Image courtesy https://goop.com

CHALLENGE #2 Rough Terrain
 CHALLENGE #3

Difficult
Weather

Images courtesy: www.hawaii-guide.com

• The United States Geological Survey (USGS) and NOAA wanted aerial
lidar survey for the big island through the USGS GPSC 3 and NOAA
OCM’s CGSC 2 contracting vehicles

• The lidar data originally was requested to meet QL2 according to the
USGS Lidar Base Specification with 2 pts/m2

• Woolpert, assessed the project conditions which were characterized by

the following constrains and challenges:
• Rugged terrain;
• Severe weather conditions (rain, wind, clouds, etc.) and short flying
season
• Dense vegetation
• Woolpert analyzed different lidar technologies such as linear mode
lidar, Geiger mode lidar, and single photon lidar (SPL)

• Best on Woolpert’s past successes in using SPL especially in South

Dakota USGS contract, Woolpert concluded that the Single Photon
Lidar technology (SPL) is the most suitable technology for the job
considering the challenging project conditions

• We selected SPL because:

• We fly it higher and faster resulting in wider swaths coverage à
Reduced flying time
• Provides denser data which makes it more suitable for vegetation
penetrationà allow us to deliver QL1 (8 pts/m2) instead of QL2
 Why Single Photon Lidar Technology?
More efficiency with Single Photon Lidar Technologies

source laser
pulse Linear Lidar
One-to-one
spot Image received at the single
anode

Single
Photon Lidar
One-to-many
Acquisition Parameters:
• Sensor: Leica’s SPL100 sensor Flight Plan
• Project area: (3,160 + 868) square miles
• Acquisition Window: Winter of 2018 -
approximately January 1 through February 28,
2018
• Flying Altitude: 3,750 m (12,300-ft.) AGL
at 200 knots
• Swath Width: 2 km using a +/- 15°scan
angle
• Lines Overlap: 50%
SPL100 at Glance:
• Captures: 6 millions measurements per Leica SPL100
second using 100 output beams
• Operation Environment: day or night,
leaf-on or leaf-off conditions and in dense
vegetation
• Point Density: 12 - 30 pts/m2 (depending
on flying height)
• Flying Altitude: Optimal at 12,000 ft.
AGL Images courtesy: Leica Geosystems
Big Island: SPL data collected during 2108

60
Big Island: SPL data collected during 2108

61
The Big Island: SPL data collected during
2018: Vegetation Penetration
 Project Update
• Percentage Completed in 2018: 43%
• Data Quality:
• Woolpert internal positional accuracy verification: RMSE = 0.069 m or 6.9-cm
• Data density was around 30-40 pts/m2 , delivered QL1 but with 15 pts/m2
• USGS received and accepted the pilot project delivery
• Data Processing: Better data, less noise as compared to what we experienced last
year
• 2020 Plan: To resume flying the remaining flight lines during winter of 2020
• Winter of 2019 was excluded because of the El Nino forecasted for 2019
 The State of Geiger Mode Lidar
PINELLAS COUNTY SAMPLE DATA – 20 PPSM
By Harris

Flying Altitude: 13,000 ft

Density: ≥ 20 ppsm
Excellent Details
Excellent
Details
Excellent Details
Excellent Details: Power lines Delineation
Excellent Details: Trees Penetration
Poor return from
water (expected
with NIR laser)
 Performance Comparison of different Lidar

Performance Factor MMS Linear Mode Lidar Geiger mode / Single

Photon Lidar
Range of Operational Altitude 200 300 to 5,000 3,000 to 10,000
AGL/ Distance (m)
Point Density (pts/m^2)** 500 to 6,000 1 to 15 (opt. 2) 12 to 32
Accuracy (cm) 1.8 5 to 10 10
** density at reasonable operational altitudes

71
 Where we are heading with Lidar technologies?
• Demand for Lidar data is still going strong as compared to aerial
imagery
• Lidar Technology in general is keeping up with users needs and
demands
• Increased demand for denser Lidar data:
– 8 ppm density Lidar data will be in a popular demand within the next
five years.
– Hexagon just released the Terrain Mapper with pulse rate of 2,000,000
pps (2 MHz pulse repetition frequency). Other lidar manufacturers
announced similar innovations.
– We expect the commercial use of single photon and Geiger mode
lidar to expand in the next few years
These systems make it possible to fly area with the following specifications:
• Flying altitude: 15,000 ft. to 25,000 ft. (4.7 to 7.6 km)
• Swath width: 7,900 ft (2.4 km) or wider
• Point density: 12 to 25 ppm 72
 Where we are heading with Lidar technologies?
• Although it is still sluggish, we expect a wider utilization of full
waveform digitization as application software catch up with the
advancement in the hardware technologies.

• Mapping companies will continue to provide their custom services

to federal and local government clients.

• Crowdsourcing and subscription-based use of lidar data gets more

popular as more suppliers enter this market – this will take a while.

• Digital surface terrain data derived from imagery (3D GIS) will take
some share off the Lidar market

73
 Where we are heading with Lidar technologies?

• Due to its high efficiency, new Lidar capability such as single photon
system will enable more frequent Lidar surveys for wide area and
regional coverage. This will generate more demand for lidar data.

• Increased use of the street view and Mobile Lidar to support the 3D
GIS

• Increased utilization of Bathy Lidar

• UAS-based lidar may fill the gap in the market for small projects. It is
bridging mobile lidar capability with aerial lidar

• Lidar from space? It is possible but will not be dense enough

74
 The latest in
Digital Aerial Cameras

Multiple Configurations, Multiple Options

75
 Hexagon DMC III
25,728 pixels across swath

76
The Largest Camera in the Market:
VEXCEL UltraCam CONDOR
37,800 pixel swath width

77
 Hexagon ADS100
CCD linear array size*: 20,000
pixels

78
79
FLIR SC8300 Thermal Camera
Resolution 1,344 × 784 pixel

80
 The latest in

Cameras For the

Unmanned Aircraft Systems

81
Imagery-based Points Cloud Sample

3D Point Clouds & Textured Meshes from Imagery

82
Source: acute3D
Power Station by UAS: >2000 pts/m2
Points cloud
from
imagery

80 points/m2

84
 Points
cloud
from
imagery

210
points/m2

85
Points cloud from imagery
210 points/m2

86
Points cloud from imagery
210 points/m2

87
Bypass Construction project
2-cm Orthos Details

88
Imagery-based Points Cloud Sample Bypass Construction project

89
Imagery-based Points Cloud Sample Bypass Construction project
90
Imagery-based Points Cloud - Clarendon Bridge, Spanning White River
Clarendon, Monroe County, AR

91
Points Cloud

What is it?

92
It is 3D points in space with know X, Y, Z used to represent the terrain

93
LIDAR OR IMAGERY?
Imagery Lidar

100 pts/m2 30 pts/m2

We know LIDAR-based points cloud very
well but….
How much do we understand imagery-
based points cloud?
 UAS versus MMS: Data Smoothness
UAS

Lidar
UAS versus MMS: Data
Smoothness
UAS

MMS
 UAS versus MMS: Features Mapping
UAS
Power lines &
power poles

UAS products may catch up with the

MMS products accuracy but not
necessarily the 3D features details
MMS
UAS versus MMS: Features Mapping
Roads Signs Only in MMS Data
MMS
Why imagery? Because UAS is taking the industry by
storm
1. Very affordable solution for small size projects < $10,000
2. Ease of deployment
3. Ease of use
4. Adaptable and maneuverable
5. High resolution products – less than 1-cm GSD imagery
6. Accurate – when planned correctly
7. Capable software:
a. Data processing
b. Measurements, automated analysis such as digital image classification
and change detection

All that resulted in putting UAS in the hands of non-mapping

professionals
Lack of Basic Knowledges Leads to False Conclusions
Accuracy and Precision

Accurate, Precise, not Accurate, and

not precise Accurate precise Not accurate, not
precise
Camera on board small Unmanned Aerial System (UAS)
Mostly consumer grade cameras:
• Sony
• Panasonic
• Cannon
• Nikon
• GoPRO 102
Why it is accurate, then?
 Success Factor 1
Low Altitude, High Resolution
Low altitude minimizes attitude errors on products and increase details

•
 Success Factor 1
Low Altitude, High Resolution
Low altitude of 100 ft. minimizes attitude errors on products

105
Success Factor II
Small Projects

• Small Projects equates to:

• Smaller block for the bundle adjustment and

least squares

• Better GCPs to frames ratio

 Success Factor III
Image redundancy with increased overlap

•
Increased imagery overlap results in strong reliability

GCP measured in 40 images

Digital Multi-Rays Photogrammetry

Least squares finds the

best point of
intersection not human
eyes
Success Factor IV: Processing Software and Matching Algorithms
Plenty of tie points results in a strong geometrical fidelity
Plenty of tie points results in a strong geometrical fidelity

111
 Success Factor IV: Processing Software
It handles camera self calibration and structure from
motion

Platform used: Woolpert Kespry UAS

 Accepting UAS Survey?
Yes you can, How?
1. Demand a rigorous photogrammetric workflow if
you are after high positional accuracy

2. Ask for a mapping standards

Specify the New ASPRS Positional Accuracy Standards

for Digital Geospatial Data, if you can
 Why…

The New ASPRS Positional Accuracy Standards

for Digital Geospatial Data
The new ASPRS mapping accuracy standard simplifies things for
you…
YOU: Set your final
product accuracy
requirements

ASPRS Standards: Set

Aerial Triangulation Check Points

GCPs Accuracy Accuracy Reporting
Accuracy Accuracy
 New Standard for a New Era
• New map accuracy is influenced by many factors such as:
– the quality of camera calibration parameters;
– quality and size of a Charged Coupled Device (CCD) used in the digital
camera CCD array;
– amount of imagery overlap;
– quality of parallax determination or photo measurements;
– quality of the GPS signal;
– quality and density of ground controls;
– quality of the aerial triangulation solution;
– capability of the processing software to handle GPS drift and shift;
– capability of the processing software to handle camera self-calibration,
– the digital terrain model used for the production of orthoimagery.
116
 New Standard for a New Era

• These factors can vary widely from project to project, depending on the sensor
used and specific methodology. For these reasons, existing accuracy measures
based on map scale, film scale, GSD, c-factor and scanning resolution no longer
apply to current geospatial mapping practices.

• Elevation products from the new technologies and active sensors such as lidar,
UAS, and IFSAR are not considered by the legacy mapping standards. New
accuracy standards are needed to address elevation products derived from these
technologies.

117
 New Standard for a New Era

• Within the past two decades (during the transition period between the hardcopy
and softcopy mapping environments), most standard measures for relating GSD
and map scale to the final mapping accuracy were inherited from
photogrammetric practices using scanned film.

• New mapping processes and methodologies have become much more

sophisticated with advances in technology and advances in our knowledge of
mapping processes and mathematical modeling.

• Mapping accuracy can no longer be associated with the camera geometry and
flying altitude alone (focal length, xp, yp, B/H ratio, etc.).

118
Why mapping accuracy can no longer be associated with the camera
geometry and flying altitude alone (focal length, xp, yp, B/H ratio, etc.).

Traditional Stereo Geometry Today’s Stereo Geometry

B B Traditional Stereo Geometry ok for digital
Intersection at a point correlation
Good for manual stereo
compilation

119
Digital Imagery and Increased overlap

GCP shows in 45 images 120

Digital Multi-Rays Photogrammetry

Least squares finds the

best point of
intersection not human
eyes

121
Why mapping accuracy can no longer be associated with the camera
geometry and flying altitude alone (focal length, xp, yp, B/H ratio, etc.)
The New ASPRS Standard is sensor agnostic data driven
Focal
Camera Length Flying Resulting
(mm) Altitude (ft) GSD (cm)
ADS80 62.77 2,363 7.5
DMC IIe 230 92.00 4,042 7.5
UltraCAM Falcon Prime 100.00 4,100 7.5
UltraCAM Eagle 210 210.00 9,937 7.5

If you pick a long focal length, plan for tight work flow and very accurate ABGPS and
aerial triangulation

122
ASPRS Standard uses Root Mean Square Error (RMSE)
• RMSE is not Standard deviation or sigma, they are different.
• Root Mean Square Error (RMSE) is computed as follow:

RMSE = √ ( Σ ( Z – Zi )2 / n)
Where,
Z = Measured Value from the data
Zi = Control Value (field surveyed)
n = number of measurements

123
Why RMSE and not Standard Deviation?

StDEV =3 cm. StDEV =3 cm.

Bias =7 cm.
RMSE =10
cm.

= RMSE =3
cm.

Precise, not Accurate, and

Accurate precise
Both cases result in the same StDEV
but different RMSE

(Remember: Mean or Average does not always mean Bias)

124
 New Standard Highlights

Horizontal Accuracy Standards for Geospatial Data

(unlimited horizontal accuracy classes)

Horizontal
Horizontal Orthoimagery
Accuracy at 95%
Accuracy RMSEx and Mosaic Seamline
RMSEr (cm) Confidence Level
Class RMSEy (cm) Mismatch
(cm)
(cm)

X-cm ≤X ≤1.41*X ≤2.45*X ≤ 2*X

= Radial RMSE = Circular RMSE = Two-

dimensional RMSE of X & Y
125
 Aerial Triangulation Accuracy Requirements

1. Aerial triangulation results should be twice as accurate as the

generated products:

Ortho and planimetric maps ONLY:

RMSEx(AT) or RMSEy(AT) = ½ * RMSEx(Map) or RMSEy(Map)
RMSEz(AT) = RMSEx(Map) or RMSEy(Map) of orthoimagery

For ortho/planimetric maps and elevation maps:

RMSEx(AT), RMSEy(AT) or RMSEz(AT) = ½ * RMSEx(Map), RMSEy(Map)or RMSEz(DEM)

* according to the ASPRS Positional Accuracy Standards for Digital Geospatial Data

126
 Ground Control Points Accuracy Requirements
2. Control points for aerial triangulation should be twice as
accurate as aerial the triangulation:

For ortho and planimetric maps ONLY:

RMSEx or RMSEy = 1/4 * RMSEx(Map) or RMSEy(Map),
RMSEz = 1/2 * RMSEx(Map) or RMSEy(Map)

For ortho/planimetric maps and elevation maps:

RMSEx, RMSEy or RMSEz= 1/4 * RMSEx(Map), RMSEy(Map) or RMSEz(DEM)

* according to the ASPRS Positional Accuracy Standards for Digital Geospatial Data

127
 Horizontal Accuracy
Orthoimage Mosaic
Horizontal Accuracy Class RMSEr at the 95%
Seamline Maximum
RMSEx and RMSEy (cm) (cm) Confidence Level
Mismatch (cm)
(cm)
0.63 0.9 1.3 1.5
1.25 1.8 2.5 3.1
2.50 3.5 5.0 6.1
5.00 7.1 10.0 12.2
Common Horizontal 7.50 10.6 15.0 18.4
Accuracy Classes 10.00 14.1 20.0 24.5
according to the new 12.50 17.7 25.0 30.6
standard 15.00 21.2 30.0 36.7
17.50 24.7 35.0 42.8
20.00 28.3 40.0 49.0
22.50 31.8 45.0 55.1
25.00 35.4 50.0 61.2
27.50 38.9 55.0 67.3
30.00 42.4 60.0 73.4
45.00 63.6 90.0 110.1
60.00 84.9 120.0 146.9
75.00 106.1 150.0 183.6
100.00 141.4 200.0 244.8
150.00 212.1 300.0 367.2
200.00 282.8 400.0 489.5
250.00 353.6 500.0 611.9
300.00 424.3 600.0 734.3
500.00 707.1 1000.0 1223.9
1000.00 1414.2 2000.0 2447.7 128
 Recommended
Common Horizontal
Accuracy Class Orthoimage RMSEx
Orthoimag
RMSEx and and RMSEy in terms of Recommended use
ery Pixel
RMSEy (cm) pixels
Sizes

≤1.3 ≤1-pixel Highest accuracy work

Standard Mapping and
Recommended 1.25 cm
2.5 2-pixels
GIS work
Visualization and less
Digital Orthoimagery ≥3.8 ≥3-pixels
accurate work
≤2.5 ≤1-pixel Highest accuracy work
Accuracy Examples for 5.0 2-pixels
Standard Mapping and
2.5 cm GIS work
Current Large and Medium ≥7.5 ≥3-pixels
Visualization and less
accurate work
Format Metric Cameras ≤5.0 ≤1-pixel Highest accuracy work
Standard Mapping and
10.0 2-pixels
5 cm GIS work
Visualization and less
≥15.0 ≥3-pixels
accurate work
≤7.5 ≤1-pixel Highest accuracy work
Standard Mapping and
15.0 2-pixels
7.5 cm GIS work
Visualization and less
≥22.5 ≥3-pixels
accurate work
≤15.0 ≤1-pixel Highest accuracy work
Standard Mapping and
30.0 2-pixels
15 cm GIS work
Visualization and less 129
≥45.0 ≥3-pixels 129 accurate work
Horizontal Accuracy/Quality Examples for High Accuracy Digital Planimetric Data

Equivalent to map
ASPRS 2014
scale in
Horizontal Horizontal
Accuracy Accuracy at Approximate GSD ASPRS ASPRS Equivalent to
RMSEr
Class RMSEx the 95% of Source Imagery 1990 1990 map scale in
(cm)
and RMSEy Confidence (cm) Class 1 Class 2 NMAS
(cm) Level (cm)
0.63 0.9 1.5 0.31 to 0.63 1:25 1:12.5 1:16
1.25 1.8 3.1 0.63 to 1.25 1:50 1:25 1:32
2.5 3.5 6.1 1.25 to 2.5 1:100 1:50 1:63
5.0 7.1 12.2 2.5 to 5.0 1:200 1:100 1:127
7.5 10.6 18.4 3.8 to 7.5 1:300 1:150 1:190
10.0 14.1 24.5 5.0 to 10.0 1:400 1:200 1:253
12.5 17.7 30.6 6.3 to12.5 1:500 1:250 1:317
15.0 21.2 36.7 7.5 to 15.0 1:600 1:300 1:380
17.5 24.7 42.8 8.8 to 17.5 1:700 1:350 1:444
20.0 28.3 49.0 10.0 to 20.0 1:800 1:400 1:507
22.5 31.8 55.1 11.3 to 22.5 1:900 1:450 1:570
25.0 35.4 61.2 12.5 to 25.0 1:1000 1:500 1:634
27.5 38.9 67.3 13.8 to 27.5 1:1100 1:550 1:697
30.0 42.4 73.4 15.0 to 30.0 1:1200 1:600 1:760
130
 Common questions I often faced with
1. We can not afford 20 check points small projects, how should we
handle a situation like this?

2. How accurate do the checkpoints need to be? What formula(s)

would we use to verify our accuracy?

3. Can we use other sources for checkpoints?

– Such as 3” Ortho-photos from Statewide Orthoimagery Program.
Can we scale horizontal control from these photos?
– What about other sources such as LiDAR?
131
 Best Practices in Determining Product Accuracy*
1. Check data should not be used in calibrating the tested
products:
ü Totally independent check points
2. Check data must be more accurate than tested data:
ü 3 times more accurate
3. Check data must be well distributed around the project
area
4. Check data must be a valid statistical sample:
ü Minimum of 20 check points for orthos
ü Minimum of 25 check points for elevation data
* according to the ASPRS Positional Accuracy Standards for Digital Geospatial Data
132
 Common questions I often faced with

1. We can not afford 20 check points small projects, how should we

handle a situation like this?

Answer: For valid statistical sample, you need minimum of 20 check

points. This is statistical science and theory of sampling, we can not
bend it.

However, fewer points for small projects is better than no check points
at all.
133
 Common questions I often faced with
2. How accurate do the checkpoints need to be? What formula(s)
would we use to verify our accuracy?

Answer:
• Check point should be 3 times more accurate than the
tested product

• Check data must be well distributed around the project

area

134
 Common questions I often faced with
3. Can we use other sources for checkpoints?
– Such as 3” Ortho-photos from Statewide Orthoimagery Program
Answer:
According to ASPRS 2014 standards, 3” (7.5-cm) orthos (usually with 6”
(15-cm) accuracy) is only suitable to QC orthos/maps with 18” (45-cm)
accuracy or coarser
– What about other sources such as LiDAR?
Answer:
– No you can’t use the lidar intensity to QC the orthos as the intensity
images are of lower resolution, usually 1-m image resolution. You can
use the lidar points cloud to check your elevation data, i.e. vertical QC.
135
 Horizontal accuracy requirements for elevation data
Photo identifiable points on intensity image, is it reliable?
• Yes, if the intensity image with high resolution
• No, if it is a standard 1 meter GSD intensity image

Image courtesy: USGS and Woolpert

136
The new ASPRs Standards on number of check points

137
 New Standard Highlights

Vertical Accuracy Standards for Geospatial Data

(unlimited vertical accuracy classes)

Absolute Accuracy Relative Accuracy (where applicable)

Vertical Within- Swath Swath-to-Swath Swath-to-Swath

RMSEz NVA at 95%
Accuracy VVA at 95th Hard Surface Non-Vegetated Non-Vegetated
Non- Confidence
Class Percentile Repeatability Terrain Terrain
Vegetated Level
(cm) (Max Diff) (RMSDz) (Max Diff)
(cm) (cm)
(cm) (cm) (cm)

X-cm ≤X ≤1.96*X ≤3.00*X ≤0.60*X ≤0.80*X ≤1.60*X

138
Examples on Vertical Accuracy and Recommended Lidar Point
Density for Digital Elevation Data according to the new ASPRS
2014 standard
Absolute Accuracy
Recommended Recommended
Vertical RMSEz NVA
Minimum NPD Maximum NPS7
Accuracy Class Non-Vegetated at 95% Confidence
(pts/m2) (m)
(cm) Level (cm)
1-cm 1.0 2.0 ≥20 ≤0.22
2.5-cm 2.5 4.9 16 0.25
5-cm 5.0 9.8 8 0.35
10-cm 10.0 19.6 2 0.71
15-cm 15.0 29.4 1 1.0
20-cm 20.0 39.2 0.5 1.4
33.3-cm 33.3 65.3 0.25 2.0
66.7-cm 66.7 130.7 0.1 3.2
100-cm 100.0 196.0 0.05 4.5
333.3-cm 333.3 653.3 0.01 10.0

10-cm with 2 pts/m2 is the QL2 LiDAR

standard for the nationwide 3DEP 139
USGS Lidar Base Specifications v 1.2 and later

140
 Horizontal accuracy requirements for elevation data
What to do?
• Trust the manufacturer estimate for horizontal accuracy assuming you are
meeting the vertical accuracy

• Use the ASPRS2014 estimation of horizontal accuracy for lidar

Altitude (m) Positional RMSEr (cm) Altitude (m) Positional RMSEr (cm)

500 13.1 3,000 41.6

1,000 17.5 3,500 48.0

1,500 23.0 4,000 54.5

2,000 29.0 4,500 61.1 Most QL2

flown at this
2,500 35.2 5,000 67.6 altitude

141
 Horizontal accuracy requirements for elevation data
according to ASPRS2014 Standards
• Photogrammetric elevation data: the horizontal accuracy equates to the
horizontal accuracy class that would apply to planimetric data or digital
orthoimagery produced from the same source imagery, using the same
aerial triangulation/INS solution.

• Lidar elevation data: use the following formula:

tan
0.55894170

See Page A7, Section 7.5 Horizontal Accuracy Requirements for Elevation Data

142
 Testing and Reporting Horizontal Products Accuracy*
Independent Accuracy Testing Statement:
“This data set was tested to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data
(2014) for a ___ (cm) RMSEx / RMSEy Horizontal Accuracy Class. Actual positional accuracy was found
to be RMSEx = ___ (cm) and RMSEy = ___ cm which equates to Positional Horizontal Accuracy = +/-
___ at 95% confidence level.”

Producer Accuracy Testing Statement:

“This data set was produced to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data
(2014) for a ___ (cm) RMSEx / RMSEy Horizontal Accuracy Class which equates to Positional Horizontal
Accuracy = +/- ___ cm at a 95% confidence level.”

* according to the ASPRS Positional Accuracy Standards for Digital Geospatial Data

143
 Testing and Reporting Vertical Products Accuracy*
Independent Accuracy Testing Statement:
“This data set was tested to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data
(2014) for a___ (cm) RMSEz Vertical Accuracy Class. Actual NVA accuracy was found to be RMSEz =
___ cm, equating to +/- ___ cm at 95% confidence level. Actual VVA accuracy was found to be +/-
___ cm at the 95th percentile.”

Producer Accuracy Testing Statement:

“This data set was produced to meet ASPRS Positional Accuracy Standards for Digital Geospatial
Data (2014) for a ___ cm RMSEz Vertical Accuracy Class equating to NVA =+/-___cm at 95%
confidence level and VVA =+/-___cm at the 95th percentile”

* according to the ASPRS Positional Accuracy Standards for Digital Geospatial Data

144
How Accurate UAS Products Are?

It Can Be Very Accurate

145
 UAS Accuracy Assessment CASE I:

A C

D E

F G
146
How Accurate are the UAS-derived Products?
Imagery GSD = 2.7 cm Altitude 350 ft. AGL

Processing Scenario
Accuracy Term
A B C D E F G
Number of Control Points 29 0 4 5 7 9 13
Number of Check Points 20 49 45 44 42 40 36
RMSE E (ft.) 0.22 2.34 0.16 0.18 0.17 0.18 0.18
RMSE N (ft.) 0.18 1.40 0.14 0.14 0.14 0.14 0.15
Horizontal Radial RMSE N,E (ft.) 0.29 2.73 0.21 0.23 0.22 0.23 0.24
Vertical RMSE Elev. (ft.) 0.32 1.62 1.35 0.32 0.23 0.25 0.29
Horizontal Accuracy at 95% (ft.) 0.49 4.72 0.36 0.40 0.39 0.39 0.41
Vertical Accuracy at 95% (ft.) 0.62 3.17 2.65 0.63 0.45 0.49 0.57

Can we obtain < 0.2 ft.

vertical accuracy?
Perhaps if we fly it from an
147
altitude of 150 to 200 ft. AGL
Can we meet DOTs Vertical Accuracy Specs ?

Radial RMSE N,E (ft.) 0.29 2.73 0.21 0.23 0.22 0.23 0.24
RMSE Elev. (ft.) 0.32 1.62 1.35 0.32 0.23 0.25 0.29

Ohio DOT Specifications

Maximum Allowable Maximum Allowable
DTM Accuracy Class Recommended Use
Average Dz (feet) RMSE (feet)

Class A Paved areas ± 0.07 0.16

Vegetated areas outside of pavement
that are maintained at a minimum
Class B biannual frequency (i.e.: farm fields, ± 0.25 0.32
residential yards, roadside R/W,
etcetera)

Class C Vegetated areas that are not maintained ± 0.50 0.5

Areas where vertical accuracy is not

Class D critical or warranted (i.e.: planning ± 1.00 1.00
engineering projects) 148
 Can we meet DOTs Horizontal Accuracy Specs ?

Radial RMSE N,E (ft.) 0.29 2.73 0.21 0.23 0.22 0.23 0.24
RMSE Elev. (ft.) 0.32 1.62 1.35 0.32 0.23 0.25 0.29

Ohio DOT Specifications

Planimetric Maximum Allowable

Recommended Use
Accuracy Class RMSE (ft)

Projects that require Class I planimetric features listed

Class I in Appendix A to be identified and mapped (ie: design 0.3
engineering projects)

Projects that require Class II planimetric features listed

Class II in Appendix A to be identified and mapped (ie: 1.0
planning studies)
149
 Accuracy Assessment CASE II:

B C D E F G
1.3 miles

600 ft.
150
 How Accurate are the Renaissance-derived Products?

Processing Scenario
Accuracy Term
A B C D E F G
Number of Control Points 0 4 6 8 10 21 38
Number of Check Points 38 34 32 30 28 17 0
RMSE E (ft.) 4.47 0.23 0.16 0.18 0.13 0.05 0.05
RMSE N (ft.) 1.89 0.26 0.20 0.14 0.14 0.07 0.05
Horizontal Radial RMSE N,E (ft.) 4.86 0.35 0.26 0.23 0.19 0.08 0.06
Vertical RMSE Elev. (ft.) 13.51 0.54 0.71 0.40 0.35 0.26 0.17
Horizontal Accuracy at 95% (ft.) 8.40 0.60 0.45 0.39 0.34 0.14 0.11
Vertical Accuracy at 95% (ft.) 26.49 1.05 1.40 0.78 0.69 0.52 0.34

The winner: Pair of GCPs

every 500 to 700 ft. along
the route

151
 UAS Accuracy Assessment CASE III:
Intersection of Petersburg and Overman Roads, Ohio

152
Executed Flights 100 ft. AGL Altitude

153
Executed Flights 300 ft. AGL Altitude

154
 2013 and 2017 Check Points

Green – 2013 Red - 2017 155

Mobile Mapping (MMS) Data

156
 Check points with MMS Data

Created five points along a cross section of the road every 25 ft.
157
Accuracy Evaluation According to 2013 Ground Controls
lidar DSM accuracy according to 2013 controls
Number of Check Points 79
Mean Error 0.023 ft. 0.007 m
Standard Deviation (StDEV) 0.037 ft. 0.011 m
Root Mean Squares Error (RMSEz) 0.043 ft. 0.013 m
NSSDA Vert Accuracy at 95% Confidence Level 0.085 ft. 0.026 m

100 ft. DSM Accuracy according to 2013 check points

Number of Check Points 73
Mean Error 0.085 ft. 0.026 m
Standard Deviation (StDEV) 0.130 ft. 0.040 m
Root Mean Squares Error (RMSEz) 0.154 ft. 0.047 m
NSSDA Vert Accuracy at 95% Confidence Level 0.302 ft. 0.092 m

300 ft. DSM Accuracy according to 2013 check points

Number of Check Points 73
Mean Error 0.067 ft. 0.020 m
Standard Deviation (StDEV) 0.290 ft. 0.088 m
Root Mean Squares Error (RMSEz) 0.296 ft. 0.090 m
NSSDA Vert Accuracy at 95% Confidence Level 0.579 ft. 0.177 m
158
 Comparing MMS lidar Data to UAS Data

Lidar DSM Versus UAS DSM

Number of Check Points 528 509 509
DSM300 - DSM100 DSM300 - lidar DSM100 - lidar
Mean Error -0.058 ft. 0.030 ft. 0.080 ft.
Standard Deviation (StDEV) 0.327 ft. 0.247 ft. 0.124 ft.
Root Mean Squares Error
0.332 ft. 0.249 ft. 0.147 ft.
(RMSEz)
NSSDA Vert Accuracy at 95%
0.651 ft. 0.488 ft. 0.289 ft.
Confidence Level

Established 528 positions to compare MMS points cloud to UAS points cloud
159
 Accuracy Reporting According to ASPRS Standard

Independent Vertical Accuracy Testing Statement:

“This data set was tested to meet ASPRS Positional Accuracy Standards for
Digital Geospatial Data (2014) for a 10-cm RMSEz Vertical Accuracy Class.
Actual NVA accuracy was found to be RMSEz = 4.7-cm, equating to +/- 9.2 cm
at 95% confidence level. Actual VVA accuracy was found to be +/- N/A cm at
the 95th percentile.”

160
 What does ASPRS standards mean for UAS products?
Required accuracy for the products:
Ortho Accuracy: 4 cm (RMSEx or y)
DSM Accuracy: 4 cm (RMSEz)

ASPRS Standards Requires:

RMSEx, RMSEy or RMSEz (ground control)= ¼ * RMSEx(Map), RMSEy(Map) or RMSEz(DEM)

Ground Control for AT accuracy = 1 cm (RMSEx,y,z)

Check points for QC accuracy = 1.33 cm

The Problem: People are claiming they can deliver 2-cm accuracy

The Question: Did you survey your controls to 0.5 cm?

Ask surveyors about what it takes to get this type of accuracy. No GPS
survey supports 0.5 cm accuracy

Fact: Usually control points are surveyed to 2 cm accuracy which support ortho
accuracy of 8-cm 161
 Visiting last year question..
Does Alaska needs different standards and guidelines?
Probably not, but we may need different interpretation of some of the
guidelines on ground controls and check points requirements to allow for
Alaska unique difficulties in performing ground surveying in some regions
of the state.

Examples:

1. Number of check points could be reduced based on terrain difficulty

and site accessibility, only if there is no way to survey 20 check points

2. Accuracy of check points could be relaxed to twice more accurate

than the tested products instead of three times
162
 Thank you!

Qassim Abdullah:
qassim.abdullah@woolpert.com
Mapping_matters@asprs.org