[Secret Project]

technical_review_#2.ppt

This file is one of Govdocs (http://digitalcorpora.org/corpora/govdocs)

The first page is added by NIST CFReDS project.
All following pages have no connection with to the scenario.
 Scientific Data Stewardship
Pilot Studies –
Engineering and Science
John Bates, NOAA
Chief, Remote Sensing Applications Division
NCDC, NESDIS
SDS – Climate data records, data sets, data provenance
 The Future CLASS
The Future - CLASS

NVDS CLASS Archive and Data Base

Web Interface Archive Archive

Archive Archive Data
Archive Archive Base
Archive Archive
Internet Web Access

Search Visualization Geospatial Satellite SDS

Image Browser

E
Archive and Distribution System (ADS) Interface
x
t Metadata Data Ingest
e
NESDIS e-Commerce System (NeS)

Management
r

System
n Order Entry and Inventory and Accounting and
Archive and Retrieval and
a Management Catalog Reporting
Migration Distribution
l
Order Entry and
D i-Store (Online Store) Reporting Management
a
t
a
Catalog with Catalog without
Pricing Pricing
S
o NeS Oracle CLASS Informix
Database Database
u
r
c Data Centers (Archive & Delivery - FTP and Manual) CLASS Archive & Delivery (FTP Only)
e
s

SDS Interface
 SDS Reprocessing
T-3 Suitland CLASS
Facility at NCDC Polarjet

Computers
Existing RSAD
T-3 BellSouth Internet 2.5GHz
440GB RAID (38% used)
T-3 UNCA/Abilene
Suomi
2.5GHz
T-1 Suitland #1
~20 Desktop 660GB RAID (20% used)

T-1 Suitland #2 PCs

Haze
4x2.8GHz
150 GB RAID (0% used)
NCDC Web Servers
Router
NCDC FTP Servers

New RSAD Linux Computers

New Linux Box 1
2x3.1GHz
2048GB RAID
NCDC LAN
Firewall Network Switch New Linux Box 2
100 Mb/s 2x3.1GHz
2048GB RAID

Server New Linux Box 3

2x3.1GHz
2048GB RAID

C3P0 Server
Tape Robot Systems

Fast access to HDSS New Linux Cluster

2x3.1GHz
R2D2 1Gb/s 1Gb/s 2x3.1GHz
Drizzle 2x3.1GHz
Access to B2 Data
IG88
Rime RAID
OU812 Access to HDSS DataServer 2048 GB
2048 GB
2x3.1GHz 2048 GB
2048 GB
 Scientific Data Stewardship
Reprocessing Facility at NCDC
• Need to scope data rates, CPU cycles,
software engineering, depositing
reprocessed climate data records into
CLASS
• This requires a set of pilot studies both at
NCDC and between ORA/CLASS Suitland
• Pilot study results will be incorporated into
concept of operations for NOAA’s climate
data records
 Pilot Study using Sampled Geo and
Leo data - ISCCP-B1 Processing
Sampled Processing Products …
data … sites …
1.2Tb Albedo
ISCCP 61 Gb/yr NCDC Example of internal processing
B1 RSAD Research NCDC will archive
Data Reprocessed Data products
Cloud Algorithms
Climatology Metadata
GEWEX …
ISCCP GPCP, GACP, …
e.g., from ISCCP

B1
Other Parameters
Access
Other
IDL & Fortran
Read, Collaborators
navigation and
Government (e.g., NASA/Langley),
Surface Radiation
calibration New data
routines
Universities, etc. Budget Parameters quality and
Data Characterization updated
Example of external processing B1 Metadata calibration
information

Current ability
Developing capability
 ISCCP B1 Data
• Total Volume ~1.2 TB (reduced from full res.)
• Temporal Data Coverage
– 1983 to present
• Spatial Data Coverage
– Global
• Satellites
– GOES, GMS, Meteosat, Insat (B2 Data)
– NOAA afternoon and morning polar orbiters (B2 data)
 Current data rates
Full Resolution ISCCP B1
Europe/Africa Meteosat-8 XX GB/day 72? MB/day
Meteosat-7 XX 36
Asia Meteosat-5 XX 36
Western Pac. GOES-9 14 72
Eastern Pac. GOES-10 14 72
Americas GOES-12 14 72

Total Global Rate per day XX 288MB/day

Total Global Rate per year 102Gb/yr

• ISCCP is reduced temporal (3hr) and spatial (10km) resolution

• All values are approximate
 ISCCP Data Access
• Data Access
– Decoders
• 18 satellites from 5 Satellite Processing Centers (SPCs) in 9
different formats
– Navigation
~ 5 navigation routines needed
– Calibration
• ISCCP collected a special dataset designed for satellite calibration
• The result: BT calibration data (which is specific for each image)
• Data Characterization
– Data Quality Assurance tests will quantify noise levels,
calibration, satellite movements, etc.
 Data Characterization
• Apply a set of data analysis tools to determine data
characteristics.
• For example:
– Means and standard deviations on varying temporal and spatial
scales
– Hovmoller diagrams
– First and second harmonics of annual cycle
– Empirical Orthogonal Function Analysis
– One-point Correlation Diagrams and Lag Correlation
– Frequency Histogram
– Power Spectra
– Cross Correlation and Cross Spectra
– Wavelet Analysis
– Wavenumber-Frequency
 B1 Metadata
• Quantify information about the data that
will facilitate reprocessing …
– Missing lines
– Data gaps
– Noise levels
– Satellite subpoints (and changes therein)
– Calibration
– Navigation
– Available channels
– Etc.
 ia n ce b
ias Satellite Geophysical I ns
tru
Rad rections m
rro ent
Cor Observed Variables cha e
ra c r
t er
st isti
F o r eca ss
e
Radiances Sp
cs
ir s t Gu res ectra
F
fun pons l
er a ctio e
Oth data ns
ri
prio
Inverse Model Forward Model
F-1(I) = q, T, … F(q, T, …) = I

Forward Problem
Inverse Problem

Understanding
Geophysical Simulated
retrievals
Satellite Radiances
Data

Observed Process studies

Observed
Geophysical of climate Radiances
Variables applications

Improved
Generic R.T. Climate
Prediction
 P
ISCC tion Geophysical I ns
tru
bra
Cali ation
B1 Visible & IR m
rro ent
rm Variables cha e
ra c r
Info Radiances t er
isti
cs
ou s Sp
G ase res ectra
P r
NCE bsorbe fun pons l
A ts ctio e
m oun ) ns
A 3
O,O
(H2
LUT retrieval Look Up SBDART
F-1(I) =  F(, U(H2O), …) = I

Forward Problem
Table
Inverse Problem

Understanding
Albedo Simulated
retrievals
Satellite Radiances
Data

Process studies
Observed Observed
of climate Radiances
Albedo
applications

Albedo Improved
Climate
Retrieval Prediction Generic RT
 Possible Collaborators
• When the B1 dataset is available, many studies will be
able to make use of the advantages of this dataset:
– Global observations
– 3 hour temporal resolution
– VIS and IR observations (and sometimes more)
– Long term observations (1983-present)
• Also this work can be extended to earlier satellite
observations. For example:
– 1979-1983 TIROS-N, NOAA-6, SMS, GOES
– Early Meteosats
• Possible collaborators include:
– Other government institutions
– Academia
 Satellite Coverage

• Missing NOAA-16/17
• Need to Color code GEOs by location
Polar Imager and Sounder 1b Data
Rates (per satellite-day)
Sounder Full Sampled Imager Full Sampled
Resolution Resolution Resolution Resolution
VTPR 7 MB VHRR 800 MB

TOVS 70 MB AVHRR 1 GB

ATOVS 300 MB AVHRR 1 GB

AIRS 30 GB MODIS 600 GB

IASI 50 GB AVHRR 1 GB

CrIS 50 GB VIIRS 600 GB

Polar Imager and Sounder History
 HIRS Processing I ns
S tru
HIR Bias
ce
(also applies to MSU/AMSU) cha e
m
rro ent
ian ra c r
Rad rections t er
Cor Cloud Cleared
isti
HIRS Sampled ERA40 cs
Limb corrected l1b T(p), q(p), and Ch IRS
H
er a Calibrated a
Oth data ozone spe nnel
ri ct r
prio a
Inverse Model Forward Model
Neural network Other Retrieval MODTRAN

Forward Problem
F-1(I) = q, T, … methods F(q, T, …) = I
Inverse Problem

Understanding
Retrievals of Simulated Radiances
T(p), q(p), …
HIRS NN Training
Data

studies of Atmospheric Observed

Observed
Temperature and HIRS Channel
T(p), q(p), …
Humidity Variability Radiances

QC/QA Analysis

Archived with Corresponding Metadata

 WSR88D
Level II Level III
(~70 TB/year) (~1.5 TB/year)

QC/QA
QC/QA
QPE

Data Resolution
NCDC Processing Time

Radar Rainfall
Climatology
 QC/QA
Processing Time/
Clutter Complexity
Suppression

Bright Band
Contamination

AP Identification

Advection Effect
Correction
Range Effect
Correction
Quantitative Precipitation Estimation
Clean In Situ
Reflectivity Estimate Data

Parameter Estimation

Radar-Gauge

Supplemental
Information
Inverse Model Satellite Rainfall
Z=ARb Estimates

Regridding
Polar to Cartesian

Temporal Integration
5-min, 15-min,Hourly
Radar Rainfall Climatology
Mosaic

Regional
Accumulations

Large-Scale
Long-Term
Accumulations
 Level II
• Full Volume Scan Data
• Variables – Reflectivity,
Radial Velocity, Spectral
Width
• 4–19 Elevation Angles
• 5, 6, 10 Minute Resolution
• 1o x 1km Spatial
Resolution
• Radar Reflectivity
0.5 dBZ precision
Level III
• WSR-88D Site Processing
• 20 + Products -Reflectivity
and Precipitation Products
(QPE)
• 4 Elevation Angles
• 5,6,10 Minute Resolution
• 1o x 1km, 2 x 2km, 4 x 4km
Spatial Resolution
• Data Resolution Degraded
(4 or 5 dBZ precision)
 Data Size
• Level II 500
GB/radar/year
• Level III 10
GB/radar/year
• 140 + Radars in
CONUS
• Possible - 70 TB/year
Level II
• Possible - 1.5 TB/year
level III
 QC/QA
• Quality Issue • Reason • Algorithm - Complexity

Ground AP/Clutter Threshold

Return
Neural
Bright Band Network
Large Reflectivity
Value Image
Processing
Hail

Under Advection
Effect
Estimation
Radar Sampling Correction

Vertical
Profile
Adjustment
NEXRAD Data Volume (NCDC)
NEXRAD Data Storage

600

500
5 TB/Month or 60 TB/Year

400
Terabytes of Data

300

200

100

Start of NEXRAD
0
Jan-93 Jan-94 Jan-95 Jan-96 Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03
Year