Professional Documents
Culture Documents
Astronomy, Petabytes, and MySQL
Astronomy, Petabytes, and MySQL
MySQL
MySQL Conference
Santa Clara, CA
April 16, 2008
Kian-Tat Lim
Stanford Linear Accelerator Center
Outline
LSST
LSST Database
LSST Database + MySQL
2 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST
What Is It?
Why Build It?
3 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST
What Is It?
Why Build It?
4 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Telescope
Proposed
telescope to be
built in Chile
5 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Large
6 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Synoptic Survey
Wide
Deep
Fast
7 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST
What Is It?
Why Build It?
8 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Dark Matter and Energy
Photo: J. A. Tyson, W. Colley, E. 9 / 47
MySQL Conference
L. Turner, and NASA April 16, 2008 Santa Clara, CA
Variable Objects
10 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Transient Objects
11 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Moving Objects
What’s In It?
How Big?
How Often?
What Queries?
Unusual Needs
13 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database
What’s In It?
How Big?
How Often?
What Queries?
Unusual Needs
14 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Database: Components
Moving
Objects Object Catalog
Catalog
Provenance
Source Catalog Statistics
Summaries
Image Metadata
15 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Astronomical Objects
Moving
Objects Object Catalog
Catalog
Provenance
Source Catalog Statistics
Summaries
Image Metadata
16 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Sources
Moving
Objects Object Catalog
Catalog
Provenance
Source Catalog Statistics
Summaries
Image Metadata
17 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Changes
Moving
Objects Object Catalog
Catalog
Provenance
Source Catalog Statistics
Summaries
Image Metadata
18 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Image Metadata
Moving
Objects Object Catalog
Catalog
Provenance
Source Catalog Statistics
Summaries
Image Metadata
19 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Calibration and Facility
Moving
Objects Object Catalog
Catalog
Provenance
Source Catalog Statistics
Summaries
Image Metadata
20 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database
What’s In It?
How Big?
How Often?
What Queries?
Unusual Needs
21 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Sagans of Rows
49 billion objects
22 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Lots of Columns
(for now)
23 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Database Size
Grows to >14 PB
24 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database
What’s In It?
How Big?
How Often?
What Queries?
Unusual Needs
25 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Frequency
Nightly updates
Semi-annual data
releases
26 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database
What’s In It?
How Big?
How Often?
What Queries?
Unusual Needs
27 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Queries
What’s In It?
How Big?
How Often?
What Queries?
Unusual Needs
29 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Unusual Needs
Flexibility
Provenance
30 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database + MySQL
Why MySQL?
Scalability?
Performance?
31 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database + MySQL
Why MySQL?
Scalability?
Performance?
32 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
MySQL
Relational database
management system
33 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Open Source
Vibrant community
34 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Hardware
Runs on commodity
hardware
35 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
In-Memory Tables
36 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database + MySQL
Why MySQL?
Scalability?
Performance?
37 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
“MySQL Grid”
38 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Partitioning
41 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
LSST Database + MySQL
Why MySQL?
Scalability?
Performance?
42 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Per-Column Indexing
2X data size
43 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Needs: Optimizer
44 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Needs: Indexes
Bitmap/compressed
indexes
45 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA
Needs: Storage Engine
Building a petabyte DB
47 / 47
MySQL Conference
April 16, 2008 Santa Clara, CA