Professional Documents
Culture Documents
Agile Data Warehouse Design For Big Data Presentation (720p - 30fps - H264-192kbit - AAC) PDF
Agile Data Warehouse Design For Big Data Presentation (720p - 30fps - H264-192kbit - AAC) PDF
!2
Introduction
• a2c
• Boutique EDM (Enterprise Data Management)
consultancy firm:
• Data Warehousing
• Master Data Management
• Closed Look Analytics and Visualization
• Data & Application Architecture
• John DiPietro
• Principal, Chief Technology Officer
• Jim Stagnitto
• Data Warehouse & MDM Architect
!3
a2c Corporate Overview
& Industry Experience
!4
Company Overview
• Technology Solution Consultancy headquartered in Philadelphia with
regional offices in New York and Boston
• Servicing Healthcare, Life Science, Tel-Com and Financial Services
industries with recent obtainment of our GSA schedule to pursue Federal
Government opportunities
• Consultant base of over 2500 proven IT professionals throughout the North
East Region with a recruiting network which provides national coverage
• Flexible approach to helping our clients with their initiatives
• Project-based Solutions
• Staff Augmentation
• Managed Service Offerings – “On-Shore QA , Development & Application Support”
• Executive & Professional Search
!5
Competitive Advantage
• Founders of a2c were part of the fastest growing privately held IT consulting and staff
augmentation firm in the US from 1994-2002. Our Executive Management Team has over a
100 years collective experience and been responsible for delivering over a half-billion dollars
of IT Consulting and staff augmentation revenue from 1994 through to the present day.
• a2c’s Recruiting Engine and Methodology is one of the best in the industry, capable of
producing quality results, on-demand for our clients
• Resource Managers continually “Silo” disciplines with available candidates whom have
proven their abilities with us over the last 10 years
• Our solutions organization is instrumentally involved during the screening and selection
process to ensure that candidates submitted to our clients are an ideal match
• a2c’s Culture provides an ability to attract and retain the best talent in the industry and fosters
creativity, integrity, growth and teamwork
• a2c provides our clients with an alternative solution to a “Big 4” consultancy at substantial
savings for projects that are between $500K and $5M due to our flexibility, agility and focus
!6
Representative Clients
03/19/12
!7
a2c Solution Engagement Structures
!8
a2c Solutions Capabilities
• Enterprise Data Management Practice helps clients manage their complete Information
Lifecycle from their On-line Transactional systems to their Data Warehousing, Enterprise
Reporting, Data Migration, Back-Up and Recovery Strategies (See Slide 7)
• Business Architecture & Optimization Practice utilizes “Six Sigma Lean” methodologies to
analyze, re-engineer and automate our client’s business processes to leverage human
workflow and business rules engine technologies to create efficiencies and provide
business unit owners with the necessary metrics to continually improve performance
• Program Management Office oversees all aspects of solutions planning and delivery
across client engagement teams and provides the methodology and frameworks which
are based on PMI® industry standards
• Application Development & Managed Services Practice helps clients architect, implement
and deploy the latest Microsoft and Enterprise Java based applications which are built on
proven frameworks and architectures for the enterprise
• a2c's SDLC Delivery Model is comprised of over 20 years collective best practices and
industry proven methodologies that allow our delivery teams to rapidly design, develop
and implement solutions. Our SDLC model has been designed to complement our project
management methodology, utilizing iterative development cycles that enable project
teams to provide consistently high quality, on-time deliverables, regardless of technology
platform
!9
Agile DW Design
Overview
!10
Modeling for End Users
• How to Design to Answer
Business Questions?
• Think about how questions are articulated
• And how the answers should be
deliveredIdentify a common question
framework
!11
How Do We Ask Questions?
When What Who
!12
How Do We Ask Questions?
• Events / Transactions
• e.g. Sale
• a immutable "fact" that occurs in a time and (typically a)
place
• Interrogatives:
• Who, What, When, Where, Why
• Descriptive context that fully describes the event
• a set of “dimensions" that describe events
!13
Dimensional Value Proposition
• It makes sense to present answers to people using the same
taxonomy of events and interrogatives (aka: facts and dimensions
- dimensional structure) that they use when forming questions
• Events are instances of processes :
• It’s best to present information to people who will ask the system
questions in dimensional form
• This is true regardless of the type of information being
interrogated, it’s source, or IT stuff (like database technologies
utilized)
• It’s best to model this presentation layer based on the events (aka:
business processes) that underlie the questions
!14
How
Wh o
en W h
How
Many
re Wh
h e at
W
Why
!15
Scenarios
!16
Kimball Dimensional DW
Dimensional BI Semantic Layer
Source Data
(Structured)
!17
Kimball with Big Data
Dimensional BI Semantic Layer
!18
Corporate Information Factory (CIF)
Dimensional Tier
(Virtual or Physical)
Source Data
(Structured)
!19
CIF with Big Data
Dimensional BI Semantic Layer
Dimensional Tier
(Virtual or Physical)
!20
Data Vault
Dimensional BI Semantic Layer
Dimensional Tier
(Virtual or Physical)
Data Vault
Source Data
(Structured)
!21
Data Vault with Big Data
Dimensional BI Semantic Layer
Dimensional Tier
(Virtual or Physical)
!22
Etc.
!23
Common Framework
Dimensional BI Semantic Layer
Dimensional Tier
[Physical (Kimball) or Virtual (CIF or Data Vault)
Unstructured ->
Persistant Un/ Persistent Structured Data Insight
Structured
Semi-Structured Repository Generation /
Data Discovery
Staging Area (not needed for Kimball) Data Mining
Processing
Un/Semi-Structured Data
Structured Data Movement
Movement
!25
eCommerce Example: Clickstream
Raw Clickstream Data!
Semi-Structured 25 52 164 240 274 328 368 448 538 561 630 687 730 775 825
834
39 120 124 205 401 581 704 814 825 834
Recording of every page request 35 249 674 712 733 759 854 950
made by a user 39 422 449 704 825 857 895 937 954 964
15 229 262 283 294 352 381 708 738 766 853 883 966 978
26 104 143 320 569 620 798
Includes some structural elements – 7 185 214 350 529 658 682 782 809 849 883 947 970 979
such as when the request was 227 390
71 192 208 272 279 280 300 333 496 529 530 597 618 674 675
made and who the user is 720 855 914 932
183 193 217 256 276 277 374 474 483 496 512 529 626 653 706
878 939
Requires significant prep work in 161 175 177 424 490 571 597 623 766 795 853 910 960
order to fit into a traditional row- 125 130 327 698 699 839
392 461 569 801 862
based relational database 27 78 104 177 733 775 781 845 900 921 938
101 147 229 350 411 461 572 579 657 675 778 803 842 903
71 208 217 266 279 290 458 478 523 614 766 853 888 944 969
Apples and Oranges: Pre- 43 70 176 204 227 334 369 480 513 703 708 835 874 895
Sessionized Page Visits, Detailed 25 52 278 730
Product Views, Catalogue 151 432 504 830 890
71 73 118 274 310 327 388 419 449 469 484 706 722 795 810
Requests, Shopping Cart Adds / 844 846 918
Deletes / Abandons, etc. 130 274 432 528 967
188 307 326 381 403 523 526 722 774 788 789 834 950 975
89 116 198 201 333 395 653 720 846
Needs to be converted into 70 171 227 289 462 538 541 623 674 701 805 946 964
143 192 317 471 487 631 638 640 678 735 780 865 888 935
seperate-but-relatable dimensional 17 242 471 758 763 837 956
facts - with many shared 52 145 161 283 375 385 676 721 731 790 792 885
182 229 276 529
(conformed) dimensions 43 522 565 617 859
!26
Typical Clickstream “Page View” Dimensional
Model
When What
What
Why Who
!27
eCommerce Example: Web Sales
!28
eCommerce Dimensionality
Referring Promotion Activity
Facts (below) & Time! Customer! Web Page! Product!
URL! / Type
Dimensions (right) (When) (Who) (Where) (What) Campaign
(Where) (How)
(Why)
View Start Current
View End
Page Visit Session
Visitor Previous ✔
Next
Start
Session End
View Start
Detailed Product Current
View End Prospect Previous ✔ ✔
View Session Next
Start
Session End
Shopping Cart Activity Start
Prospect ✔ ✔ ✔ ✔
Activity Activity End
Sale Start
Sale (Checkout) Sale End
Customer ✔ ✔ ✔ ✔
Customer
Shipment
Shipment / Delivery Delivery
Delivery ✔
Recipient
!29
Agile DW Design
Overview
!30
The first dimensional modeler:
Rudyard
RalphR.K.
Kimball?
Kipling
!31
I keep six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who…
–Rudyard Kipling
!32
!32
Who
!33
What
!34
When
!35
Where
!36
Why
!37
How
!38
How Many
!39
The 7Ws
Framework
How
Wh o
en Wh
How
Many
re Wh
he at
W
Why
How did we get here?
DW Architectures: A Brief History
£$€
Where
What
Location
Product
Geographic
?? Why
Service
Store
Causal
Transactions
Ship To
Promotion
Hospital Reason
Weather
Competition
BEAM
How
Wh o
en Wh
How
Many
e Wh
her at
W
Why
Business Event Analysis & Modeling
How
do you design a data warehouse?
Tech Design Artifacts?
CALENDAR PRODUCT
Date Key Product Key
Date Product Code
Day Product Description
Day in Week Product Type
Day in Month SALES FACT Brand
Day in Qtr Subcategory
Date Key
Day in Year Category
Product Key
Month
Store Key
Qtr
Promotion Key
Year
Weekday Flag
Quantity Sold
Holiday Flag
Revenue
Cost
Basket Count
STORE PROMOTION
Store Key Promotion Key
Analysis
Design
Development
This Year Next Year
BDUF Test
Release
Stakeholder
Requirements Data
ETL BI VALUE?
Input Model
DATA
Agile DW/BI Development
Stakeholder interaction
JEDUF
? ETL
BI
Prototyping
Review
Release
DATA
State of The
DW Field
Solid:
Dimensional Data Warehouse Design is Mature
Proven Design Patterns Exist for Common
Requirements
Hit or Miss:
Collecting Unambiguous and Thorough
Requirements
Slotting Requirements into Proven Design
Patterns
End-User Ownership and Validation
Too Often: Snatching Defeat from the Jaws of
Victory
!52
Modelstorming
Quick Inclusive
Data
Modeler BI Stakeholders
Interactive Fun
BEAM✲ Methodology
Structured, non-technical, collaborative working
conversation directly with BI Users
BEAM✲
• Logical and Physical
(Kimball-esque)
• BI User’s Business
Dimensional Data Models
Process, Organizational,
• Example data
Hierarchical, and Data
Knowledge
• Detailed and Testable ETL
Specification
• Focused Data Profiling
• Instantiated DW
Data
BI Stakeholders Prototype
Modeler
Requirements =
Design
55
Collaboration at Every
Step
Agile Data Modeling Requirements
!57
What
kind of model?
CALENDAR PRODUCT
Date Key Product Key
Date
Product Code
Day
Product Description
Day in Week
Product Type
Day in Month
SALES FACT Brand
Day in Qtr
Subcategory
Day in Year
Date Key
Category
Month
Product Key
Qtr
Store Key
Year
Promotion Key
Weekday Flag
Holiday Flag
Quantity Sold
Revenue
Cost
Basket Count
STORE PROMOTION
Store Code
Promotion Code
Store Name
Promotion Name
URL
Promotion Type
Store Manager
Discount Type
Region
Ad Type
Country
Holiday Type Customer Type
Month Country
Calendar Customer
Sales Fact
Store Product
City Category
64
Collaborative / Conversational Design
BEAM✲
BI Users
Modeler Subjects Verb Objects
Design Using Natural Language
!66
“Spreadsheet”-like Models
Event Table Name (filled in later)
Verb
Interrogative
Details
Subject-Verb-Object 1
1
1
1 2
3 1
1
1
1
1
When 1
1
Declare Event Type 1
6
1 Initial Data Examples
Where 1
1
1
1
7
1
How 1
(many) 1
1
1 Quantities - Facts
8
1
1
Why 1
1
1
9
1
Sufficient Detail Fact 1
Granularity How 1
Capture Example Data
verb on/at/every
SUBJECT OBJECT EVENT
DATE
Old, Low Old, Low Value Oldest needed Near Min, Negative, 0
New, High New, High Most Recent, Future Far Max, Precision Exceptional Exceptional
Illustrate exceptions
Detailed ETL
Specification
Identify Event Type Early
Adjust Conversation Based on Event Type
!72
Capture When Details
!77
Modeling Dimensions
Annotate w Targeted Data Profiling
Proceed Through the Business Process Value Chain
Collaborative Dimension Conformance
Sales
Campaigns
Dimensions
Identify Hierarchy Types
Simple
Complex
Graphically Depict Hierarchies
Visualize The Hierarchies
Paint The Organization
Prototype! Not “Data Model Review”
Recap
• Collaborative and Agile
• Data Modeling
• Data Sourcing
• Data Conformance
• Requirements = Design
• Slots directly into proven and mature dimensional data warehousing
design patterns
!87
If you have been affected by
any of the issues raised
in this presentation
!
Agile Data Warehouse Design
Lawrence Corr, Jim Stagnitto, Decision Press, November 2011
!
Questions / Comments