Opensap Hana1 Warmup

Lecture:
Trends and Concepts in the

Software Industry

Introduction

Dr.-Ing. Jrgen Mller

Course Statistics
. hup://www.openPl.de
. 13,126 paruclpanLs, 4.161 acuve
conLrlbuLors ln dlscusslons
. 6 course modules for 6 weeks
. 33 lndlvldual module lLems (vldeos /
slldes / readlng maLerlal)
. = 300 exam / asslgnmenL / self-LesL
quesuons
. 309 slldes expllclLly creaLed for Lhls course
. 213 pages book readlng maLerlal
. = 8h cuL vldeo maLerlal (12h raw maLerlal)
. 1eachlng 1eam: 4 colleagues
2
!
#!
$!
%!
&!
'!!
!
'!!!
#!!!
(!!!
$!!!
)
*
*
+

'

)
*
*
+

#

)
*
*
+

(

)
*
*
+

$

)
*
*
+

,

)
*
*
+

%

-
.
/
0

1203*4 56 7/489:;/<=> :< ?>>:@<0*<=> A -./0
?B*4/@* C*>2D=> :< E
Goals
ueep Lechnlcal undersLandlng of a column-orlenLed dlcuonary-
encoded ln-memory daLabase and lLs appllcauon ln enLerprlse
compuung

. 1he fuLure of enLerprlse compuung
. loundauons of daLabase sLorage Lechnlques
. ln-memory daLabase operaLors
. Advanced daLabase sLorage Lechnlques
. loundauons for a new enLerprlse appllcauon developmenL era
3
December 2006: The Basic Idea
4
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Chapter 1:
The Future of
Enterprise Computing

New Requirements for
Enterprise Computing
Sensors
1rac|ng harmaceunca| ackages |n Lurope
! 13 bllllon packages / 34 bllllon read evenLs per year
! ulsLrlbuLed reposlLorles for sLorlng read evenLs
! 8eferences Lo read evenLs are sLored ln cenLral dlscovery servlce

Mon|tor|ng I1 car erformance
! 8eLween 300 and 600 sensors per car
! Muluple evenLs per second per sensor
! 1racklng every Crand rlx lap or LesL run
8
A|rp|ane Ma|ntenance at 8oe|ng
! MalnLenance workers aL 8oelng wrlLe reporLs aer repalrs
! 8eporLs and menuoned parL numbers geL lndexed
! Analyucs reveal whlch parLs ln oLher planes may be defecuve

Med|ca| D|agnos|s at Char|t
! uocLors wrlLe medlcal reporLs aer every dlagnosls
! ulagnosls and menuoned sympLoms geL lndexed
! Comparlson wlLh slmllar cases for opumal LreaLmenL
Combination of Structured
and Unstructured Data
9
Mobile
Mob||e |nverts trad|nona| corporate structures

! Lnables cusLomer-faclng
personnel
! 8esponse umes < 1 second
! Lxample: uunnlng
Sales, Servlce
Cperauons
ConLrolllng
Consolldauon
SLraLegy
Consumers / CusLomers
CLC
10
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Enterprise Application
Characteristics

1ransacnona|
Data Lntry

Sources: Machlnes, 1ransacuonal
Apps, user lnLeracuon, eLc.

1ext Ana|yncs,
Unstructured Data

Sources: web, soclal, logs,
supporL sysLems, eLc.
Lvent rocess|ng,
Stream Data

Sources: machlnes, sensors,
hlgh volume sysLems
kea|-nme Ana|yncs,
Structured Data

Sources: 8eporung, Classlcal
Analyucs, plannlng, slmulauon
uaLa
ManagemenL
!"#$
&'()*+!,-. / !012.3
4056 4.',-7
Challenge:
Diverse Applications
13
! Modern enLerprlse resource plannlng (L8) sysLems are
challenged by m|xed work|oads, lncludlng CLA-sLyle
querles. lor example:
" CL1-sLyle: creaLe sales order, lnvolce, accounung documenLs,
dlsplay cusLomer masLer daLa or sales order
" CLA-sLyle: dunnlng, avallable-Lo-promlse, cross selllng,
operauonal reporung (llsL open sales orders)
! 8uL: 1oday's daLa managemenL sysLems are opumlzed
e|ther for dally transacnona| or ana|ynca| workloads
sLorlng Lhelr daLa along rows or columns
Cn|lne 1ransacuon
rocesslng
Cn|lne Analyucal
rocesslng
OLTP vs. OLAP
14
Drawbacks of
the Separation
! CLA sysLem does noL have Lhe |atest daLa
! CLA sysLem does only have predehned subset of Lhe daLa
! Cost-|ntens|ve L1L process has Lo synch boLh sysLems
! 1here ls a loL of redundancy
! D|erent data schemas lnLroduce complexlLy for
appllcauons comblnlng sources
13
OLTP Access Pattern
Myth
! Workload analysls of enLerprlse sysLems shows:
CL1 and CLA workloads are noL LhaL dlerenL
16
Comb|ne CL1 and CLA data
uslng modern hardware and daLabase sysLems
Lo creaLe a s|ng|e source of truth,
enable rea|-nme ana|yncs and
s|mp||fy appllcauons and daLabase sLrucLures.

Addluonally,
! exLracuon, Lransformauon, and loadlng (L1L) processes
! pre-compuLed aggregaLes and maLerlallzed vlews become
obsoleLe.
Vision
17
! Many columns are noL used even once
! Many columns have a low cardlnallLy of values
! nuLL values/defaulL values are domlnanL
! Sparse dlsLrlbuuon faclllLaLes hlgh compresslon
SLandard enLerprlse soware daLa ls sparse and w|de
Enterprise Data
Characteristics
18
Low Cardinality of Values
Within Many Columns
8esulLs from analyzlng nanclals
ulsuncL values ln accounung documenL headers
(99 aurlbuLes)
CC Loglsucs
Plgh Lech ulscreLe
manufacLurlng
8anklng

19
Many Columns are not
Used Even Once
SS unused columns per company ln average
40 unused columns across all companles
0%
10%
20%
30%
40%
50%
60%
70%
80%
1 - 32 33 - 1023 1024 - 100000000
13%
9 %
78%
24%
12%
64%
Number of Distinct Values
Inventory Management
Financial Accounting
%

o
f

C
o
l
u
m
n
s
20
Wide Tables
Analysls of wldLh of 144 mosL used* Lables
* LargesL ln Lerms of cardlnallLy
0
3
10
13
20
23
30
1
-
9

1
0
-
1
9

2
0
-
2
9

3
0
-
3
9

4
0
-
4
9

3
0
-
3
9

6
0
-
6
9

7
0
-
7
9

8
2

9
9

1
1
0
-
1
1
9

1
2
0
-
1
2
9

1
3
8

1
4
0
-
1
4
9

1
3
6

1
8
0
-
1
8
9

2
0
0
-
2
0
9

2
3
0

3
1
2

3
9
9

#

1
a
b
|
e
s

# Co|umns
21
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Changes in Hardware

Advances in Hardware
! 64 blL address space - 418
ln currenL server boards
! 23C8/s daLa LhroughpuL,
Cu - u8AM
! CosL-performance rauo
rapldly decllnlng
! Mulu-Core ArchlLecLure
8 x (8-16) core Cu per blade
! arallel scallng across blades
! Cne blade =$30.000 =
1 LnLerprlse Class Server
A
24
Copy S0G8 data v|a Inhn|band - 10s
23
CPU
Registers
Main Memory
Flash
Hard Disk
H
i
g
h
e
r

P
e
r
f
o
r
m
a
n
c
e
L
o
w
e
r

P
r
i
c
e

/

H
i
g
h
e
r

L
a
t
e
n
c
y
CPU
Caches
Memory Hierarchy
Latency Numbers
L1 cache reference (cached data word) 0.Sns
8ranch mlspredlcL 3ns
L2 cache reference 7ns
MuLex lock/unlock 23ns
Ma|n memory reference 100ns 0.1s
Send 2k byLes over 1 Cb/s neLwork 20,000ns 20s
SSu random read 130,000ns 130s
8ead 1 M8 sequenually from memory 230,000ns 230s
D|sk seek 10,000,000ns 10ms
Send packeL CA Lo neLherlands Lo CA 130,000,000ns 130ms
26
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
A Blueprint of SanssouciDB

SanssouciDB: An In-Memory
Database for Enterprise Applications
Main Memory
at Blade !
Log
Snapshots Passive Data (History)
Non-Volatile
Memory
Recovery Logging
Time
travel
Data
aging
Query Execution Metadata TA Manager
nterface Services and Session Management
Distribution Layer
at Blade !
Main Store Differential
Store
Active Data
M
e
r
g
e C
o
l
u
m
n
C
o
l
u
m
n
C
o
m
b
i
n
e
d
C
o
l
u
m
n
C
o
l
u
m
n
C
o
l
u
m
n
C
o
m
b
i
n
e
d
C
o
l
u
m
n
ndexes
nverted
Object
Data Guide
ln-Memory uaLabase (lMu8)
! uaLa resldes permanent|y ln
maln memory
! Maln Memory ls Lhe pr|mary
89.-$5$:.61.;
! Sull: logglng Lo d|sk/
recovery from d|sk
! Maln memory access ls
Lhe new bou|eneck
! Cache-consclous algorlLhms/
daLa sLrucLures are cruc|a|
(locallLy ls klng)

Main Memory
at Server i
Distribution Layer
at Server i
Chapter 2:
Foundations of
Database Storage Techniques
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Dictionary Encoding

Motivation

! Maln memory access ls Lhe new bouleneck
! Compresslon reduces number of l/C operauons Lo
maln memory
! Cperauon dlrecLly on compressed daLa
! Csemng wlLh blL-encoded xed-lengLh daLa Lypes
! 8ased on llmlLed value domaln
33
Dictionary Encoding
Example
! 8 bllllon humans
! AurlbuLes:
" rsL name
" lasL name
" gender
" counLry
" clLy
" blrLhday
! 200 byLe per Luple
! Lach aurlbuLe ls dlcuonary encoded
34
Sample Data
33
recID fname |name gender c|ty country b|rthday
. . . . . . .
39 !ohn SmlLh m Chlcago uSA 12.03.1964
40 Mary 8rown f London uk 12.03.1964
41 !ane uoe f alo AlLo uSA 23.04.1976
42 !ohn uoe m alo AlLo uSA 17.06.1932
43 eLer SchmldL m oLsdam CL8 11.11.1973
. . . . . .
1able: world_populauon
Dictionary Encoding
a Column
! A column ls spllL lnLo a dlcuonary and an aurlbuLe vecLor
! ulcuonary sLores all dlsuncL values wlLh lmpllclL valuelu
! AurlbuLe vecLor sLores valuelus for all enLrles ln Lhe column
! osluon ls sLored lmpllclLly
! Lnables osemng wlLh blL-encoded xed-lengLh daLa Lypes
36
recID fname
. .
39 !ohn
40 Mary
41 !ane
42 !ohn
43 eLer
. .
D|cnonary for "fname"
va|ueID Va|ue
. .
23 !ohn
24 Mary
23 !ane
26 eLer
. .
Aur|bute Vector for "fname"
pos|non va|ueID
. .
39 23
40 24
41 23
42 23
43 26
. .
Querying Data using
Dictionaries
Search for AurlbuLe value
(l.e. reLrleve all persons wlLh fname Mary")

1. Search valuelus for requesLed value (Mary")
2. Scan AurlbuLe vecLor for valuelu (24")
3. 8eplace valuelus ln resulL wlLh correspondlng
dlcuonary value
37
Sorted Dictionary
! ulcuonary enLrles are sorLed elLher by Lhelr numerlc value or
lexlcographlcally
! ulcuonary lookup complexlLy: C(log(n)) lnsLead of C(n)
! ulcuonary enLrles can be compressed Lo reduce Lhe amounL
of requlred sLorage
! Selecuon crlLerla wlLh ranges are less expenslve
38
Compression Rate
! uepends on cardlnallLy / enLropy
! CardlnallLy
" 1able cardlnallLy: number of Luples ln a relauon
" Column cardlnallLy: number of dlsuncL values ln a column
! LnLropy
" measure for lnformauon denslLy
" LnLropy = column cardlnallLy / Lable cardlnallLy
39
Data Size Examples
40
Co|umn Card|-
na||ty
8|ts
Needed
Item S|ze |a|n S|ze S|ze w|th D|cnonary
(D|cnonary + Co|umn)
Compress|on
Iactor
llrsL
names
3 mllllons 23 blL 30 8yLe 373C8 238.4M8 + 21.4C8 = 17
LasL
names
8 mllllons 23 blL 30 8yLe 373C8 381.3M8 + 21.4C8 = 17
Cender 2 1 blL 1 8yLe 7C8 2.0b + 933.7M8 = 8
ClLy 1 mllllon 20 blL 30 8yLe 373C8 47.7M8 + 18.6C8 = 20
CounLry 200 8 blL 47 8yLe 330C8 9.2k8 + 7.3C8 =47
8lrLhday 40000 16 blL 2 8yLe 13C8 78.1k8 + 14.9C8 = 1
1ota|s 200 8yte = 1.618 = 92G8 = 17
Data Layout in Main Memory

Basics
0x0 0xFFFF FFFF FFFF FFFF
FFFF FFFF FFFF FFFF
FFFF FFFF FFFF FFFF
...
1he memory layouL ls only llnear, every hlgher-dlmenslonal
access ls mapped Lo Lhls llnear band.
42
Row Data Layout
! uaLa ls sLored Luple-wlse
! Leverage co-locauon of aurlbuLes for a slngle Luple
! Low cosL for reconsLrucuon, buL hlgher cosL for sequenual
scan of a slngle aurlbuLe
C B A C B A C B A C B A
C B A C B A C B A C B A
Column Operation
Row Operation
Row Row Row
Row Row Row Row
43
Columnar Data Layout
! uaLa ls sLored aurlbuLe-wlse
! Leverage sequenual scan-speed ln maln memory
! 1uple reconsLrucuon ls expenslve
C C C C B B B B A A A A
C C C C B B B B A A A A
Column Operation
Row Operation
Column Column Column
Column Column Column
44
Chapter 3:
In-Memory Database Operators
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Insert

Example
recID fname |name gen
der
country c|ty b|rthday
0 Marun AlbrechL m CL8 8erlln 08-03-1933
1 Mlchael 8erg m CL8 8erlln 03-03-1970
2 Panna Schulze f CL8 Pamburg 04-04-1968
3 AnLon Meyer m Au1 lnnsbruck 10-20-1992
4 Sophle Schulze f CL8 oLsdam 09-03-1977
... ... ... ... ... ... ...
Lxample 1able: world_populauon
lnSL81 ln1C world_populauon
vALuLS (karen, Schulze, f, CL8, 8osLock, 06-20-2012)
48
INSERT Without
New Dictionary Entry
fname |name gender country c|ty b|rthday
... ... ... ... ... ... ...
lnSL81 ln1C world_populauon vALuLS (karen, Schulze, f, CL8, 8osLock, 06-20-2012)
0 0
1 1
2 3
3 2
4 3
u Av
AurlbuLe vecLor (Av)
ulcuonary (u)
0 AlbrechL
1 8erg
2 Meyer
3 Schulze
49
1. Look-up on dlcuonary ! enLry found
ulcuonary (u)
u
... ... ... ... ... ... ...
Av
0 0
1 1
2 3
3 2
4 3
INSERT Without
0 AlbrechL
1 8erg
2 Meyer
3 Schulze
30
0 0
1 1
2 3
3 2
4 3
3 3
1. Look-up on dlcuonary ! enLry found
2. Append valuelu Lo aurlbuLe vecLor
ulcuonary (u)
3 Schulze
... ... ... ... ... ... ...
Av
0 AlbrechL
1 8erg
2 Meyer
3 Schulze
u
INSERT Without
31
3 Schulze
... ... ... ... ... ... ...
0 0
1 0
2 1
3 2
4 3
u Av
0 8erlln
1 Pamburg
2 lnnsbruck
3 oLsdam
ulcuonary (u)
INSERT With
32
1. Look-up on dlcuonary ! no enLry found
u Av
0 0
1 0
2 1
3 2
4 3
0 8erlln
1 Pamburg
2 lnnsbruck
3 oLsdam
3 Schulze
... ... ... ... ... ... ...
ulcuonary (u)
INSERT With
33
2. Append new value Lo dlcuonary (no re-sorung necessary)
0 8erlln
1 Pamburg
2 lnnsbruck
3 oLsdam
4 8osLock
u
3 Schulze
... ... ... ... ... ... ...
ulcuonary (u)
Av
0 0
1 0
2 1
3 2
4 3
INSERT With
34
2. Append new value Lo dlcuonary (no re-sorung necessary)
3. Append valuelu Lo aurlbuLe vecLor
3 Schulze 8osLock
... ... ... ... ... ... ...
ulcuonary (u)
0 0
1 0
2 1
3 2
4 3
3 4
0 8erlln
1 Pamburg
2 lnnsbruck
3 oLsdam
4 8osLock
u Av
INSERT With
33
0 AnLon
1 Panna
2 Marun
3 Mlchael
4 Sophle
3 Schulze 8osLock
... ... ... ... ... ... ...
u Av
0 2
1 3
2 1
3 0
4 4
ulcuonary (u)
INSERT With
36
3 Schulze 8osLock
... ... ... ... ... ... ...
ulcuonary (u)
0 AnLon
1 Panna
2 Marun
3 Mlchael
4 Sophle
u Av
0 2
1 3
2 1
3 0
4 4
INSERT With
37
3 Schulze 8osLock
... ... ... ... ... ... ...
ulcuonary (u)
0 AnLon
1 Panna
2 Marun
3 Mlchael
4 Sophle
3 karen
u Av
0 2
1 3
2 1
3 0
4 4
INSERT With
2. Append new value Lo dlcuonary
38
3 Schulze 8osLock
... ... ... ... ... ... ...
ulcuonary (u)
0 AnLon
1 Panna
2 Marun
3 Mlchael
4 Sophle
3 karen
u (old) Av
0 2
1 3
2 1
3 0
4 4
0 AnLon
1 Panna
2 karen
3 Marun
4 Mlchael
3 Sophle
u (new)
INSERT With
3. SorL dlcuonary
39
3 Schulze 8osLock
... ... ... ... ... ... ...
ulcuonary (u)
Av (old)
0 AnLon
1 Panna
2 karen
3 Marun
4 Mlchael
3 Sophle
u (new) Av (new)
0 3
1 4
2 1
3 0
4 3
0 2
1 3
2 1
3 0
4 4
INSERT With
3. SorL dlcuonary
4. Change valuelus ln aurlbuLe vecLor
60
3. SorL dlcuonary
4. Change valuelus ln aurlbuLe vecLor
3. Append new valuelu Lo aurlbuLe vecLor
3 karen Schulze 8osLock
... ... ... ... ... ... ...
ulcuonary (u)
0 AnLon
1 Panna
2 karen
3 Marun
4 Mlchael
3 Sophle
u Av
0 3
1 4
2 1
3 0
4 3
3 2
INSERT With
61
Result
recID fname |name gender country c|ty b|rthday
4 ulrlke Schulze f CL8 oLsdam 09-03-1977
3 karen Schulze f CL8 8osLock 06-20-2012
... ... ... ... ... ... ...
lnSL81 ln1C world_populauon
vALuLS (karen, Schulze, f, CL8, 8osLock, 06-20-2012)
62
Lxample 1able: world_populauon
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Scan Performance

Scan Performance (1)
8 bllllon humans
! AurlbuLes:
" rsL name
" lasL name
" gender
" counLry
" clLy
" blrLhday
! 200 byLe

! Cuesuon: Pow many women, how many men?
! Assumed scan speed: 2M8/ms/core
63
Row
1
Row
2
Row
3
...
Row
8 x 10
First Name
Last Name
Gender
Country
City
Birthday
Table: humans
9
8ow SLore - LayouL
66
1ab|e: wor|d_popu|anon
Row
1
Row
2
Row
3
...
Row
8 x 10
First Name
Last Name
Gender
Country
City
Birthday
Table: humans
9
8ow SLore - LayouL
67
! 1able slze:
8 bllllon Luples
200 byLes per
Luple = 1.6 18

Row
1
Row
2
Row
3
...
First Name
Last Name Country
City
Birthday
Table: humans
Data loaded
and used
Data loaded
but not used
Row
8 x 10
9
Gender
! 1able slze:
8 bllllon Luples
200 byLes per
Luple = 1.6 18
! Scan Lhrough
all rows wlLh
2M8/ms/core
! 800 s
wlLh 1 core

8ow SLore - lull 1able Scan
68
Row
1
Row
2
Row
3
...
First Name
Last Name Country
City
Birthday
Table: humans
Data
not loaded
Data loaded
and used
Data loaded
but not used
Row
8 x 10
9
Gender
! 8 bllllon cache
accesses a 64 byLe
= S12 G8
! 8ead wlLh
2M8/ms/core
! 2S6 s
wlLh 1 core
8ow SLore - SLrlde Access Cender"
69
First Name
Last Name Country
City
Birthday
Table: humans
Gender
! AurlbuLe vecLor
slze = 91 G8
! ulcuonary slze
= 700 M8
! 1able slze = 92 G8
! Compresslon
facLor = 17
Column SLore - LayouL
70
First Name
Last Name Country
City
Birthday
Table: humans
Data
not loaded
Data loaded
and used
Data loaded
but not used
Gender
Column SLore - lull Column Scan Cender"
71
! Slze of aurlbuLe
vecLor Cender":
8 bllllon Luples
1 blL per Luple
= 1 G8
! Scan Lhrough
column wlLh
2M8/ms/core
! 0.S s wlLh 1 core
First Name
Last Name Country
City
Birthday
Table: humans
Data
not loaded
Data loaded
and used
Data loaded
but not used
Gender
! Slze of aurlbuLe
vecLor 8lrLhday":
8 bllllon Luples
2 byLe per Luple
= 16 G8
! Scan Lhrough
column wlLh
2M8/ms/core
! 8 s wlLh 1 core
Column SLore - lull Column Scan 8lrLhday"
72

Pow many women, how many men?

Scan Performance
Summary
Row Store
Column Store
Full table scan Stride access
Time in seconds 800 256 0.5
Factors compared
to best in class
1,600 512 1
73
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Tuple Reconstruction

Tuple Reconstruction (1)
! All aurlbuLes
are sLored
consecuuvely
! 200 byLe ! 4 cache
accesses a 64 byLe
! 2S6 byte
! 8ead wlLh
2M8/ms/core
! = 0.128 s
wlLh 1 core

Accesslng a record ln a row sLore
76
! All aurlbuLes are
sLored ln separaLe
columns
! lmpllclL record lus
are used Lo
reconsLrucL rows

77
vlrLual record lus
! 1 cache access for
each aurlbuLe
! 6 cache accesses
a 64 byLe
! 384 byte
! 8ead wlLh
2M8/ms/core
! = 0.192 s
wlLh 1 core
78
vlrLual record lus
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Select

SELECT Example
SELECT fname, lname FROM world_population
WHERE country=Italy and gender=m
|d fname |name country gender
2394 Clanlulgl 8uon lLaly m
3010 Lena Cercke Cermany f
3040 Marlo 8aloLelll lLaly m
3949 Manuel neuer Cermany m
4902 Lukas odolskl Cermany m
20102 klaas-!an PunLelaar neLherlands m
81
Query Plan
! redlcaLes are evaluaLed and generaLe posluon llsLs
! lnLermedlaLe posluon llsLs are loglcally comblned
! llnal posluon llsL ls used for maLerlallzauon
82
country = "Italy" gender = "m"
fname,
lname
position
list
position
list
positional
AND
!
Query Execution
os|non
0
2
os|non
0
2
3
4
3
os|non
0
2
country = 3 ("Ita|y")
gender = 1 ("m")
AND
Va|ueID D|cnonary
for "country"
0 Algerla
1 lrance
2 Cermany
3 lLaly
4 neLherlands
.
Va|ueI
D
D|cnonary
for "gender"
0 f
1 m
|d fname |name country gender
2394 Clanlulgl 8uon 3 1
3010 Lena Cercke 2 0
3040 Marlo 8aloLelll 3 1
3949 Manuel neuer 2 1
4902 Lukas odolskl 2 1
20102 klaas-!an PunLelaar 4 1
83
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Chapter 4:
Advanced Database
Storage Techniques
Differential Buffer

Motivation
! lnserung new Luples dlrecLly lnLo a compressed sLrucLure can be
expenslve
" new values may requlre a reorganlzauon of Lhe dlcuonary
" number of blLs requlred Lo encode all dlcuonary values may change,
aurlbuLe vecLor has Lo be reorganlzed
! ueleuon of Luples ls expenslve
" All aurlbuLe vecLors have Lo be reorganlzed, valuelus of followlng Luples
have Lo be ad[usLed
87
Differential Buffer
! new values are wrluen Lo a dedlcaLed dlerenual buer (delLa)
! Cache Sensluve 8+ Lree (CS8+) used for fasLer search on delLa
!"#$%
Main
Store
Differential
Buffer
D
a
t
a

M
o
d
i
f
y
i
n
g

O
p
e
r
a
t
i
o
n
s
R
e
a
d

O
p
e
r
a
t
i
o
n
s
Table
ulcuonary AurlbuLe vecLor
fname
.
(
c
o
m
p
r
e
s
s
e
d
)

Maln SLore
ulcuonary
AurlbuLe vecLor
CS8+
fname
.
ulerenual 8uer
WrlLe
8ead
world_population
0 0
1 1
2 1
3 3
4 2
3 1
0 AnLon
1 Panna
2 Mlchael
3 Sophle
0 Angela
1 klaus
2 Andre
0 0
1 1
2 1
3 2
8 8llllon enLrles
up Lo 30,000
enLrles
88
Differential Buffer
! lnserLs of new values are fasLer, because dlcuonary and
aurlbuLe vecLor does noL need Lo be resorLed
! 8ange selecL on dlerenual buer expenslve, based on
unsorLed dlcuonary
! ulerenual 8uer requlres more memory:
" no aurlbuLe vecLor compresslon
" addluonal CS8+ Lree for dlcuonary
89
ulerenual 8uer
Maln SLore
Tuple Lifetime
recId fname |name gender country c|ty b|rthday
3 Sophle Schulze f CL8 8osLock 06-20-2012
... ... ... ... ... ... ...
8 10
9
Zacharlas erdopolus m C8L ALhen 03-12-1979
Mlchael moves from 8erlln Lo oLsdam
uuA1L '<,-)=>9,9()0*,6?
SL1 15:7@A",:$=0'?
WPL8L B60'.@ 845120.); Anu )60'.@8C.-D;
90
ulerenual 8uer
Maln SLore
Tuple Lifetime
... ... ... ... ... ... ...
8 10
9
uuA1L '<,-)=>9,9()0*,6?
SL1 15:7@A",:$=0'?
WPL8L B60'.@ 845120.); Anu )60'.@8C.-D;
91
Tuple Lifetime
... ... ... ... ... ... ...
8 10
9
uuA1L '<,-)=>9,9()0*,6?
SL1 15:7@A",:$=0'?
WPL8L B60'.@ 845120.); Anu )60'.@8C.-D;
ulerenual 8uer
Maln SLore
0 M|chae| 8erg m GLk otsdam 03-0S-1970
92
Tuple Lifetime
! roblem: 1uples are now avallable ln Maln SLore and ulerenual
8uer
! 1uples of a Lable are marked by a valldlLy vecLor Lo reduce Lhe
requlred amounL of reorganlzauon sLeps
" Llke an aurlbuLe vecLor for valldlLy
! lnvalldaLed Luples sLay ln Lhe daLabase Lable, unul Lhe nexL
reorganlzauon Lakes place
! Search resulLs are reduced uslng Lhe valldlLy vecLor
! 1 blL requlred per daLabase Luple
93
recId fname |name gender country c|ty b|rthday va||d
0 Marun AlbrechL m CL8 8erlln 08-03-1933 1
1 Mlchael 8erg m CL8 8erlln 03-03-1970 0
2 Panna Schulze f CL8 Pamburg 04-04-1968 1
3 AnLon Meyer m Au1 lnnsbruck 10-20-1992 1
4 ulrlke Schulze f CL8 oLsdam 09-03-1977 1
3 Sophle Schulze f CL8 8osLock 06-20-2012 1
... ... ... ... ... ... ...
8 10
9
Zacharlas erdopolus m C8L ALhen 03-12-1979 1
Tuple Lifetime
uuA1L '<,-)=>9,9()0*,6?
SL1 15:7@A",:$=0'?
WPL8L B60'.@ 845120.); Anu )60'.@8C.-D;
0 M|chae| 8erg m GLk otsdam 03-0S-1970 1 ulerenual
8uer
Maln SLore
94
Tuple Lifetime
uuA1L '<,-)=>9,9()0*,6?
SL1 15:7@A",:$=0'?
WPL8L B60'.@ 845120.); Anu )60'.@8C.-D;
recId fname |name gender country c|ty b|rthday va||d
0 Marun AlbrechL m CL8 8erlln 08-03-1933 1
1 Mlchael 8erg m CL8 8erlln 03-03-1970 0
2 Panna Schulze f CL8 Pamburg 04-04-1968 1
3 AnLon Meyer m Au1 lnnsbruck 10-20-1992 1
4 ulrlke Schulze f CL8 oLsdam 09-03-1977 1
3 Sophle Schulze f CL8 8osLock 06-20-2012 1
... ... ... ... ... ... ...
8 10
9
Zacharlas erdopolus m C8L ALhen 03-12-1979 1
0 M|chae| 8erg m GLk otsdam 03-0S-1970 1 ulerenual
8uer
Maln SLore
93
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Merge

Handling Write
Operations
! All WrlLe operauons (lnSL81, uuA1L) are sLored wlLhln a
dlerenual buer (delLa) rsL
! 8ead-operauons on dlerenual buer are more expenslve Lhan
on maln sLore
! CuLdaLed Luples values ln maln sLore are lnvalldaLed
! 1o guaranLee opumal performance Lhe dlerenual buer needs
Lo be merged perlodlcally wlLh Lhe maln sLore
98
Merge Process
Overview
! 1he merge process ls worklng Lable wlse
! ConslsLs of:
1. repare merge
2. AurlbuLe merge
3. CommlL merge

! ls Lrlggered by:
" AmounL of Luples ln buer
" CosL model Lo
" Schedule
" 1ake query cosL lnLo accounL
99
Online Merge
! Worklng on daLa coples allows asynchronous merge
! very llmlLed lnLerrupuon due Lo shorL lock
! AL leasL Lwlce Lhe memory needed!
100
Main
Store
Differ-
ential
Buffer
Before
Data
Modifying
Operations
Read Operations
Main
Store
Differential
Buffer
(new)
During the Merge Process
Data
Modifying
Operations
Read Operations
Main
Store
(new)
Differ-
ential
Buffer
(new)
After
Data
Modifying
Operations
Read Operations
Main
Store
(new)
Differ-
ential
Buffer
Merge Operation
Table Table Table
Prepare Attribute Merge Commit
Attribute Merge
Step 1:
! LxLracL delLa dlcuonary from delLa column
! Merge maln and delLa dlcuonary, opuonally remove unused
values
! CreaLe mapplng lf valuelu changed

Step 2:
! CreaLe new maln column (lf needed)
! Maln sLore column aurlbuLe vecLor needs Lo be rewrluen lf
! ulcuonary lncreases over #blL value range
! SorL order changes

101
Example
102
7
0
2
3
4
1
5
6
3
1
2
5
0
4
anna
charlie
daniel
frank
hans
isabell
CSB+ Leafs
0
2
3
4
1
5
0
2
3
4
5
6
new valueID
M1
uncompressed
frank
bernd 0
1
3
hans 2
anna
charlie
daniel
frank
bernd
hans
isabell
Outputs Step 1 Inputs Step 1
B1
0
2
3
4
1
5
6
D1
Dictionary Entries
o
ld

v
a
lu
e
I
D
o
ld

v
a
lu
e
I
D
n
e
w

v
a
lu
e
I
D
Delta Position
Lists
4
2
3
2
0
1
1
5
valueID
Inputs Step 2
valueID
Output Step 2
1
1
5
4
5
3
4
3
0
2
2
6
UM1
UM2
4
2
3
2
0
1
1
5
valueID
1
4
5
new valueID
3
2
0
1
11
10
8
9
7
0
2
3
4
1
5
6
11
10
9
8
7
0
2
3
4
1
5
6
0
2
1
valueID
o
ld

v
a
lu
e
I
D
AV
d
AV
m
D
m
D
c
Dictionary Entries
AUX
m
AUX
d
AV
c
AV
m
AV
ds
bernd
hans
frank
bernd
3
1
2
0
7
6
4
0
2
3
5
1
0
2
1
0
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Logging and Recovery

What is Logging and Recovery?
! ln-memory daLabases need Lo provlde durablllLy guaranLees
Lo be used ln producuve enLerprlse appllcauons
! oLenual sLraLegles
" Logg|ng: wrlung log lnformauon Lo backup sLorage (covered here)
" Server rep||canon: run Lransacuons slmulLaneously ln prlmary and
repllcaLed daLabase
! 8ecovery ls Lhe process of brlnglng a sysLem back Lo a
producuve mode aer a sysLem fallure
! ueslrable characLerlsucs of any recovery mechanlsm:
" Mlnlmal lmpacL on runnlng sysLem
" lasL recovery wlLhouL any daLa loss
103
Logging Infrastructure for an
In-Memory Database
! SnapshoLs of prlmary
daLa sLores are wrluen
perlodlcally (e.g. durlng
merge process)
! Log les for updaLes
are wrluen beLween
snapshoLs Lo recover
Lhe daLabase
! ulcuonary-compressed
column logglng allows
parallel log le wrlung
and recovery

106
Snapshots
! A snapshoL wrlLes Lhe prlmary daLa of Lhe ln-memory
daLabase Lo perslsLenL memory
! urpose of snapshoLs:
" Speed-up of recovery process, as only Lhe log les aer Lhe
snapshoL need Lo be replayed
" Creauon of a sysLem copy
! A snapshoL ls ldeally wrluen aer each merge process
" SnapshoL conLalns new read-opumlzed sLore
" Changes ln Lhe delLa sLore are logged ln log les
107
Chapter 5:
Foundations for a New
Enterprise Application
Development Era
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Implications on Application
Development

Simplified Application
Development
1radluonal Column-orlenLed
Appllcauon
cache
uaLabase
cache
rebullL
aggregaLes
8aw daLa
! no caches needed
! no redundanL daLa/
ob[ecLs
! no malnLenance of
lndexes or aggregaLes
! uaLa movemenLs are
mlnlmlzed
111
SAP ERP Financials on
In-Memory Technology
ln-memory column daLabase for an enLerprlse sysLem
! Comblned workload (parallel CL1/CLA querles)
! Leverage ln-memory capablllues Lo
" 8educe amounL of daLa
" AggregaLe on-Lhe-y
" 8un analyuc-sLyle querles (Lo replace maLerlallzed vlews)
" LxecuLe sLored procedures
! use Case: SA Lk I|nanc|a|s soluuon
" osL and change documenLs
" ulsplay open lLems
" 8un dunnlng [ob
" Analyucal querles, such as balance sheeL
112
Current Financials
Solutions
113
Cnly base Lables, algorlLhms, and some lndexes
The Target
Financials Solution
114
BKPF
accounting documents
BSEG
sum tables
secondary indices
dunning data
change documents
CDHDR
CDPOS
MHNK
MHND
BSAD
BSAK
BSAS
BSID
BSIK
BSIS
LFC1
KNC1
GLT0
In-Memory Financials on
SAP ERP
113
BKPF
accounting documents
BSEG
116
In-Memory Financials on
SAP ERP (prototype)
Feasibility of Financials on
In-Memory Technology
! Modlcauons on SA llnanclals
" kemoved secondary lndlces, sum Lables and pre-calculaLed and maLerlallzed
Lables
" keduce code complexlLy and slmpllfy locks
" lnserL Cnly Lo enable h|story (change documenL replacemenL)
" Added stored procedures wlLh buslness funcuonallLy
! Luropean dlvlslon of a reLaller
" L8 2003 LCC 6.0 Lh3
" 3.3 18 sysLem daLabase slze
" llnanclals:
" 23 mllllon headers / 8 C8 ln maln memory
" 2S2 mllllon lLems / 30 C8 ln maln memory
(lncludlng lnverLed lndlces for [oln aurlbuLes and lnserL only exLenslon)
117
DBMS IMDB
BKPF 8.7 GB 1.5 GB
BSEG 255 GB 50 GB
Secondary Indices 255 GB -
Sum Tables 0.55 GB -
Complete 519.25 GB 51.5 GB
263.7 GB 51.5 GB
Reduction by a Factor 10
118
Dunning Run
! uunnlng run deLermlnes all open and due lnvolces
! CusLomer dened querles on 230M records
! CurrenL sysLem: 20 mln
! new loglc: 1.S sec
" ln-memory column sLore
" arallellzed sLored procedures
" Slmplled llnanclals
119
Why?
! 8elng able Lo perform Lhe dunnlng run ln such a shorL ume
|owers 1CC
! Add more funcuonallLy!
! 8un oLher [obs ln Lhe meanume! - ln a mulu-Lenancy cloud
seLup hardware musL be used wlsely
120
Bring Application Logic Closer
to the Storage Layer
! SelecL accounLs Lo be dunned, for each:
" SelecL open accounL lLems from 8Slu, for each:
" CalculaLe due daLe
" SelecL dunnlng procedure, level and area
" CreaLe MPnk enLrles
! CreaLe and wrlLe dunnlng lLem Lables

121

1 SELECT
10,000 SELECTs
10,000 SELECTs
31,000 Entries
122

123
1 SELECT
10,000 SELECTs
10,000 SELECTs
31,000 Entries
Cne s|ng|e
stored procedure

1 SELECT
10,000 SELECTs
10,000 SELECTs
31,000 Entries
Cne s|ng|e
stored procedure
Ca|cu|ated on-the-y
124
Results
125
Hardware: 4 CPUs x 6 Cores (Intel Dunnington), 256GB RAM
Customer Data: 250mio line items, 380k open, 200k due
# Cperanon nANA2 Vers|on Var|ant 2 Var|ant 3
1 SelecL Cpen lLems 0.63s
1.01s
(lncl. 1047 kn83 !oln)
0.6s
(lncl. 1047 kn83 !oln)
2 uue daLe, dunnlng level 27s deferred Lo aggregauon 0.3s
3 lllLer 1 (verlfy uunnlng levels) = 19s 1.1s 0.3s
4 lllLer 2 (Check LasL uunnlng) = 13s 0.8s 0.4s
3 CeneraLe MPnk (AggregaLe) done ln #1 1.2s done ln #1
6 CeneraLe MPnu (LxecuLe lllLers) done ln #1 140ms done ln #1
1ota| = 1 M|nute
= 3.0s
(#3, #4 exec. |n para||e|)
= 1.Ss
(#3, #4 exec. |n para||e|)
Crlglnal verslon needed abouL 20 mlnuLes
! lacLor 800x accelerauon achleved
Dunning Application
126
Dunning Application
127
Available-to-Promise
Check
! Can l geL enough quanuues of a requesLed producL on a
deslred dellvery daLe?
! Coal: Analyze and valldaLe Lhe poLenual of ln-memory and
hlghly parallel daLa processlng for Avallable-Lo-romlse (A1)
! Challenges
" uynamlc aggregauon
" lnsLanL reschedullng ln mlnuLes vs. nlghLly baLch runs
" 8eal-ume and hlsLorlcal analyucs
! CuLcome
" 8eal-ume A1 checks wlLhouL maLerlallzed vlews
" Ad-hoc reschedullng
" no maLerlallzed aggregaLes

128
In-Memory Available-to-Promise
129
Demand Planning
! llexlble analysls of demand
plannlng daLa
! Zoomlng Lo choose granularlLy
! lllLer by cerLaln producLs or
cusLomers
! 8rowse Lhrough ume spans
! Comblnauon of locauon-based
geo daLa wlLh plannlng daLa ln an
ln-memory daLabase
! LxLernal facLors such as Lhe
LemperaLure, or Lhe level of
cloudlness can be overlald Lo
lncorporaLe Lhem ln plannlng
declslons
130
GORFID
! PAnA for SLreamlng uaLa rocesslng
! use Case: ln-Memory 8llu uaLa ManagemenL
! Lvaluauon of SA CL8
! roLoLyplcal lmplemenLauon of:
" 8llu 8ead LvenL 8eposlLory on PAnA
" ulscovery Servlce on PAnA (10 bllllon
daLa records wlLh ca. 3 seconds response ume)
" lronLends for lhone, lad2
! key llndlngs:
" PAnA ls sulLed for sLreamlng daLa
(uslng bulk lnserLs)
" Analyucs on sLreamlng daLa ls now posslble
131
Near Real-Time as a
Concept
Discovery Service
Read Event
Repositories
Verication
Services
SAP HANA
! !
up to 8,000 read
event notications
per second
up to 2,000
requests
per second
Discovery Service
Read Event
Repositories
Verication
Services
SAP HANA
! !
P
A
up to 8.000 read
event notications
per second
up to 2.000
requests
per second
Discovery Service
Read Event
Repositories
Verication
Services
SAP HANA
! !
P
A
up to 8.000 read
event notications
per second
up to 2.000
requests
per second
P
A
Bulk load every
2-3 seconds:
> 50,000 inserts/s
132
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Views

Dynamic Views
Excel SAP
Business
Objects
Explorer
Any Software
Presentation Layer
View
View
View View View
...
View
View Layer
(Calculations, Filter, ...)
Persistency Layer
(Main Memory)
Object Hierarchy
Node Tables Node Tables Node Tables
Node Tables
View Other DB
Logical
Log
i
i i i i
DB
Persistence
Store
Restart
Write Complete
Objects
Recovery
133
Learning Map
Ioundanons for a
New Lnterpr|se
App||canon
Deve|opment Lra
Ioundanons of
Database Storage
1echn|ques
1he Iuture of
Lnterpr|se
Compunng
Advanced
Database
Storage
1ech-
n|ques
In-Memory
Database
Cperators
Bypass Solution

Bypass Solution Allows a Smooth
Transition to In-Memory Technology
! Mllllons of old" un-opumlzed llnes of code aL Lhe
cusLomers' slLe 1ransluon requlred
! 8ow-sLore replacemenL
! arL-for-parL replacemenL wlLh bypass
! 1ransform row-sLore Lo column-sLore on Lhe y
! Change of appllcauon code
140
OLAP
Engine
Traditional DB
w/ Cubes
Bypass Solution for Transition
ERP
Traditional DB
ETL
Traditional BI Todays System
141
BIA
SAP
Excel
.
.
.

Business
Objects
OLAP
Engine
ETL
Traditional DB
IMDB

SSD
Traditional BI Todays System with IMDB
ERP
Traditional DB
w/ Cubes
BIA
SAP
Excel
.
.
.

Business
Objects
STEP 1: Install and run the in-memory database in parallel
New
Applications
OLAP
Engine
ETL
Traditional DB
IMDB

SSD
Traditional BI Todays System with IMDB
ERP
Traditional DB
w/ Cubes
BIA
SAP
Excel
.
.
.

Business
Objects
STEP 2: Generate business value from the first day on
New
Applications
OLAP
Engine
EL
Traditional DB
IMDB

SSD
Traditional BI with IMDB Todays System with IMDB
ERP
IMDB
w/o cubes
BIA
SAP
Excel
.
.
.

Business
Objects
STEP 3: Re-create traditional BI w/o cubes in IMDB
New
Applications
OLAP
Engine
EL
IMDB

SSD
ERP
IMDB
w/o Cubes
BIA
SAP
Excel
.
.
.

Business
Objects
SAP
Business
Objects
Excel
BI 2.0
.
.
.

CLA
Lnglne
Traditional DB
Traditional BI with IMDB Todays System with IMDB
STEP 4: Introduce next-gen BI running in parallel without materialized views
New
Applications
IMDB

SSD
ERP
SAP
Business
Objects
Excel
BI 2.0
.
.
.

CLA
Lnglne
Traditional DB
Todays System with IMDB
STEP 5: Eliminate all the traditional BI, virtualize all in-memory BI, using non-materialized views
New
Applications
IMDB

SSD
OLTP & OLAP
ERP Future
Releases
SAP
Business
Objects
Excel
BI 2.0
.
.
.

CLA
Lnglne
Future System with IMDB
STEP 5: Eliminate all the traditional BI, virtualize all in-memory BI, using non-materialized views
STEP 6: Run ERP and BI on IMDB

Opensap Hana1 Warmup

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Opensap Hana1 Warmup

Uploaded by

Copyright:

Available Formats

Lecture:

Trends and Concepts in the

You might also like