Lec 31-36 PDF

Query Processing &
Optimization
Professor Navneet Goyal

Department of Computer Science & Information Systems
BITS, Pilani
Topics
n Overview
n Measures of Query Cost
n Selection Operation
n Sorting
n Join Operation
n Other Operations
n Evaluation of Expressions
© Prof. Navneet Goyal, BITS, Pilani

Steps in Query
Processing
1. Parsing and translation

2. Optimization
3. Evaluation

Steps in Query
Processing

Steps in Query
Processing
n Parsing and translation
n Translate the query into its internal form.
n Translation is similar to the work performed
by the parser of a compiler
n Parser checks syntax, verifies relations
n Parse tree representation
n This is then translated into RA expression

Steps in Query
Processing
n Query Execution Plan
n In SQL, a query can be expressed is several ways
n Each SQL query can itself be translated into RA expression
in many ways
n An RA expression only partially tells you how to evaluate a
query
n Several ways to evaluate RA expression
n Annotate RA expression with instructions specifying how to
evaluate each operation
n Annotation may state the algorithm to be used for a specific
operation or the particular index to use
n Annotated RAE is called an evaluation primitive
n Sequence of primitive operations is a QEP
Steps in Query
Processing
n Evaluation
n The query-execution engine takes a query-
evaluation plan, executes that plan, and returns
the answers to the query.

Steps in Query
Processing
n Example
select balance
from account
where balance < 2500
n RAEs
n σbalance<2500(∏balance(account))
n ∏balance(σbalance<2500(account))
n E.g., we can use an index on balance to find
accounts with balance < 2500,
n or can perform complete relation scan and
discard accounts with balance ≥ 2500

Steps in Query
Processing
Query Execution Plan

Query Optimization
n Different QEPs for a given query can
have different costs
n Users not expected to write their
queries in a way that suggests the most
efficient QEP
n It is the system’s responsibility to
construct a QEP that minimizes the cost
n This is Query Optimization

Query Optimization
n Query Optimization: Amongst all equivalent
evaluation plans choose the one with lowest cost.
n Cost is estimated using statistical information from the
database catalog
• e.g. number of tuples in each relation, size of tuples, etc.
n First we study
n How to measure query costs
n Algorithms for evaluating relational algebra operations
n How to combine algorithms for individual operations in
order to evaluate a complete expression
n Next
n We study how to optimize queries, that is, how to find an
evaluation plan with lowest estimated cost

Query Optimization
n For optimizing a query, the Query
Optimizer must know the cost of each
operation
n Cost is hard to compute
n Depends on many parameters such as
actual memory available to the
operation
n Systems work with rough estimates

Query Optimization
n Ideally: Want to find best plan
Practically: Avoid worst plans!
n We will study the System-R
approach

Evaluating Expressions
n Materialization
n Pipelining

Materialization
n Materialized evaluation: evaluate one operation at a
time, starting at the lowest-level. Use intermediate results
materialized into temporary relations to evaluate next-level
operations.
n E.g., in figure below, compute and store
σ balance<2500 (account )
then compute the store its join with customer, and finally
compute the projections on customer-name.

Materialization (Cont.)
n Materialized evaluation is always applicable
n Cost of writing results to disk and reading them
back can be quite high
n Our cost formulas for operations ignore cost of writing
results to disk, so
• Overall cost=Sum of costs of individual operations +
cost of writing intermediate results to disk
n Double buffering: use two output buffers for
each operation, when one is full write it to disk
while the other is getting filled
n Allows overlap of disk writes with computation and
reduces execution time

Pipelining
n Several operations are grouped together
into a pipeline, in which each of the
operations start working on its tuples even
as they are being generated by another
operation
n Results of one operation are passed along to
the next operation in the pipeline
n Pipelined evaluation
n Reduces the number of temporary relations
that are produced
n Eliminates the cost of reading and writing
temporary relations
Pipelining
n Pipelined evaluation : evaluate several operations
simultaneously, passing the results of one operation on to the
next.
n E.g., in previous expression tree, don’t store result of
σ balance< 2500 (account )
n instead, pass tuples directly to the join. Similarly, don’t store result
of join, pass tuples directly to projection.
n Much cheaper than materialization: no need to store a temporary
relation to disk.
n Pipelining may not always be possible
n For pipelining to be effective, use evaluation algorithms that
generate output tuples even as tuples are received for inputs to
the operation.
n Pipelines can be executed in two ways: demand driven and
producer driven
Pipelining (Cont.)
n In demand driven or lazy evaluation
n system repeatedly requests next tuple from top level
operation
n Each operation requests next tuple from children operations
as required, in order to output its next tuple
n In between calls, operation has to maintain “state” so it
knows what to return next
n In producer-driven or eager pipelining
n Operators produce tuples eagerly and pass them up to their
parents
• Buffer maintained between operators, child puts tuples in
buffer, parent removes tuples from buffer
• if buffer is full, child waits till there is space in the buffer, and
then generates more tuples
n System schedules operations that have space in output
buffer and can process more input tuples
n Alternative name: pull and push models of pipelining

Measures of Query Cost
n Cost is generally measured as total elapsed
time for answering query
n Many factors contribute to time cost
• disk accesses, CPU, or even network communication
n Typically disk access is the predominant cost,
and is also relatively easy to estimate.
Measured by taking into account
n Number of seeks * average-seek-cost
n Number of blocks read * average-block-read-cost
n Number of blocks written * average-block-write-cost
• Cost to write a block is greater than cost to read a block
• data is read back after being written to ensure that the
write was successful (checksum)
n For simplicity we just use the number of block
transfers from disk and the number of seeks as
the cost measures
n tT – time to transfer one block
n tS – disk seek time + rotational latency
n Cost of operations that transfers b blocks and performs S
seeks
b * tT + S * tS
n We ignore CPU costs for simplicity
n Real systems do take CPU cost into account
n We do not include cost to writing output to disk in
our cost formulae (taken into account separately
when required)
n Cost depends on size of buffer in MM
n Best case – all the data reqd. by a query can be read
into the buffers and the disk need not be accessed
again
n Worst case – buffer can hold only a few blocks of data:
approx. one block per relation
n When presenting cost estimates, we generally assume
worst case
n Several algorithms can reduce disk IO by using extra
buffer space
n Amount of real memory available to buffer depends on other
concurrent queries and OS processes, known only during
execution
• We often use worst case estimates, assuming only the minimum
amount of memory needed for the operation is available

Selection Operation
n In query processing, file scan is the lowest-level operator
to access data
n File scans are search algorithms that locate & retrieve
records that fulfill a selection condition
n In relational systems, a file scan allows an entire relation to
be read in those cases where the relation is stored in a
single dedicated file (table scan & index scan)
n Basic Algorithms:
n Linear Search Cost estimate = br block transfers + 1 seek =
br *tT + ts
• br denotes number of contiguous blocks containing records from relation r
n Binary Search
n Indexes

Sorting
n Sorting of data plays an important role in
database systems:
n User wants data to be sorted
n Several relational operators like joins, can be
implemented efficiently if the input relations
are first sorted
n For relations that fit in memory,
techniques like quicksort can be used. For
relations that don’t fit in memory,
external sort-merge is a good choice.

Join Operation
n Several different algorithms to implement joins
n Nested-loop join
n Block nested-loop join
n Indexed nested-loop join
n Merge-join
n Hash-join
n Choice based on cost estimate

n Examples use the following information
n Number of records of customer: 10,000 depositor: 5000
n Number of blocks of customer: 400 depositor: 100

Nested-Loop Join
n To compute the theta join r θ s
for each tuple tr in r do begin

for each tuple ts in s do begin
test pair (tr,ts) to see if they satisfy the join condition θ
if they do, add tr • ts to the result.
end
end
n r is called the outer relation and s the inner relation of the
join.
n Requires no indices and can be used with any kind of join
condition.
n Expensive since it examines every pair of tuples in the two
relations.
Nested-Loop Join (Cont.)
n In the worst case, if there is enough memory only to hold one
block of each relation, the estimated cost is
nr ∗ bs + br
block transfers, plus
nr + br seeks
n Best Case: Both r & s fit into memory
Reduces cost to br + bs block transfers and 2 seeks
n If the smaller relation fits entirely in memory, use that as the
inner relation.
Reduces cost to br + bs block transfers and 2 seeks
Nested-Loop Join (Cont.)
n Assuming worst case memory availability cost estimate is
n with depositor as outer relation:
• 5000 ∗ 400 + 100 = 2,000,100 block transfers,

• 5000 + 100 = 5100 seeks
n with customer as the outer relation
• 10000 ∗ 100 + 400 = 1,000,400 block transfers

• 10,400 seeks
n If smaller relation (depositor) fits entirely in memory, the cost
estimate will be 500 block transfers.
n Block nested-loops algorithm is preferable.
Block Nested-Loop Join
n If the buffer is too small to hold either relation entirely in
memory
n How to OPTIMIZE?
n Pair every block of r with every block of s
n Variant of nested-loop join in which every block of inner
relation is paired with every block of outer relation.
for each block Br of r do begin
for each block Bs of s do begin
for each tuple tr in Br do begin
for each tuple ts in Bs do begin
Check if (tr,ts) satisfy the join condition
if they do, add tr • ts to the result.
end
end
end
end
Block Nested-Loop Join
n Worst case estimate: br ∗ bs + br block transfers + 2 * br
seeks
n Each block in the inner relation s is read once for each
block in the outer relation (instead of once for each tuple
in the outer relation)
n Assuming worst case memory availability cost estimate is
• 100 ∗ 400 + 100 = 40,100 block transfers,
• 2*100 = 200 seeks
n Assuming best case memory availability cost estimate is
• 100 + 400 = 500 block transfers
• 2 seeks
Hash-Join
n Applicable for equi-joins and natural joins.
n A hash function h is used to partition tuples of both relations
n h maps JoinAttrs values to {0, 1, ..., n}, where JoinAttrs denotes the
common attributes of r and s used in the natural join.
n r0, r1, . . ., rn denote partitions of r tuples
• Each tuple tr ∈ r is put in partition ri where i = h(tr [JoinAttrs]).

n r0,, r1. . ., rn denotes partitions of s tuples
• Each tuple ts ∈s is put in partition si, where i = h(ts [JoinAttrs]).

Hash-Join (Cont.)
Hash-Join (Cont.)
n r tuples in ri need only to be compared
with s tuples in si Need not be compared
with s tuples in any other partition, since:
n an r tuple and an s tuple that satisfy the
join condition will have the same value
for the join attributes.
n If that value is hashed to some value i,
the r tuple has to be in ri and the s tuple
in si.
Hash-Join Algorithm
The hash-join of r and s is computed as follows:
1. Partition the relation s using hashing function h. When

partitioning a relation, one block of memory is reserved as
the output buffer for each partition.
2. Partition r similarly.
3. For each i:
(a) Load si into memory and build an in-memory hash
index on it using the join attribute. This hash index
uses a different hash function than the earlier one h.
(b) Read the tuples in ri from the disk one by one. For
each tuple tr locate each matching tuple ts in si using
the in-memory hash index. Output the concatenation of
their attributes.
Relation s is called the build input and
r is called the probe input.
Hash-Join algorithm
(Cont.)
n The value n and the hash function h is chosen such that
each si should fit in memory.
n Typically n is chosen as ⎡bs/M⎤ * f where f is a “fudge
factor”, typically around 1.2
n The probe relation partitions si need not fit in memory
n Recursive partitioning required if number of partitions n

is greater than number of pages M of memory.
n instead of partitioning n ways, use M – 1 partitions for s
n Further partition the M – 1 partitions using a different

hash function
n Use same partitioning method on r
n Rarely required: e.g., recursive partitioning not needed

for relations of 1GB or less with memory size of 2MB,
with block size of 4KB.
Handling of Overflows
n Partitioning is said to be skewed if some partitions have
significantly more tuples than some others
n Hash-table overflow occurs in partition si if si does not
fit in memory. Reasons could be
n Many tuples in s with same value for join attributes
n Bad hash function
n Overflow resolution can be done in build phase

n Partition si is further partitioned using different hash
function.
n Partition r i must be similarly partitioned.
Handling of Overflows
n Overflow avoidance performs partitioning carefully to
avoid overflows during build phase
n E.g. partition build relation into many partitions, then
combine them
n Both approaches fail with large numbers of duplicates
n Fallback option: use block nested loops join on
overflowed partitions
Cost of Hash-Join
n If recursive partitioning is not required: cost of hash join is
3(br + bs) +4 ∗ nh block transfers +
2( ⎡br / bb⎤ + ⎡bs / bb⎤) seeks
n If recursive partitioning required:
n number of passes required for partitioning build relation
s is ⎡logM–1(bs) – 1⎤
n best to choose the smaller relation as the build relation.
n Total cost estimate is:
2(br + bs ⎡logM–1(bs) – 1⎤ + br + bs block transfers +
2(⎡br / bb⎤ + ⎡bs / bb⎤) ⎡logM–1(bs) – 1⎤ seeks
n If the entire build input can be kept in main memory no
partitioning is required
n Cost estimate goes down to br + bs.
Example:Cost of Hash-Join
customer depositor
n Assume that memory size is 20 blocks
n bdepositor= 100 and bcustomer = 400.
n depositor is to be used as build input. Partition it into five
partitions, each of size 20 blocks. This partitioning can be done
in one pass.
n Similarly, partition customer into five partitions,each of size 80.
This is also done in one pass.
n Therefore total cost, ignoring cost of writing partially filled
blocks:
n 3(100 + 400) = 1500 block transfers +
2( ⎡100/3⎤ + ⎡400/3⎤) = 336 seeks
Query Optimization
n Introduction
n Transformation of Relational Expressions
n Catalog Information for Cost Estimation
n Statistical Information for Cost
Estimation
n Cost-based optimization
n Dynamic Programming for Choosing
Evaluation Plans
n Materialized views

Introduction
n Alternative ways of evaluating a given query
n Equivalent expressions
n Different algorithms for each operation
n Cost difference between a good and a bad way
of evaluating a query can be enormous
n Need to estimate the cost of operations
n Statistical information about relations. Examples:
• number of tuples,
• number of distinct values for an attributes,
• Etc.
n Statistical estimation for intermediate results
• to compute cost of complex expressions

Transformation of
Relational Expressions
n Two relational algebra expressions are said to be
equivalent if on every legal database instance the two
expressions generate the same set of tuples
n Note: order of tuples is irrelevant
n In SQL, inputs and outputs are multisets of tuples
n Two expressions in the multiset version of the relational
algebra are said to be equivalent if on every legal database
instance the two expressions generate the same multiset of
tuples
n An equivalence rule says that expressions of two forms
are equivalent
n Can replace expression of first form by second, or vice versa
Equivalent Expressions
n Relations generated by two equivalent
expressions have the same set of attributes and
contain the same set of tuples
n although their tuples/attributes may be ordered
differently.

Multiple Transformations
(Cont.)

Equivalence Rules
1. Conjunctive selection operations can be deconstructed
into a sequence of individual selections.
2. Selection operations are commutative.
3. Only the last in a sequence of projection operations is

needed, the others can be omitted.
Π L1 (Π L2 (! (Π Ln ( E )) !)) = Π L1 ( E )
4. Selections can be combined with Cartesian products
and theta joins.
a. σθ(E1 X E2) = E1 θ E2
b. σθ1(E1 θ2 E2 ) = E1 θ1∧ θ2 E2
Equivalence Rules (Cont.)
5. Theta-join operations (and natural joins) are
commutative.
E1 θ E2 = E2 θ E1
6. (a) Natural join operations are associative:
(E1 E2) E3 = E1 (E2 E3)
(b) Theta joins are associative in the following manner:
(E1 θ1 E2) θ2∧ θ3 E3 = E1 θ2∧ θ3 (E2 θ2 E3)
where θ2 involves attributes from only E2 and E3.

Pictorial Depiction of
Equivalence Rules
7. The selection operation distributes over the theta join
operation under the following two conditions:
(a) When all the attributes in θ0 involve only the
attributes of one
of the expressions (E1) being joined.
σθ0(E1 θ E2) = (σθ0(E1)) θ E2
(b) When θ 1 involves only the attributes of E1 and θ2

involves
only the attributes of E2.
σθ1∧θ2 (E1 θ E2) = (σθ1(E1)) θ (σθ2 (E2))
9. The set operations union and intersection are commutative
E1 ∪ E2 = E2 ∪ E1
E1 ∩ E2 = E2 ∩ E1
■ (set difference is not commutative).
10. Set union and intersection are associative.
(E1 ∪ E2) ∪ E3 = E1 ∪ (E2 ∪ E3)
(E1 ∩ E2) ∩ E3 = E1 ∩ (E2 ∩ E3)
11. The selection operation distributes over ∪, ∩ and –.
σθ (E1 – E2) = σθ (E1) – σθ(E2)
and similarly for ∪ and ∩ in place of –
Also: σθ (E1 – E2) = σθ(E1) – E2
and similarly for ∩ in place of –, but not for ∪
12. The projection operation distributes over union
ΠL(E1 ∪ E2) = (ΠL(E1)) ∪ (ΠL(E2))
Evaluation Plan
n An evaluation plan defines exactly what algorithm is used for
each operation, and how the execution of the operations is
coordinated.

Introduction (Cont.)
n Generation of query-evaluation plans
for an expression involves several
steps:
1. Generating logically equivalent
expressions using equivalence rules.
2. Annotating resultant expressions to get
alternative query plans
3. Choosing the cheapest plan based on
estimated cost
n The overall process is called cost
based optimization.
Join Ordering Example
n For all relations r1, r2, and r3,
(r1 r2) r3 = r1 (r2 r3 )
n If r2 r3 is quite large and r1 r2 is
small, we choose
(r1 r2) r3
so that we compute and store a
smaller temporary relation.
SIMPLE!!
Cost-Based Optimization
n Consider finding the best join-order for r1 r2 . . . rn.
n How many different join orders?
There are (2(n – 1))!/(n – 1)! different join

orders for above expression. With n = 7,
the number is 665280, with n = 10, the
number is greater than 176 billion!
No need to generate all the join orders. Using
dynamic programming, the least-cost join
order for any subset of
{r1, r2, . . . rn} is computed only once and
stored for future use.
Join Order Example
n Best join-order of the form
(r1 r2 r3) r4 r5
12 different join orders for first part

12 orders for computing the join of this
result with r4 & r5
144 join orders to examine???
Dynamic Programming allows us to

examine only 12 + 12 choices to find the
best join order

Heuristic Optimization
n Cost-based optimization is expensive, even with dynamic
programming.
n Systems may use heuristics to reduce the number of
choices that must be made in a cost-based fashion.
n Heuristic optimization transforms the query-tree by using
a set of rules that typically (but not in all cases) improve
execution performance:
n Perform selection early (reduces the number of tuples)
n Perform projection early (reduces the number of attributes)
n Perform most restrictive selection and join operations before
other similar operations.
n Some systems use only heuristics, others combine heuristics
with partial cost-based optimization.

n Heuristic based optimizer would use the rules without
finding out whether the cost is reduced by the
transformation
n Example: σθ(r s), the condition refers only attributes in
s.
n The selection should be performed before the join!
n If r is extremely small compared to s, and if there is an
index on the join attributes of s, but no index on the
selection attributes, then probably it is a bad idea to
perform selection early
n Better to perform the join using the index on s, then
reject the tuples that fail the selection.

n Usually better to perform selections earlier than
projections
n Will potentially reduce the size of relations greatly
n Enable the use of indices to access tuples

Structure of Query
Optimizers
n The System R/Starburst optimizer considers only left-deep
join orders.
n Reduces optimization complexity and generates plans
amenable to pipelined evaluation.
n System R/Starburst also uses heuristics to push selections
and projections down the query tree.
n Heuristic optimization used in some versions of Oracle:
n For an n-way join, consider n evaluation plans
n Each plan uses a left deep join order, starting with a different
one of the n relations
n Repeatedly pick “best” relation to join next
• Starting from each of n starting points. Pick best among these.
Structure of Query
Optimizers
n Many applications execute the same query repeatedly
n Recent txs. on an account repeatedly, but with different
values for the account number.
n Most query optimizers optimize a query once, with
whatever values were provided, for the constants when
the query was first submitted and cache the query plan
n When the query is executed again (with new values), the
cached query plan is reused.
n Optimal plan for the new constants may differ from the
optimal plan for the initial values, but as a heuristic, the
cached plan is reused.
Left Deep Join Trees
n System-R does not consider all join
orders, but restrict the search to
particular kinds of join orders
n System R considers only those join orders
in which the right-hand-side input for
each join is a relation, not the result of an
intermediate join.
n left-deep join trees

Left Deep Join Trees
n How is this advantageous?

n Convenient for pipelined evaluation, since the
right operand is a stored relation, and thus only
one input to each join is pipelined
Materialized Views
n Query Modification
n View Materialization
n Which Views to Materialize?
n How to exploit Materialized
Views to answer queries?
n View Maintenance

View Modification
(Evaluate On Demand)
View CREATE VIEW RegionalSales(category,sales,state)
AS SELECT P.category, S.sales, L.state
FROM Products P, Sales S, Locations L
WHERE P.pid=S.pid AND S.locid=L.locid
Query
SELECT R.category, R.state, SUM(R.sales)
FROM RegionalSales AS R GROUP BY R.category, R.state
Modified SELECT R.category, R.state, SUM(R.sales)
Query FROM (SELECT P.category, S.sales, L.state
FROM Products P, Sales S, Locations L
WHERE P.pid=S.pid AND S.locid=L.locid) AS R
GROUP BY R.category, R.state
Materialized Views
n A view whose tuples are stored in the
database is said to be materialized
n Provides fast access, like a (very high-
level) cache.
n Need to maintain the view as the
underlying tables change.
n Ideally, we want incremental view
maintenance algorithms.
n Close relationship to Data
Warehousing, OLAP,
Issues in View
Materialization
n What views should we materialize, and
what indexes should we build on the
precomputed results?
n Given a query and a set of materialized
views, can we use the materialized views
to answer the query?
n How frequently should we refresh
materialized views to make them
consistent with the underlying tables?
(And how can we do this incrementally?)

View Materialization:
Example
Both queries
SELECT P.Category, SUM(S.sales) require us to
FROM Product P, Sales S join the Sales
WHERE P.pid=S.pid table with
GROUP BY P.Category another table &
aggregate the
result
SELECT L.State, SUM(S.sales)
FROM Location L, Sales S How can we use
WHERE L.locid=S.locid materialization
GROUP BY L.State to speed up
these queries?
Example
n Pre-compute the two joins involved
( product & sales & Location & sales)
n Pre-compute each query in its entirety
n OR let us define the following view:
CREATE VIEW TOTALSALES (pid, lid, total)

AS Select S.pid, S.locid, SUM(S.sales)
FROM Sales S
GROUP BY S.pid, S.locid

Example
n The View TOTALSALES can be
materialized & used instead os Sales in
our two example queries
SELECT P.Category, SUM(T.Total)
FROM Product P, TOTALSALES T
WHERE P.pid=T.pid
GROUP BY P.Category
SELECT L.State, SUM(T.Total)

FROM Location L, TOTALSALES T
WHERE L.locid=T.locid
GROUP BY L.State
View Maintenance
n A materialized view is said to be refreshed
when it is made consistent with changes ot
its underlying tables
n Often referred to as VIEW MAINTENANCE
n Two issues:
n HOW do we refresh a view when an underlying
table is refreshed? Can we do it incrementally?
n WHEN should we refresh a view in response to a
change in the underlying table?

View Maintenance
n The task of keeping a materialized view up-to-date with
the underlying data is known as materialized view
maintenance
n Materialized views can be maintained by recomputation on
every update
n A better option is to use incremental view maintenance
n Changes to database relations are used to compute
changes to materialized view, which is then updated
n View maintenance can be done by
n Manually defining triggers on insert, delete, and update of
each relation in the view definition
n Manually written code to update the view whenever database
relations are updated
n Supported directly by the database
View Maintenance
n Two steps:
n Propagate: Compute changes to view when data
changes.
n Refresh: Apply changes to the materialized view
table.
n Maintenance policy: Controls when we do
refresh.
n Immediate: As part of the transaction that
modifies the underlying data tables. (+
Materialized view is always consistent; - updates
are slowed)
n Deferred: Some time later, in a separate
transaction. (- View becomes inconsistent; + can
scale to maintain many views without slowing
updates) © Prof. Navneet Goyal, BITS, Pilani
Deferred Maintenance
Three flavors:
n Lazy: Delay refresh until next query on view;
then refresh before answering the query.
n Periodic (Snapshot): Refresh periodically.
Queries possibly answered using outdated
version of view tuples. Widely used, especially
for asynchronous replication in distributed
databases, and for warehouse applications.
n Event-based: E.g., Refresh after a fixed number
of updates to underlying data tables.

View Maintenance:
Incremental Algorithms
n Recomputing the view when an
underlying table is modified –
straightforward approach
n Not feasible to do so for all changes
made
n Ideally algorithms for refreshing a
view should be incremental
n Cost of refresh is proportional to
the extent of the change
View Maintenance:
Incremental Algorithms
n Note that a given row in the
materialized view can appear many
times (duplicates are not eliminated)
n Main idea behind incremental
algorithms is to efficiently compute
changes to the rows of the view
• New rows
• Changes to count associated with a row
• A row is deleted if its count becomes 0

View Maintenance
n The changes (inserts and deletes) to a relation
or expressions are referred to as its differential
n Set of tuples inserted to and deleted from r are denoted
ir and dr
n To simplify our description, we only consider
inserts and deletes
n We replace updates to a tuple by deletion of the tuple
followed by insertion of the update tuple
n We describe how to compute the change to the
result of each relational operation, given
changes to its inputs
n We then outline how to handle relational algebra
expressions © Prof. Navneet Goyal, BITS, Pilani
Join Operation
n Consider the materialized view v = r s and an
update to r
n Let rold and rnew denote the old and new states
of relation r
n Consider the case of an insert to r:
n We can write rnew s as (rold ∪ ir) s
n And rewrite the above to (rold s) ∪ (ir s)
n But (rold s) is simply the old value of the materialized
view, so the incremental change to the view is just ir s
n Thus, for inserts vnew = vold ∪(ir s)
n Similarly for deletes vnew = vold – (dr s)

Selection & Projection
Operations
n Selection: Consider a view v = σθ(r).
n vnew = vold ∪σθ(ir)
n vnew = vold - σθ(dr)
n Projection is a more difficult operation
n R = (A,B), and r(R) = { (a,2), (a,3)}
n ∏A(r) has a single tuple (a).
n If we delete the tuple (a,2) from r, we should not delete the
tuple (a) from ∏A(r), but if we then delete (a,3) as well, we
should delete the tuple
n For each tuple in a projection ∏A(r) , we will keep a count
of how many times it was derived
n On insert of a tuple to r, if the resultant tuple is already in
∏A(r) we increment its count, else we add a new tuple with
count = 1
n On delete of a tuple from r, we decrement the count of the
corresponding tuple in ∏A(r)
• if the count becomes 0, we delete the tuple from ∏A(r)
Aggregate Operations
n count : v = Agcount(B)(r)
( count of the attribute B, after grouping r by attribute A)
n When a set of tuples ir is inserted

• For each tuple t in ir, if the group t.A is present
in v, we increment its count, else we add a new
tuple (t.A, 1) with count = 1
n When a set of tuples dr is deleted
• for each tuple t in ir.we look for the group t.A in
v, and subtract 1 from the count for the group.
• If the count becomes 0, we delete from v the tuple
for the group t.A

Example
n Relation account grouped by branch-name:

branch_name account_number balance
Perryridge A-102 400
Perryridge A-201 900
Brighton A-217 750
Brighton A-215 750
Redwood A-222 700
branch_name g sum(balance) (account)

branch_name sum(balance)
Perryridge 1300
Brighton 1500
Redwood 700
n sum: v = A gsum (B)(r)
n We maintain the sum in a manner similar to count,
except we add/subtract the B value instead of
adding/subtracting 1 for the count
n Additionally we maintain the count in order to
detect groups with no tuples. Such groups are
deleted from v
• Cannot simply test for sum = 0 (why?)
n To handle the case of avg, we maintain the
sum and count aggregate values separately,
and divide at the end
n min, max: v = Agmin (B) (r).
n Handling insertions on r is
straightforward.
n Maintaining the aggregate values min
and max on deletions may be more
expensive. We have to look at the
other tuples of r that are in the same
group to find the new minimum

Example
TEST IDNO Marks

T1 A-102 15
T1 A-103 20
T2 A-102 25
T2 A-103 10
T3 A-104 10
Test g Min(marks) (student_record))

TEST Min(marks)
T1 15
T2 10

Dynamic Programming in
Optimization
n To find best join tree for a set of n relations:
n To find best plan for a set S of n relations, consider
all possible plans of the form: S1 (S – S1) where
S1 is any non-empty subset of S.
n Recursively compute costs for joining subsets of S
to find the cost of each plan. Choose the cheapest
of the 2n – 1 alternatives.
n When plan for any subset is computed, store it and
reuse it when it is required again, instead of
recomputing it
• Dynamic programming

Q&A
Thank You

Lec 31-36 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lec 31-36 PDF

Uploaded by

Copyright:

Available Formats

Query Processing &

Professor Navneet Goyal

© Prof. Navneet Goyal, BITS, Pilani

1. Parsing and translation

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

Query Execution Plan

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

n Block nested-loop join

n Indexed nested-loop join

n Choice based on cost estimate

© Prof. Navneet Goyal, BITS, Pilani

for each tuple tr in r do begin

• 5000 ∗ 400 + 100 = 2,000,100 block transfers,

• 10000 ∗ 100 + 400 = 1,000,400 block transfers

• Each tuple tr ∈ r is put in partition ri where i = h(tr [JoinAttrs]).

• Each tuple ts ∈s is put in partition si, where i = h(ts [JoinAttrs]).

1. Partition the relation s using hashing function h. When

n Recursive partitioning required if number of partitions n

n Further partition the M – 1 partitions using a different

n Rarely required: e.g., recursive partitioning not needed

n Bad hash function

n Overflow resolution can be done in build phase

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

2. Selection operations are commutative.

3. Only the last in a sequence of projection operations is

(b) Theta joins are associative in the following manner:

(E1 θ1 E2) θ2∧ θ3 E3 = E1 θ2∧ θ3 (E2 θ2 E3)

where θ2 involves attributes from only E2 and E3.

σθ0(E1 θ E2) = (σθ0(E1)) θ E2

(b) When θ 1 involves only the attributes of E1 and θ2

© Prof. Navneet Goyal, BITS, Pilani

There are (2(n – 1))!/(n – 1)! different join

12 different join orders for first part

Dynamic Programming allows us to

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

n How is this advantageous?

© Prof. Navneet Goyal, BITS, Pilani

© Prof. Navneet Goyal, BITS, Pilani

CREATE VIEW TOTALSALES (pid, lid, total)

© Prof. Navneet Goyal, BITS, Pilani

SELECT L.State, SUM(T.Total)

© Prof. Navneet Goyal, BITS, Pilani