Professional Documents
Culture Documents
Data Analysis PDF
Data Analysis PDF
Objectives
l Overall: Reinf orce your understanding f rom the main lecture l Specific: * Concepts of data analysis * Some data analysis techniques * Some tips for data analysis l What I will not do: * To teach every bit and pieces of statistical analysis techniques
Statistical Methods
l Something to do with statistics l Statistics: meaningf ul quantities about a sample of objects, things, persons, events, phenomena, etc. l Widely used in social sciences. l Simple to complex issues. E.g. * correlation * anova * manova * regression * econometric modelling l Two main categories: * Descriptive statistics * Inferential statistics
Descriptive statistics l Use sample information to explain/make abstraction of population phenomena . l Common phenomena : l * Association (e.g. 1,2.3 = 0.75) l * Tendency (left-skew, right-skew) l * Causal relationship (e.g. if X, then, Y) l * Trend, pattern, di spersion, range l Used in non-parametric analysis (e.g. chisquare, t-test, 2-way anova)
300,000
No. of houses
1 32635.8
2 38100.6
3 42468.1
4 47684.7
5 48408.2
6 61433.6
7 77255.7
8 97810.1
ru
ng
ho
Ye ar (1990 - 1997) Tre nds in pr operty loan, shop house de m and & s upply
Jo
Ko
Ba
District
200
14
180
10 8 6 4 2 0
12
Proportion (%)
160
140
120
100
0-
-3
-1
-2
-4
-5
-6
-7
30
60
10
20
40
50
70
80 0 20 40 60 80 100 120
M u Po ar n Se tian ga m at
gi ng M
ha
ah
ua
Pa
rB
tu
Kl
ta
er
Ti
si
ng
50.00
Distance from Rakaia (km)
160
40.00
% prediction error
100.00 80.00 60.00 40.00 20.00 0.00 -20.00 -40.00 -60.00 -80.00 -100.00
140
30.00
120
20.00
100
10.00
80 20 40 60 80 100 120
Inferential statistics l Using sample statistics to infer some phenomena of population parameters l Common phenomena : cause-and-effect * One-way r/ship Y = f(X) * Multi-directional r/ship Y1 = f(Y2, X, e1) Y2 = f(Y1, Z, e2) * Recursive Y1 = f(X, e1)
Y2 = f(Y1, Z, e2)
Examples of relationship
Dep=9t 215.8 Dep=7t 192.6
Coefficientsa Unstandardized Coefficients B Std. Error 1993.108 239.632 -4.472 1.199 6.938 .619 4.393 1.807 -27.893 6.108 34.895 89.440 Standardized Coefficients Beta -.190 .705 .139 -.241 .020
Model 1
l Infeasible techniques. E.g. How to design ex-ante effects of KLIA? Development occurs before and after! What is the control treatment? Further explanation! l Abuse of statistics. E.g. l Simply exclude a technique
Using R2 and/or F-value of a model Multi-dimensional scaling, Likert scaling Multi-dimensional scaling, Likert scaling
Principles of analysis
l Goal of an analysis: * To explain cause-and-effect phenomena * To relate research wi th real-world event * To predict/forecast the real -world phenomena based on research * Finding answers to a particular problem * Making conclusions about real-world event based on the probl em * Learning a lesson from the problem
Number 6 4 10 15
Basic Concepts
l l l l Population: the whole set of a universe Sample: a sub-set of a population Parameter: an unknown fixed value of population characteristic Statistic: a known/calculable value of sample characteristic representing that of the population. E.g. = mean of population, = mean of sample Q: What is the mean price of houses in J.B.? A: RM 210,000
1
= 300,000
= 120,000
SD
3
= 210,000
DST
Basic Concepts (contd.) l Randomness : Many things occur by pure chances rainfall, disease, birth, death,.. l Variability: Stochasti c processes bri ng in them various different dimensions, characteristics, properties, features, etc., in the population l Statistical analysis methods have been developed to deal with these very nature of real world.
Central Tendency
Measure Mean
(Sum of all values no. of values)
Advantages * Best known average * Exactly calculable * Make use of all data * Useful for statistical analysis * Not influenced by extreme values * Obtainable even if data distribution unknown (e.g. group/aggregate data) * Unaffected by irregular class width * Unaffected by open-ended class * Unaffected by extreme values * Easy to obtain from histogram * Determinable from only values near the modal class
Disadvantages * Affected by extreme values * Can be absurd for discrete data (e.g. Family size = 4.5 person) * Cannot be obtained graphically * Needs interpolation for group/ aggregate data (cumulative frequency curve) * May not be characteristic of group when: (1) items are only few; (2) distribution irregular * Very limited statistical use
Median
(middle value)
Mode
(most frequent value)
14 24 18 20 12
= 96;
= 12
Thus,
= 96/12 = 8
l What is the mean rental across the areas? = 23; = 3317.5 Thus, = 3317.5/23 = 144.24
Upper quartile = (n+1) = 19.5 th. Taman UQ = 145 + (3/7 x 5) = RM 147.1/month Lower quarti le = (n+1)/4 = 26/4 = 6.5 th. Taman LQ = 135 + (3.5/5 x 5) = RM138.5/month Inter-quartile = UQ LQ = 147.1 138.5 = 8.6 th. Taman IQ = 138.5 + (4/5 x 5) = RM 142.5/month
Variability
l Indicates dispersion, spread, variation, deviation l For single populat ion or sample data:
where 2 and s2 = population and sample variance respectively, xi = individual observations, = population mean, = sample mean, and n = total number of individual observations.
standard deviation
standard deviation
Variability (contd.)
l Why measure of dispersion important? l Consider ret urns from two categories of shares: * Shares A (%) = {1.8, 1.9, 2.0, 2.1, 3.6} * Shares B (%) = {1.0, 1.5, 2.0, 3.0, 3.9} Mean A = mean B = 2.28% But, different variability! Var(A) = 0.557, Var(B) = 1.367 * Would you invest in category A shares or category B shares?
Variability (contd.)
l Coefficient of variation COV std. deviation as % of the mean:
l Could be a better measure compared to std. dev. COV(A) = 32.73%, COV(B) = 51.28%
Variability (contd.)
l Std. dev. of a frequency distribution
The following table shows the age distribution of second-time home buyers:
x^
Probability Distribution
l Defined as of probability density function (pdf). l Many types: Z, t, F, gamma, etc. l God-given nature of the real world event. l General form: (continuous)
(discrete)
l E.g.
Dice1 Dice2
1 2 3 4 5 6 7
2 3 4 5 6 7 8
3 4 5 6 7 8 9
4 5 6 7 8 9 10
5 6 7 8 9 10 11
6 7 8 9 10 11 12
1 2 3 4 5 6
Discrete values
Discrete values
Many real world phenomena take a form of continuous random variable Can take any values between two limits (e.g. income, age, weight, price, rental, etc.)
y c n e u q e r4 F
Mean = 4.0628 Std. Dev. = 1.70319 N = 32 0 2.00 3.00 4.00 5.00 6.00 7.00
Rental (RM/sq.ft.)
0.206
* Bell-shaped, symmetrical
= mean of variable x
* Has a function of
= std. dev. Of x = ratio of circumference of a circle to its diameter = 3.14 e = base of natural log = 2.71828
Probability distribution
1 = ? 2 = ? 3 = ?
= ____% from total observation = ____% from total observation = ____% from total observation
Probability distribution
* Has the following distribution of observation
Probability distribution
l There are various other types and/or shapes of distribution. E.g.
Note: Sp(AGE=age) 1 How to turn this graph into a probability distribution function (p.d.f.)?
Z-Distribution
f(X=x) is given by area under curve Has no standard algebraic method of integration Z ~ N(0,1) It is called normal distribution (ND) Standard reference/approximation of other distributions. Since there are various f(x) forming NDs, SND is needed l To transform f(x) into f(z): x- Z = --------- ~ N(0, 1) 160 155 E.g. Z = ------------- = 0.926 5.4 l l l l l Probability is such a way that: * Approx. 68% -1< z <1 * Approx. 95% -1.96 < z < 1.96 * Approx. 99% -2.58 < z < 2.58
Z-distribution (contd.)
l When X= , Z = 0, i.e.
l When X = + , Z = 1 l When X = + 2, Z = 2 l When X = + 3, Z = 3 and so on. l It can be proven that P(X 1 <X< Xk) = P(Z 1 <Z< Zk) l SND shows the probability to the right of any particular value of Z. l Example
Normal distributionQuestions
Your sample found that the mean price of affordable homes in Johor Bahru, Y, is RM 155,000 with a variance of RM 3.8x107. On the basis of a normality assumption, how sure are you that: (a) The mean price is really RM 160,000 (b) The mean price is between RM 145,000 and 160,000
Answer (a):
P(Y 160,000) = P(Z ---------------------------) = P(Z 0.811) 3.8x10 7 = 0.1867 Using Z-table, the required probability is: 1-0.1867 = 0.8133
160,000 -155,000
Always remember: to convert to SND, subtract the mean and divide by the std. dev.
Normal distributionQuestions
Answer (b): Z1 = ------ = ---------------- = -1.622
3.8x10 7 X1 - 145,000 155,000
X2 -
160,000 155,000
Normal distributionQuestions
You are told by a property consultant that the average rental for a shop house in Johor Bahru is RM 3.20 per sq. After searching, you discovered the following rental data: 2.20, 3.00, 2.00, 2.50, 3.50,3.20, 2.60, 2.00, 3.10, 2.70 What is the probability that the rental is greater than RM 3.00?
Student s t-Distribution
l Similar to Z-distribution: * t(0,) but n1 * - < t < + * Flatter with thicker tails * As n t(0,) N(0,1) * Has a function of
where G=gamma distribution; v=n -1=d.o.f; p=3.147
Student s t-Distribution
l Given n independent measurement s, xi, let
where is the population mean, is the sample mean, and s is the estimator for population standard deviation. l Distribution of the random variable t which is (very loosely) the "best" that we can do not knowing .
Student s t-Distribution
l Student's t-distribution can be derived by: * transforming Student's z-distribution using
Student s t-Distribution
l fr(t) =
Fr(t) = = = where r n-1 is the number of degrees of freedom, -<t<,G(t) is the gamma function, B(a,b) is the beta function, and I(z;a,b) is the regularized beta function defined by l
Correlation
l Co-exist .E.g. * left shoe & right shoe, sleep & lying down, food & drink l Indicate some co-existence relationship. E.g. * Linearly associated (-ve or +ve) Formula: * Co-dependent, independent l But, nothing to do with C-A-E r/ship!
Example: After a field survey, you have the fol lowing data on the distance to work and distance to the city of residents in J.B. area. Interpret the results?
Contingency
l A form of conditional co-existence: * If X, then, NOT Y; if Y, then, NOT X * If X, then, ALSO Y * E.g. + if they choose to live close to workplace, then, they will stay away from city + if they choose to live close to city, then, they will stay away from workplace + they will stay close to both workplace and city
Test yourselves!
Q1: Calculate the min and std. variance of the following data: PRICE - RM 000 SQ. M OF FLOOR 130 137 128 390 140 241 342 143 135 140 100 360 175 270 200 170
Q2: Calculate the mean price of the following low-cost houses, in various localities across the country:
36 3
37 14
38 10
39 36
40 73
41 27
42 20
43 17
Test yourselves!
Q3: From a sample information, a population of housing estate is believed have a normal distribution of X ~ (155, 45). What is the general adjustment to obtain a Standard Normal Distribution of this population? Q4: Consider the f ollowing ROI for two types of investment: A: 3.6, 4.6, 4.6, 5.2, 4.2, 6.5 B: 3.3, 3.4, 4.2, 5.5, 5.8, 6.8 Decide which investment you would choose.
Test yourselves!
Q5: Find: f(AGE > 30-34) f(AGE 20-24) f( 35-39 AGE < 50-54)
Test yourselves!
Q6: You are asked by a property marketing manager to ascertain whether or not distance to work and distance to the city are equally important factors influencing people s choice of house location. You are given the following data for the purpose of testing: Explore the data as follows: Create histograms for both distances. Comment on the shape of the histograms. What is you conclusion? Construct scatter diagram of both distances. Comment on the output. Explore the data and give some analysis. Set a hypothesis that means of both distances are the same. Make your conclusion.
Thank you