Professional Documents
Culture Documents
PLN08 BO3 Hubbard How To Measure Anything
PLN08 BO3 Hubbard How To Measure Anything
What is AIE?
Applied AppliedInformation InformationEconomics Economics(AIE) (AIE)is isthe thepractical practicalapplication applicationof ofscientific scientificand andmathematical mathematical methods to quantify the value of management choices regardless of how difficult methods to quantify the value of management choices - regardless of how difficultthe the measurement challenge appears to be. measurement challenge appears to be. Economics Operations Research Modern Portfolio Theory Options Theory Decision/Game Theory Statistics Information Theory
Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits. The Gartner Group AIE represents a rigorous, quantitative approach to improving IT investment decision making..this investment will return multiples by enabling much better decision making. Giga recommends that IT executives learn more about AIE and begin to adopt its tools and methodologies, especially for large IT projects. Giga Information Group
The perceived impossibility of measurement is an illusion caused by not understanding: the Concept of measurement the Object of measurement the Methods of measurement See my Everything is Measurable article in CIO Magazine (go to articles link on www.hubbardresearch.com
Copyright HDR 2008 dwhubbard@hubbardresearch.com 5
Studies also show that measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement Training can calibrate people so that of all the times they say they are 90% confident, they will be right 90% of the time
10
In the English Language, the word strategy is used more often than the word celebrate.
True/False ____% Confidence
11
12
90%
Percent Correct
21 45 65 68 152 75 71 65 58 21
30% 50%
60%
70%
80%
90% 100%
The formula for the value of information has been around for almost 60 years. It is widely used in many parts of industry and government as part of the decision analysis methods but still mostly unheard of in the parts of business where it might do the most good. What it means: 1.Information reduces uncertainty 2.Reduced uncertainty improves decisions 3.Improved decisions have observable consequences with measurable value
Copyright HDR 2008 dwhubbard@hubbardresearch.com 14
15
16
5%
10%
15%
20%
25%
EVPI ENBI
EVI
Maximum ENBI
EVPI Expected Value of Perfect Information ECI Expected Cost of Information EVI Expected Value of Information ENBI Expected Net Benefit of Information
ECI
$0 Low accuracy High accuracy
Copyright HDR 2008 dwhubbard@hubbardresearch.com 18
Also, we found that, if anything, fewer measurements were required after the information values were known.
See my article The IT Measurement Inversion in CIO Magazine (its also on my website at www.hubbardresearch.com under the articles link)
Copyright HDR 2008 dwhubbard@hubbardresearch.com 19
Economic Relevance
Typical Attention
20
Practical Assumptions
Its been measured before You have more data than you think You need less data than you think Its more economical than you think Your subjective estimate of possible measurement errors is exaggerated Its amazing what you can see when you look Yogi Berra
21
8 10 12 14 16 18 20
1%
0.5% 0.2%
0.1%
10
23
Several clever sampling methods exist that can measure more with less data than you might think Examples: estimating the population of fish in the ocean, estimating the number of tanks created by the Germans in WWII, extremely small samples, etc.
Copyright HDR 2008 dwhubbard@hubbardresearch.com 24
Reducing Inconsistency
The Lens Model is another method used to improve on expert intuition The chart shows the reduction in error from this method on intuitive estimates In every case, this method equaled or bettered the judgment of experts IT Portfolio Priorities Battlefield Fuel Forecasts Student ratings of teaching effectiveness Cancer patient life-expectancy Psychology course grades Graduate students grades Changes in stock prices IQ scores using Rorschach tests Mental illness using personality tests Business failures using financial ratios Life-insurance salesrep performance
Source: Hubbard Decision Research
My My Studies Studies
0%
10%
20%
30%
40%
Reduction in Errors
25
In studies where people were asked to do this, thier results were usually not irrational compared to what would be computed with Bayesian statistics calibrated people do even better
Copyright HDR 2008 dwhubbard@hubbardresearch.com 26
Comparison of Methods
Typical Un-calibrated Expert Overconfident (Stated uncertainty is lower than rational) Gullible Stubborn
NonBayesian Bayesian Statistics Calibrated Expert Under-confident (Stated uncertainty is higher than rational)
Vacillating, Indecisive
Overly Cautious
Traditional non-Bayesian statistics (what you probably learned in the first semester of stats) assumes you knew nothing prior to the samples you took - this is almost never true in reality Most un-calibrated experts are overconfident and slightly overemphasize new information Calibrated experts are not overconfident, but slightly ignore prior knowledge Bayesian analysis is the perfect balance; neither under- nor over- confident, uses both new and old information
27
5% 10%
10% 20%
15% 30%
ROI
-50% 0% 50% 100%
28
29
Optimize Decision
30
32
Final Tips
Learn how to think about uncertainty, risk and information value in a quantitative way Assume its been measured before You have more data than you think and you need less data than you think Methods that reduce your uncertainty are more economical than many managers assume Dont let exception anxiety cause you to avoid any observations at all Just do it
Copyright HDR 2008 dwhubbard@hubbardresearch.com 33
Questions?
Doug Hubbard Hubbard Decision Research dwhubbard@hubbardresearch.com www.hubbardresearch.com 630 858 2788
If you want electronic copies of this presentation and copies of supporting articles I mention, please leave me a business card with Presentation written on the back
34