Performance Analysis

Loriana Pelizzon
University of Venice

Overview
Asset allocation Performance Analysis Attribution analysis Mutual Funds Analysis Benchmarks Peer group analysis Style analysis

Asset allocation
how one decides to allocate assets among various asset classes such as stocks, bonds, and cash. How do you decide how to allocate your assets? The goal of asset allocation is to create a diversified portfolio with an acceptable level of risk and the highest possible return given that level of risk. A portfolio or asset allocation that maximizes return for the level of risk is called an efficient portfolio

Asset allocation
The most difficult aspect of this procedure is to accurately predict expected returns. In the case study we used long term historical returns, but is it reasonable to expect that history will repeat itself? There are a number of other ways to make such predictions. One of the most promising methods is a model developed by Fischer Black and Robert Litterman while they were at Goldman Sachs.

Performance analysis
Risk-adjusted measures based on absolute benchmark Risk adjusted measures based on relative benchmark Risk adjusted measures based on Customized Benchmark

Risk-adjusted measures based on absolute benchmark

Sharpe ratio
Sharpe Ratio = fund ' s average excess return standard deviation of fund excess return

Treynor ratio
Treynor Ratio = fund ' s average excess return

Risk-adjusted measures based on absolute benchmark

Sortino ratio:
Sortino Ratio = fund's excess return downside deviation

where:
downside deviation = VAR

min( 0, fund's excess return )

Risk adjusted measures based on relative benchmark

Morningstar Risk-Adjusted Rating.
Morningstar Return = Load Adjusted Return on the Fund T Bill Higher of (Average Category Excess Return or T Bill)

Morningsta rRisk =

Fund' s Average Underperformance Average Underperformance of its Category

Risk adjusted measures based on relative benchmark

To calculate a funds summary star-rating, the Morningstar Risk scores are then subtracted from the Morningstar Return scores.

Risk adjusted measures based on Customized Benchmark

The information ratio, which is the ratio of excess return to standard deviation of excess return (or tracking error), is a measure of a managers skill and the consistency with which the manager has been able to outperform

Informatio n Ratio =

Alpha Tracking Error

Manager performance: practice

One way to measure manager performance is by looking at how an initial investment would have grown over a specific time period

Manager performance: practice

Manager performance: practice

Manager performance: practice

The rolling window graph is also helpful for detecting structural changes in a portfolio. A risk control process that was initiated by a manager several years ago, for example, would probably lower the portfolio's tracking error. As a result, the symbols would shift to the left as they do for Active Manager A (blue).

Manager performance: practice

Manager performance: practice

Manager performance: practice

Manager performance: practice

Manager performance: practice

Attribution Analysis
To what do we attribute a managers performance? Is it stock picking, investing in the right style, or market timing? Were certain sectors over or underweighted?

Attribution Analysis
Most of a managers returns are attributed to asset class returns. A US equity managers returns depend mostly on how well the US stock market does. The second most important factor for an equity manager is investment style. Most growth stock managers perform well when growth stocks are in favor. Conversely they perform badly when growth stocks are out of favor.

Attribution Analysis
The first goal is to find out how much of the managers return comes from the general market and investment style We accomplish this using a technique called style analysis

Attribution Analysis
Example. Using only the monthly returns for The Needham Growth Fund and the monthly returns from the four Russell style indices and T-Bills, we find the combination of indices that best describes Needhams behavior/style.

Attribution Analysis

Attribution Analysis
In Figure 2 the red portion of the pie chart shows that these indices account for 77.5% of the variance in Needhams return. The variance of Needhams return that cant be explained by the market and style is represented by the green portion of the pie. This residual variance or behavior is likely due to the managers stock selection or sector bets.

Attribution Analysis

Attribution Analysis
The portion of the managers returns that are explained by exposures to the style indices could be passively replicated by buying the appropriate percentages of index funds or ETFs that represent the style indices. The managers alpha is generated by the portion of the fund that we cannot passively replicate. This represents the managers active bets. They could be stock bets, sector bets, or even market timing bets.

Attribution Analysis
In an attempt to identify these sources of returns, we start by constructing a custom benchmark called a style benchmark that is based on the index weights in Figure 1. The performance graph and table (Figure 3) show that Needham beat its custom style benchmark by an annualized 16.92%. This is the excess return Needham achieved over what we could passively construct to represent Needhams investment style. This is the result of either manager skill or luck (how we differentiate between the two is explained in Mutual Fund Analysis). For now we assume manager skill.

Attribution Analysis

Attribution Analysis: stock selection

Attribution Analysis
Should Needhams excess return be attributed to stock selection, sector weightings, and/or market timing? To see the impact of sector bets we perform another style analysis using sector indices rather than style indices. Because we are using returns and a rolling window we dont expect to precisely identify the sector weights at any specific time but rather get an idea of what the sector exposures have been over the life of the fund and how they have changed over time.

Attribution Analysis: sector weightings

Attribution Analysis
Figure 4 above contains the results of the sector analysis, which shows that Needham is heavily weighted in technology, health care, and T-Bills. T-Bills represent cash or anything that makes the portfolio behave like cash. Based on the Needham Fund's prospectus the fund is run somewhat like a hedge fund. It shorts stocks and uses derivatives to reduce risk. Once again, using the exposure to the indices used in the style analysis, we construct a custom style benchmark.

Attribution Analysis

Attribution Analysis
Figure 5 below shows that Needham outperforms this benchmark by 13.84% annualized. So of the 16.92% outperformance, about 3% is from their sector bets. The balance, 13.8%, is the result of either stock selection or market timing

Attribution Analysis

Attribution Analysis
Market timing doesnt necessarily mean moving from 100% stocks to 100% cash. It can be as subtle as buying low beta stocks when one perceives the market is over valued. It could also be a value manager building cash because he cant find good valuations.

Attribution Analysis

Attribution Analysis
One way to evaluate the results of market timing is to see how managers do in both up and down markets. If a manager goes up more than the benchmark when the benchmark goes up, the manager plots above the horizontal line in Figure 6 above. If a manager goes down more than the benchmark when the benchmark goes down, he plots to the right of the vertical line. If the manager goes down less he plots to the left of the vertical line.

Attribution Analysis
Aggressive managers who go up more and down more plot in the northeast quadrant. Defensive managers who go up less and down less plot in the southwest corner. Managers that go up more and down less, as is the case with Needham, plot in the northwest quadrant. We believe that this is the result of good market timing particularly if there is a consistent pattern of such behavior, as seen in Figure 7 below.

Attribution Analysis

Attribution Analysis
Managers with bad market timing that go up less and down more fall into the southeast quadrant. Needham went up 26% more than the benchmark when the benchmark had a positive return. When the benchmark went down the fund declined about 22% less.

Mutual Fund Analysis

When investing in non-index mutual funds, investors make two critical assumptions: 1) that skillful managers exist, 2) that they have the ability to recognize them. If an investor is not willing to make these two assumptions, they should invest in non-active funds like index funds or exchange traded funds (ETFs).

Mutual Fund Analysis

Mutual fund analysis, both qualitative and quantitative, attempts to identify skillful active managers. Qualitative analysis looks at factors such as the background and experience of the manager and the mutual fund company. Here, we look only at the quantitative factors such as manager performance, style, style consistency, risk, risk-adjusted performance, etc.

Mutual Fund Analysis

What is the best way to analyze, and ultimately select, mutual funds? Financial journalists are not equipped to analyze mutual funds. In most cases they are simply reporting the performance figures they received from the managers themselves or the marketing/public relations people. Mutual fund rating services are good data collectors but lack any real sophistication in fund analysis. These services are oriented toward the retail fund investor. Consequently sophisticated advisors, plan sponsors and consultants must perform their own mutual fund analysis.

Mutual Fund Analysis

The two biggest mistakes in quantitative mutual fund analysis are improper: benchmarking end point bias. How can you avoid these mistakes?

Benchmark
The most common error made when measuring a managers performance is the selection of an improper benchmark. Morningstars star ratings, for example, are based on funds performance relative to a broad group of fund returns, as opposed to a more specific benchmark that reflects the manager's true style.

Benchmark
Because of this, on February 28, 2000, at the very peak of the growth stock bubble, most of Morningstars five star funds were growth funds while there were no five star value funds. Two years later, after the value funds did well and the growth funds crashed, most of the five star funds were value funds. What makes a good benchmark?

Benchmark
At the heart of a quality manager analysis is a good benchmark. In order for a benchmark to be a valid and effective tool for measuring a managers performance, it must be: Unambiguous Investable Measurable Appropriate Reflective of current investment opinions Specified in advance

Benchmark
A benchmark with all of these characteristics is the style benchmark. The style benchmark is the result of Nobel Laureate William F. Sharpes returns-based style analysis. The style benchmark is a custom benchmark produced by weighting a set of indices in a unique combination that reflects the style of the manager. The most important advantage of a custom style benchmark over a standard market benchmark is that it accounts for the style characteristics of the manager.

Benchmark
If the manager specializes in small cap growth stocks then the benchmark should be made up of small cap growth stocks. In fact, the ideal benchmark explains all of the returns of the manager that come from systematic factors such as style and market movements. If this is the case, any performance over (or under) the benchmark can be attributed to manager skill. A benchmark that does not do a good job of capturing the style of the manager will always leave you wondering did the manager outperform because of style differences with the benchmark?

Benchmark
The investment industry uses a number of inappropriate benchmarks, the most common of which is a manager universe or peer group. Manager universes are not investable, not specified in advance, and since they are made up of active managers they are not the passive equivalent of an active manager. Additionally, manager universes suffer from survivor bias (the poor performing managers drop out and / or are merged with better performing funds). Most importantly, they are usually too broadly defined to accurately judge the skill of a specific manager.

Benchmark
Broad market indices such as the S&P 500, Russell 3000, Wilshire 5000 etc. are not good benchmarks for most active non-large core managers. Even style indices such as the Russell 1000 Growth, 1000 Value, 2000 Growth, or 2000 Value are not appropriate for the vast majority of managers

Style Benchmarks vs. Market Benchmarks

Style benchmarks are superior to single index benchmarks for the majority of managers. Figures 1 and 2 show the result of a style analysis of the Dodge & Cox Fund. The combination of Russell style indices and T-Bills that best defines the style of this fund is shown in Figure 1.

Style Benchmarks vs. Market Benchmarks

Style Benchmarks vs. Market Benchmarks

The Manager Style graph, shown in Figure 2, maps the funds style relative to the four Russell style indices. As explained earlier, the analytic technique that enables us to determine the funds effective asset mix, create the Manager Style Map, and build the custom style benchmark is called returns-based style analysis. The custom style benchmark is made up of the styles and weights shown on the bar chart. For Dodge & Cox, the style benchmark is a composite returns series made up of 2.1% in T-Bills, 76.5% in the Russell Large (1000) Value Index, and 21.4 % in the Russell Small (2000) Value Index.

Style Benchmarks vs. Market Benchmarks

Can we prove that this is a better benchmark?

A good test of a benchmark is to see how correlated the benchmarks returns are to the managers returns. The higher the correlation the better the benchmark. The red portion of the left pie chart on Figure 3 shows the R-squared (correlation squared) of the managers returns to the custom style benchmark, 90.0%. The pie chart on the right shows that the Rsquared to the S&P 500 is only 67.5%.

Benchmark
The green portion of the pie chart measures the variance in the funds returns that is not explained by the benchmark. Notice for the S&P 500 it is more than three times greater than for the style benchmark. A good benchmark includes all of the systematic factors (market, style) so that the unexplained variance is due exclusively to nonsystematic or idiosyncratic factors that are primarily the result of the managers stock selection.

End Point Bias

The other common mistake made in performance analysis is called end point bias. Most of the funds recommended by various financial publications are ones that recently performed well. When looking at cumulative statistics, recent performance above the benchmark creates the illusion that the fund has consistently outperformed. Cumulative statistics are calculated through the most recent time period. Annualized return for one, three, five, and seven years, for example, is often used to evaluate mutual funds.

End Point Bias

Notice that the most recent year is included in all of these periods. Due to the nature of these statistics, recent performance often hides past performance. Here is an example. The September 15, 2003 Forbes magazine heralded the Mairs & Power Growth fund as one of the three best funds to own based on its long term record. In Figure 1 the long term annualized returns for this fund look quite good.

End Point Bias

End Point Bias

End Point Bias

Because this fund outperformed its benchmark (the S&P 500 is a reasonable benchmark for this fund) for 2, 3, 5, 7, 10, 15, 20, and 25 year periods, one would think that the fund is a consistently good performer. Now look at Figure 2 below. The red and green shaded area at the bottom of the performance graph shows the cumulative return relative to the benchmark. If you had purchased this fund twenty five years ago you would have spent all but the last couple of years below your benchmark. It has only been the very good performance in the last few years that give it the high annualized rates of return found in Figure 1.

End Point Bias

This is what we mean by an end point bias. We could also call it the broken clock syndrome (a broken clock will be right twice a day). Similarly, if a manager has been managing money for twenty five years, even with no skill, there is likely to be several years of good performance. If your end point (the date on which your analysis ends) is particularly good, cumulative statistics may create the illusion of consistently good performance.

End Point Bias

End Point Bias

Would Forbes have recommended this fund three years earlier? We doubt it. Figure 3 shows us that through February 2000 the fund had under performed its benchmark by 831 percentage points.

End Point Bias

End Point Bias

One way to avoid end point bias is to look at rolling time periods. Figure 5 shows rolling three year periods of excess returns. Here you can see an almost equal amount of time underperforming and outperforming the benchmark. To have confidence that a manager is skillful and that the skill will likely result in beating the benchmark in the future, we prefer managers that consistently outperform.

End Point Bias

End Point Bias

Lets take a look at a consistent outperformer. Figure 6 shows the performance of the Fidelity Low Priced Stock Fund. It outperformed its benchmark by 6.58% annually!

End Point Bias

End Point Bias

Mutual fund analysis

Mutual fund analysis

The bottom panel of Figure 6 contains some useful statistics. One statistical measure of consistency is tracking error, which is the volatility (standard deviation) of excess return. All things equal, the less volatile the excess returns the greater the chance the manager is skillful rather than lucky. Tracking error is used to calculate a risk-adjusted measure of performance called the information ratio. The information ratio is the annualized excess return divided by the tracking error. The information ratio for the Fidelity fund is a very high at 1.48. What are the chances that a manager could have achieved this information ratio by being lucky? Part of the answer will depend on how long she achieves a high information ratio. The longer the good performance persists, the less chance of luck and the more chance of skill.

Mutual fund analysis

Significance Level statistic measures the probability of luck vs. skill. To have confidence that the manager was skillful and not just lucky the significance level should be at least 95%. For the Fidelity Fund it is 100% (see the bottom panel of Figure 6). The most important first step is to select the proper benchmark. If that is not done all of the fancy statistics we have discussed will be meaningless. Investors can accurately measure a managers performance, evaluate the consistency of the performance, and determine the probability that the managers performance is the result of skill. Such an analysis dramatically improves the likelihood that our second assumption our ability to pick skillful managers is true and in doing so that our selections may lead to superior future performance.

Peer Group Analysis

Investors often try to gauge a managers skill by comparing the managers performance to the performance of a group of similar managers. This is typically called a universe or peer group analysis. Many consider a universe or universe composite to be a good benchmark for a manager. In fact, manager universes do not have the qualities that make a good benchmark. They are not investable, they are not specified in advance, and they are usually too broadly defined.

Peer Group Analysis

Universes also suffer from what is known as survivor bias because many of the poor performing managers drop out of the universe. Poorly performing mutual funds, for example, are often merged with more successful funds. When this happens only the successful funds track record is maintained, so the poor performance is not represented in any universe that includes the fund. Once a product no longer exists, for whatever reason, it is dropped out of the universe.

Peer Group Analysis

Despite these limitations, peer group comparisons continue to be popular with investors. Figure 1 is a peer group analysis using the standard floating bar chart. Notice that the floating bar graph shows where Fidelity Magellan ranked in the universe for only six time periods. The RHS graph shows where Magellan ranked every month for the last twenty five years. Looking at the LHS graph one might conclude that Magellan had never been in the bottom quartile of the universe.

Peer Group Analysis

Peer Group Analysis

With the LHS graph we can see that there are two such periods. Managers may not want to show all the periods, so for them the floating bar graph might be best. Sponsors, consultants, and advisors should always use the more comprehensive graph where no time periods can be hidden.

Peer Group Analysis

Another thing to be aware of is the difference between cumulative time periods (last one year, three years, five years etc) and rolling time periods. This relates to end point bias, which is discussed in Mutual Fund Analysis.

Peer Group Analysis

Peer Group Analysis

There is no reason to limit peer group comparisons to returns

Peer Group Analysis

Peer Group Analysis

Peer Group Analysis

A discussion of peer groups wouldnt be complete without some discussion about universe construction. One methodology is to create universes of managers with similar styles.

Peer Group Analysis