Evaluate Skill, Not Style: An Invitation
Ron Surz, PPCA Inc.
949/488-8339

The motto of professional investment performance evaluators has long been "Evaluate skill, not luck." However, about five years ago, those of us in the evaluation business became aware of the significance of investment style in measuring skill. That is, we learned that skill could only be properly identified if we first lifted the thick clouds of style that routinely distort our perspective. The immediate past is a good case in point. Damning growth stock managers for their recent losses is folly because it's style, not skill, that is the culprit.

Accordingly, the motto of 21st century evaluators should be "Evaluate skill, not style." That said, many evaluators are talking the talk without walking the walk. Although most say they understand the importance of style, many either ignore style altogether by using the S&P 500 for everyone or settle for an off-the-shelf style benchmark. We know today that most managers are best represented by a blend of style benchmarks; each manager truly is unique. Being good diplomats, investment managers usually attack the indexes rather than the consultants and their clients. It's become a game that appears to be tolerated despite its foolishness. Investment managers accept accolades, and often more, when their style is in favor relative to the benchmark's style, and then cry foul when the reverse occurs.

In the spirit of promoting the game, prominent consultants are frequently quoted saying things along the lines of "We primarily use the S&P 500 because it's what clients are expecting--it's what they're going to ask about anyway--because they see it on the nightly news." Sound like a cop-out to you? At the risk of spoiling the fun, this article lays out a simple prescription for avoiding the pitfall of evaluating style rather than skill, thereby inviting you to apply the knowledge we've acquired about style.

The Prescription

The comedian Steve Martin used to do a routine on how to become a millionaire, which began: First, get a million dollars. Our prescription for sound performance evaluation and attribution begins on a similar note: First, get the right style benchmark. This can be accomplished by using a returns-based style analysis that incorporates a good family of style indexes, or you can run a history of holdings-based analyses. More on this later, but suffice it to say that we have the technology today to do a reasonable job of capturing the manager's true style essence. With a good style profile as the basis, we can proceed with the plan, as follows:

Process for Evaluating Skill, Not Style

(1) Calculate the return expected from passive implementation of the manager's style.

(2) Subtract this from the actual return. What's left is the value added by skill.

(3) Separate out the sources of skill: sector allocation and stock selection.

(4) Look for persistence as confirmation of skill rather than luck

The first three steps are shown schematically above in Exhibit 1. We'll use this schematic to illustrate a real analysis that followed the above process.

Exhibit 2: Example of Attribution That Separates Style from Skill

An Example

Exhibit 2 shows a sample analysis that accomplishes the first three steps. Note the row on the bottom labeled "Style Benchmark," which shows the style profile that is being used to evaluate the manager. In this case, it's large-mid Growth, with some Value. The floating bars in the chart above show the Portfolio Opportunity Distributions (PODs) within each market sector for the specified style. For example, the technology bar represents the range of return opportunities for all of the possible portfolios of technology stocks that fit the style profile. The middle, or median, of the bar shows the expected return in the sector for the specified style. Moving to the bar all the way on the right, note that the expected total return for this manager's style in this period is the -4.3% shown both as the median of the bar and in the table beneath the bar. Also note the red line in the graph indicating the style's natural allocation to economic sectors. This is used to see where the manager has made sector bets.

The manager's performance results are shown as the red dots in the exhibit. Since these results are generally above median, we can see that the manager generally made superior stock selections. The total fund performance of -0.8% ranks in the 27th percentile against the unique style opportunity set. As shown by the solid blue area, the manager also made a bet on the finance sector during the quarter. Since finance was the poorest performing sector for the quarter, this bet did not pay off. So this outperformance was due to good stock selection, offset by poor sector allocation.

( See Appendix for details)

Using standard attribution arithmetic, as shown in the appendix, we can summarize the results of Exhibit 2 in Exhibit 3. When viewed from a customized style perspective, this was a pretty good quarter for the manager. However, the manager's style--large-mid Growth, with some Value--was out of favor, losing 4.3% in a market that was flat. As a result, evaluators who use the S&P 500, or other standardized benchmarks, will conclude that this same manager underperformed. It's not fair to the manager, but most importantly, this conclusion could lead the client to fire a manager with a demonstrated talent in stock selection--a serious mistake.

The final step in our evaluation is looking for persistence. Exhibit 4 shows the history of value added by the sample manager over the past 12 quarters and introduces a new measure: activity. Activity is the difference between the actual return and the buy-and-hold return. It measures the value added or subtracted by the manager's trading decisions during the quarter. Note that activity and stock selection have been the biggest contributors to this manager's cumulative performance, with most of the added value from activity coming in just two quarters, i.e., the fourth quarters of 1999 and 2000. Similarly, most of the stock selection success came in the first and third quarters of 2000. Overall, a good picture, but you'd want to understand those key quarters more thoroughly. Evaluating skill is not as simple as "I'll know it when I see it," is it?

The Best Tools

The examples we've discussed use both returns-based and holdings-based analyses. Returns-based analysis can be helpful in establishing the style benchmark. Using fairly complex regression techniques, returns-based style analysis tells the evaluator the combination of styles that best matches the performance history of the manager. We've found that a regression covering the past three years works well in capturing the manager's current style. We've also found that you need to be careful with returns-based style analysis on several fronts. First, the family of style indexes that make up the style palette should be the best available; we think the Surz styles work best. Second, you want to be sure that the style fit is close as evidenced by a reasonably high correlation. Finally, the style profile should fit your understanding of the manager.

As an alternative to returns-based style analysis you could use point-in-time holdings-based analysis, over a sequence of past time periods, to determine the appropriate style benchmark. This is a little more work, but it may be worth the effort. Regardless of your approach for establishing the style benchmark, there's only one way to perform attribution analysis. Returns-based analysis cannot distinguish among the sources of return--i.e., stock and sector selection, and activity--that cause actual performance to differ from that of the style profile, nor can it detect style drift. To detect style drift and attribute performance to sector and stock selection, you need holdings-based analysis. Holdings-based analysis classifies the securities held in the portfolio into styles and sectors. It's actually much simpler to calculate than returns-based analysis and easier to understand because it's based on what's actually in the portfolio, rather than what appears to be in the portfolio.

Summary

These two technologies--returns-based style analysis and holdings-based attribution analysis--are the keys to properly evaluating skill, rather than style. Since these technologies weren't available as recently as five years ago, evaluators had to settle for less back then. Today you can do a much better job for your clients than the one-size-fits-all mistakes that have characterized the evaluation industry since its beginnings.

Appendix: Attribution Arithmetic