sign up here Sign Up | About | Privacy Policy | Transaction Policy | Legal Disclaimers | Contact Us | Home  
women investing online investing woman's funds


Consumer Reports: Mutual Fund Review - Critique

Consumer Reports is a U.S. magazine that is widely respected for the advice it provides to people who are facing difficult purchasing decisions for expensive items like autos, cameras, computers, televisions and washers. Their typical approach is to define the best product evaluation criteria to use (as well as their respective weights), and then use them to rate different options. Technically, this is known as "multi-attribute utility analysis" and in general it is a solid approach to making complex decisions. However, there are situations where it is not appropriate, at least in its traditional form. In these cases, the use of multi-attribute utility analysis runs the risk of producing answers that are at best misleading and sometimes flat-out wrong. Unfortunately, in its February, 2005 issue on choosing mutual funds, Consumer Reports appears to have made just this mistake.

The article starts out on a promising tack: "It happens all the time. While reading the newspaper or watching a finance program on TV, you learn about a mutual fund whose returns are zooming upward. You check it out and learn that it outpaced the Standard and Poor's 500 Index not only in the last year, but also for the last three years and five years. Congratulating yourself on finding a solid investment, you write out a check. Almost immediately, however, the fund plunges, and it stays in a performance trough for what seems like years. Why is it, you ask yourself, that every mutual fund you buy drops like a rock in a well the moment you invest? Don't think for a moment that your experience is unique. Millions of [investors] wind up buying high and selling low. There's an explanation: A mutual fund, like most other investments, may hit highs once in a while but perform listlessly most of the time. And the highs may be high enough to make the fund's long-term returns look respectable…"

The article goes on to quote an analyst from Morningstar, who notes that "most funds that end up leading or lagging the pack for short periods are very concentrated in certain areas." Consumer Reports notes that "as soon as these funds cool off, their performance chills. That's why funds at the top of the annual performance charts seldom repeat from one year to next."
So far, so good. The article even goes on to note the potential advantages of investing in index funds: "There are ways to avoid this investment roller coaster. One is to invest in index funds or exchange traded funds (ETFs). Both mirror the performance of a broad market index…You earn exactly what the market earns, minus expenses." Unfortunately, in the next sentence, Consumer Reports begins to go off the tracks: "[But there is a] downside [to index funds]: You can never do better than the index, and when it drops, you're stuck with losses."

Consumer Reports then proposes an alternative: "Choose funds that outperform the market -- not just once or twice a decade, but repeatedly. With such funds, you would not have to try to time market swings, a pursuit that baffles even professionals." The article then describes the approach Consumer Reports used to identify the actively managed mutual funds it recommends to investors: Starting with the 1,327 actively managed U.S. equity mutual funds that had at least a ten-year performance history, it applied the following screening criteria and weights: (1) The standard deviation of the fund's monthly returns over five and ten years (40% weighting); (2) The fund's excess return versus either the Standard and Poor's 500 Index (for large cap, mid-cap, asset-allocation, and specialty funds) or the Russell 2000 Index (for small cap funds), weighted 40%; (3) The fund's worst one and two year results, weighted 15%, and (4) The length of time the current manager has been managing the fund, weighted 5%. From the funds that passed this screen, Consumer Reports eliminated those with annual expenses of more than 1.6%, and those that received a failing fiduciary grade from Morningstar.

What does this mean in practical terms? Consumer Reports appears to making the assumption that people making investments are pursuing multiple objectives, including (a) beating the annual performance of a benchmark index; (b) minimizing the variability of annual returns; and (c) avoiding significant losses. Furthermore, it would appear that Consumer Reports assumes that most investors believe that low expenses, long manager tenure, and passing fiduciary grades are associated with the achievement of some or all of these objectives.

After applying these screening criteria, Consumer Reports was left with 70 funds (or 5.28% of the 1,327 it started with). Of these 70 funds, 18 were large cap (1 growth, 5 blend, and 12 value), 19 were mid-cap (7 blend, 12 value), 15 were small cap (6 blend and 9 value), 11 were asset allocation funds (that switch between bonds and equities), 4 were real estate investment trusts (REITS), and three were industry sector funds (one each in health care, defense, and natural resources).

Saying that we disagree with Consumer Reports' analytical methodology and conclusions is, to put it mildly, a vast understatement.

Let us start with the nature of the investor objectives assumed by Consumer Reports. As we have repeatedly written, we do not believe that beating the annual performance of a benchmark index should be the primary goal of prudent investors. Rather, these investors would be much better off if they focused on achieving a long-term (i.e., a compound annual, or, to put it differently, a geometric average) rate of return that is sufficient to achieve their goals over their target time horizon. This is equivalent to saying that a pension fund should focus on adequately funding its long-term liabilities (at the lowest possible risk) rather than beating a market index.

We have noted many times that the reference point you use to measure your investment performance is critical. Daniel Kahneman won the 2002 Nobel Prize in Economics for his writing on this subject. To oversimplify, a key insight is that for most people, "losing hurts twice as much as winning feels good." If you use the annual performance of an index as your benchmark, you will regularly be tempted to bail out of investments that have had a bad year, and move them into investments that have just had a good one. More often than not, this leads to the dreaded "buy high, sell low" outcome.

The alternative is to focus not on the performance of individual asset classes versus a benchmark (or benchmarks), but rather on the overall performance of your portfolio compared to your long-term goals. This forces you evaluate your performance over longer periods of time than one year, and substantially reduces the temptation to act precipitously (and often harmfully) on the basis of relatively little information.

However, if we take Consumer Reports' "annually beat the index" goal as a given, we have another problem with their choice of minimizing the total standard deviation of a fund's returns as an objective. In point of fact, the returns on the actively managed funds cited by Consumer Reports are a function of two factors: the style tilt they take away from the broad equity market index (e.g., toward small caps , and/or value), and manager skill. I can obtain the benefits of the style tilt quite cheaply by investing in a fund that tracks an appropriate index (e.g., a small cap value index fund). Therefore, the real issue is how much additional return the actively managed fund provides relative to the additional risks it takes to obtain it. What counts is not the overall volatility (standard deviation) of the actively managed fund, but rather the volatility of its returns compared to those on the index fund. In technical terms, returns relative to the index fund are called "alpha", and the additional risk relative to the index fund is called "tracking error" (which is the standard deviation of the alphas). Consumer Reports made a major error in using the alphas of its actively managed funds and their total volatility. The correct approach would have been to relate their alphas to their tracking errors. This is called the "Information Ratio," and it is quite well known. We don't understand why Consumer Reports did not use it.

We also question the other two screening criteria used by Consumer Reports. What does the fact that a manager has been with a mutual fund for a long time really tell you? That he or she has yet to get a high enough offer from a hedge fund? Or that he or she doesn't want to go to a hedge fund because -- why? Fear that he or she is losing his or her edge? Or something else? Who really knows? On the other hand, we agree with Consumer Reports' implicit assumption that investors' risk preferences encompass more than volatility (standard deviation). Kahneman's work showed that they also prefer to avoid large "drawdowns", or negative returns (technically, this is a function of skewness and kurtosis). However, we disagree with Consumer Reports apparent view that this preference should be taken into account at the fund level. Rather, it should apply to portfolio returns, which include the beneficial impact of diversification across asset classes.

Another major problem we have with the Consumer Reports article is its unstated assumption that successful active management is possible for great numbers of people (if only they had the sense to use the Consumer Reports funds screening process). We have written at length about the many challenges that make consistent active management success such a rarity (see, for example, in the Members' Section of our site "The Case for Active Management"). Let me briefly summarize them. First, successful active management ultimately rests on successful forecasting. In turn, this must rest on some combination of superior information and/or a superior model for making sense of it. Active managers' ability to consistently obtain superior information has been dramatically reduced due to both technological changes (e.g., the internet) and regulatory changes (e.g., Securities and Exchange Commission Regulation FD in the United States). As for superior models, most are eventually made obsolete by some combination of copying by competitors and/or changes in the underlying economy that invalidate their key assumptions.

Second, even those managers in possession of superior information or a superior model often find themselves unable to fully act on it. For example, most mutual fund managers are prevented from taking short positions in a security, or from adding leverage to maximize the benefits from their insights. Other active managers face constraints on the maximum amount of active risk they can take in pursuit of "better than index fund" returns. Finally, as people pour more money into high performing mutual funds, their managers typically find it exponentially more difficult to find ever larger, but still highly profitable investment opportunities.

Of course, many of these limitations do not apply to hedge funds (though the challenge of managing a larger fund certainly does). However, even if you have sufficient wealth to qualify as a hedge fund investor, you are still left with the challenge of identifying truly skilled active managers. Unfortunately, research has shown that impressive track records aren't any help. Many of them are not statistically different from what would expect from luck alone, and even those that are have been shown to be poor predictors of future performance (like we said, advantages based on superior information and superior models have increasingly short half-lives). The inescapable reality is that it is much easier to identify Warren Buffet than it is his successor.

But suppose you think you have. You are still left with the fourth challenge: negotiating a compensation arrangement with this superstar active manager that will leave you feeling confident that you will be better off than you would have been had you invested in an index fund. Given that many hedge fund managers charge fees equal to 2% of their assets under management, plus 20% of the profits they generate, this is no easy task (though it certainly explains why so many mutual fund managers have left to run hedge funds!).

In short, one reaction to the Consumer Reports article is "so what?" They have identified a small number of funds that, by their standards, have "outperformed" over the past ten years. Our response is that plenty of research shows that this provides no statistically significant insight about whether they will outperform over the next ten years.

But let's go a step further and ask whether Consumer Reports' 70 star mutual funds have actually "outperformed." The first thing we insist on is a fair, "apples to apples" comparison. Unfortunately, Consumer Reports mostly doesn't provide one. For example, included in the 70 funds are 11 "asset allocation" funds that switch between stocks and bonds. That they have "outperformed" the S&P 500 Index tells us very little. It may well have been the case that all of them would have been outperformed by a mix of stock and bond index funds. The only fair way to make this comparison would have been to (1) identify each of these fund's basic stock/bond asset allocation (also known as its "policy mix"), (2) construct an index based on this mix, and then (3) see if its active management (i.e., departures from its policy mix) caused the fund to outperform the index fund benchmark. Unfortunately, Consumer Reports did not do this analysis, so we will eliminate the asset allocation funds from out analysis.

We will also eliminate the REIT funds, as they are, in our view, a separate asset class from the domestic equity funds. We will also eliminate the three narrowly focused sector funds, as their performance was not compared to an equally narrow index.

This leaves us with 52 general equity funds. As we already noted, Consumer Reports measured these funds' "excess returns" (i.e., their "alpha") using either the Standard and Poors 500 or the Russell 2000 Index. We believe this is an incorrect methodology. Specifically, it conflates true active management skill with the benefits of style tilts that could have been taken, at much lower cost, via index funds. For example, the proper benchmark for an actively managed, large cap value fund is not the S&P 500 (which includes a mix of "growth" and "value" companies); rather, it is a fund that tracks a large cap value index. In our analysis, we used the following funds as benchmarks:

Equity Style Tilt Benchmark Fund
Large Cap Growth Vanguard Large Cap Growth (VIGRX)
Large Cap Blend Vanguard S&P 500 (VFINX)
Large Cap Value Vanguard Large Cap Value (VIVAX)
Mid Cap Blend Dreyfus Mid-Cap Index1
Mid Cap Value S&P Mid-Cap 400 Value ETF (IJJ)2
Small Cap Blend Vanguard Small Cap (NAESX)
Small Cap Value Vanguard Small Cap Value (VISVX) 3

Before reviewing the result of our fund alpha, tracking error, and information ratio calculations, we should also briefly note two other issues that were not covered in the Consumer Reports article. First, why should an investor even bother to take a tilt away from a broad market equity fund (e.g., the Vanguard Total Market Fund -- VTSMX for the mutual fund shares, and VTI for the exchange traded shares)? There are two conflicting schools of thought on this issue. Those who assume that markets are reasonably efficient, expect a tilt away from the broad market index to produce either a higher return with higher risk, or lower return with lower risk. This school of thought sees the so-called "value" and "small stock" return premiums as compensation for taking on higher levels of risk relative to the broad market index.

In contrast, the behavioral finance school believes that it is possible for a tilt to produce higher returns with lower risk. However, this belief rests on two critical assumptions: first, that some investors will regularly make predictable mistakes, and second, that there are significant barriers to other investors exploiting them, and arbitraging away the potentially higher returns that results from the actions of irrational investors. While we recognize that there is plenty of evidence for the existence of systematic mistakes, evidence for the existence of substantial and persistent barriers to arbitrage is much less impressive. We therefore take the position that, while markets are not perfectly efficient, they are strongly attracted to it. Hence we believe that the most reasonable expectation for the result of taking a tilt is either higher returns with higher risk, or lower returns with lower risk, but not a free lunch.

The second issue relates to the "right" index to use when calculating an actively managed fund's alpha. Reasonable people can and do disagree on this question. For example, in previous articles, we have extensively reviewed the impact of taking mid and small cap tilts using funds based on different indexes. Particularly in the area of small caps, different indexes can produce significantly different results.

With these important caveats in mind, let us now turn to our analysis of the 52 actively managed large, mid and small cap funds that passed the Consumer Reports screening criteria. We used the following methodology: (a) for each year between 1995 and 2004, we subtracted the return on the appropriate index fund from the return on the actively managed fund. (b) We calculated the average of these differences to determine each fund's alpha. (c) We calculated the standard deviation of these differences to determine each fund's tracking error. (d) We divided each fund's alpha by its tracking error to determine its Information Ratio, or IR. (e) We tested to see if the fund's IR was statistically different from zero. (For the technically inclined, we did this by comparing the t-statistics for the distribution of each fund's alpha to the 2.262 needed to achieve a 95% confidence level with 9 degrees of freedom). To put this last step in plain English, we tested to see if we could be 95% sure that a fund's IR was not due to luck.

We found the following: (1) When calculated using appropriate index benchmarks, many of the 52 Consumer Reports funds actually had negative alphas. For the whole sample, the average alpha was only 0.17%. (2) Many of the funds also had quite high tracking errors. In other words, to generate alpha they were taking substantially more risk than the comparable index fund. Across the 52 funds, tracking error averaged 9.66%. (3) Not one of the 52 actively managed equity funds selected by Consumer Reports had an Information Ratio that was significantly different from zero. The two funds that came the closest to passing this test were the Dodge and Cox Stock Fund (DODGX) and the Meridian Value Fund (MVALX), but even their Information Ratios fell well short of statistical significance.

In sum, Consumer Reports is recommending actively managed mutual funds, whose expense levels and tax costs are demonstrably higher than those on comparable index funds, without any statistically significant evidence that the active funds' risk/return trade-off is superior. This is a big and unusual mistake by an organization that is generally extremely reliable. Our fundamental conclusion therefore remains unchanged: While some degree of active management is necessary to maintain reasonably efficient markets, the great majority of investors are (still) best advised to invest their hard-earned savings in a well-diversified portfolio of index funds.

If you found this article to be useful and informative, more is available every month along with all of our archives back to 1997. We hope you subscribe today. A year subscription for The Index Investor is only U.S. $195.

Download Free Section PDF
Subscribe to The Index Investor

::: Take me to: :::