Anyone who's ever tried to beat the market over a sustained period of time probably grimaced on reading the above title. They'll know from experience what a daunting task it is. Research shows that only about 1% of active traders actually manage to outperform the most popular reference index—the S&P 500.
In 2007, Warren Buffett famously bet a million dollars that a Vanguard S&P 500 index fund would outshine a basket of hedge funds over a time period of 10 years.
Buffett won that bet easily, as the S&P 500 returned a net 7.1% annually, compared to just a net 2.2% for the hedge funds. This begs the question, why is beating the market so hard?
The Efficient Market Hypothesis
Dubbed ‘EMH’, this investment theory claims that as the market reflects all currently known information, consistently beating the market is impossible. According to this premise, there are no undervalued or overvalued assets; everything is priced exactly as it should be. As a result, there's no space or opportunity to generate extra returns.
Still, some have managed to consistently identify lucrative investment opportunities, so perhaps the reality is that while the market is efficient to a certain extent, it isn't flawless. By and large the market prices things correctly, but it does make the odd mistake here and there. And it's these mistakes that could potentially open the door for generating alpha.
However, this isn't where the difficulty ends.
Even If An Edge Exists...
To beat the market, you have to act differently than other market participants. In the past, one way of doing this was to notice a piece of actionable information missed by the rest. But this was in the days before Information was freely distributed and at a time when products like the Bloomberg Terminal didn’t exist.
Today, financial markets competition is harder than ever on two fronts: first, most professional grade investors have access to the same data in real time and are able to use algorithms for immediate action, and second, Wall Street is home to some of the brightest financial minds, and since every transaction needs a seller and a buyer, outsmarting everyone isn't an easy feat.
Not Many Stocks Overperform
Another reason outperforming the market is so challenging is that not many stocks do better than the broader market. Arizona State University research in 2017 found that four out of seven common stocks have lifetime buy and hold returns lower than one-month Treasuries.
The study showed too, that since 1926, the best performing 4% of listed companies generate the net gain for the entire stock market. For 2018, for instance, that would have included Advanced Micro Devices (NASDAQ:AMD), Netflix (NASDAQ:NFLX), and Amazon (NASDAQ:AMZN), as they rose by 103%, 62%, and 49% respectively, while the broader S&P 500 returned -6.5%.
Building a distinct portfolio raises the odds of an investor missing out on one of the 4% of stocks that could propel its returns high enough to beat the market in a particular year. J.B Heaton explains in “Why Indexing Works”, how, in a five stocks world, if 4 stocks return 10%, and one stock returns 50%, the overall market performance is 18%—but any portfolio without the one good stock would underperform it.
Emotions
The fact that the market makes it difficult to outperform it isn't the only reason most fail to do so. We are, after all, only human, and humans often act in a sub-optimal way, reacting to emotional triggers. Indeed, the effects of fear and greed have been widely documented.
The prospect theory, brought forward by Nobel prize winners Kahenman and Tversky, shows that losing $100 has a stronger negative psychological impact than winning $100 has a positive psychological impact. Humans are risk averse, and this tendency causes us to make the wrong risk/reward choices, which inevitably negatively impacts our returns. In the same way, humans blinded by a potential reward will disregard the risk factor, making themselves vulnerable to negative consequences in the longer term.
Fees and Taxes
And that's not all. To paraphrase Benjamin Franklin, in financial markets, nothing can be said to be certain, except fees and taxes. Buffett’s famous bet was on net return, after obligatory deductions. Buffett chose a tax efficient ETF, but hedge funds' and active traders' higher capital gains on greater turnover naturally leads to more substantial tax bills.
The same turnover creates fees, and crucially, fees are another a major reason so many investors fail to beat the market. A certain percentage of your trading capital goes to fees, which you then have to make up for in returns to equal the market's performance. The more you trade, the more fees you pay, and therefore, the higher of a return you need to make — and we’ve established above that generating alpha is not an easy task.
Conclusion
With all of this adding layer upon layer of difficulty that investors need to overcome, is beating the market actually possible? Investors like Warren Buffet and Peter Lynch — a strong advocate of buy-and-hold investing — have both made their fortunes by stock-picking and beating the market over a long period of time, which proves that it IS possible. But it's never been easy, and now more than ever, it requires a powerful combination of skill, luck, balanced psychology, and smart accounting. Unfortunately for most of us, this winning combination can be frustratingly elusive in our imperfect human world.