The Biggest Backtesting Mistakes You Can Make – Warrior Trading


Backtesting is testing buying and selling methods towards historic information.

It serves as a analysis instrument for merchants and analysts to emphasize check their present methods, discover new methods, or discover which issue contributes probably the most to a method’s success. 

Most backtesting is finished with code, utilizing libraries from languages like Python and R and therefore, you’ll be able to just about backtest something if in case you have the coding expertise to create it.

There are additionally some no-code or low-code backtesting options like TradeStation or MultiCharts, however they have an inclination to not be versatile sufficient for professionals. 

An instance of backtesting can be to obtain a decade’s price of market information, let’s say 2007 to 2017, and check a buying and selling technique you’re inquisitive about. You want to have the ability to put the technique right into a strict quantitative format, like “purchase when the 50-day shifting common crosses above the 200-day shifting common,” as an alternative of “purchase a bull flag.”

After the pc is finished operating the check, you get outcomes like an fairness curve, the record of trades, and a few metrics like Sharpe ratio and max drawdown.

Right here’s an instance of a “tearsheet” of analytics generated if you run a backtest on the Zipline Dealer library, an up to date model of now-defunct Quantopian’s Zipline library:


This tearsheet is after all tailor-made by the person for illustration functions. 

Backtesting can really feel like magic.

Obtain a bunch of market information, run some parameters towards it, tweak them a bit till the check reveals a easy fairness curve and presto, you’ve gotten a worthwhile buying and selling system that may print cash. It might look like the one boundaries are studying to code and understanding which indicators and parameters will yield the very best outcomes.

However doing that’s merely information mining and utilizing this technique of backtesting has little utility as a result of your checks could have no predictive energy. It represents a mistake even probably the most refined researchers repeatedly make: mistaking correlation for causation. I’ll show this with an instance: 

You’ve run 1000’s of checks utilizing machine studying to search out probably the most worthwhile buying and selling system.

The most effective check reveals that the optimum technique is to purchase XYZ inventory at 10:53 AM on a Tuesday when on yesterday, the value superior a minimum of 1.04% and the RSI is at 38 or extra. Clearly, this implies nothing. If I instructed you this was my buying and selling system, you’d snicker at me for having such a pointless system that doesn’t exploit any actual imbalances out there.

All this check did is locate the right parameters to suit to that precise historic information. 

Whereas information mining when backtesting is never this blatantly pointless, you must continually verify each bias you introduce right into a check.

Right here’s a number of of the commonest errors which can be essential to keep away from in the event you don’t need to incinerate cash when buying and selling with backtested methods. 

Look-Forward Bias

It’s 2021 and the market has kind of gone straight up for the final decade, save for an enormous crash in 2020.

Understanding this, you’ll be able to go create a backtest towards the historic information to replicate this. For instance, you would possibly say, purchase the S&P 500 on 5x margin, however promote when the value drops 5% or extra in at some point. that this technique will kill it, as a result of it avoids many of the 2020 market crash, whereas reaping large positive aspects from the remainder of the bull market. 

A check like it will look very nice and might make you want you solely had the system in 2010, since you’d be wealthy proper now. 

However you’re principally dishonest.

It’s like watching a replay of a soccer sport, after which concluding that the easiest way to win that sport was to run the ball on 4th & aim, as a result of passing the ball failed.

You’re simply taking a look at what really occurred up to now, and pretending such as you knew the proper reply all alongside. It is perhaps a enjoyable train, however it received’t have any future predictive energy. 

In fact, look-ahead bias creeps into our backtests in additional refined methods. Utilizing the identical bull market instance, momentum methods on progress shares have labored excellently, and merchants who used these methods during the last decade received large.

This may simply lead you to consider that momentum methods are inherently superior and tailor your whole model to what’s working in latest backtests. 

Over Becoming or Over Optimization

This error goes hand-in-hand with look-ahead bias. It’s the act of frequently iterating on a backtest and utilizing earlier backtest outcomes to tell the brand new ones. Whereas this generally is a legitimate follow utilized by many professionals, it’s very straightforward to get mistaken.

Let’s say you’re utilizing a easy RSI imply reversion technique.

You purchase when RSI is beneath 20 and shut the place when RSI is above 60. The backtest outcomes look fairly good, however if you analyze a number of particular person trades you discover that the most important losers end in you shopping for too early; in different phrases, you assume a decrease RSI worth would possibly end in a greater check. So that you strive testing shopping for beneath 15 and promoting above 60.

The check appears higher, however once more, when analyzing particular person trades, you discover that a few of your profitable trades go on runs, so that you resolve to increase your holding interval and shut trades when RSI is above 70. 

This course of would possibly really feel like a productive strategy of slowly optimizing your buying and selling technique to get the very best outcomes, however in actuality, you’re actually simply “p-hacking.”

You’re throwing a number of observations on one set of knowledge till a check reveals the specified outcome. This can be a drawback in lots of scientific fields, which explains that almost all tutorial research aren’t replicable.

Let’s simplify with a real-life instance. You’re keen on basketball and also you need to exhibit your expertise. You report your self throwing ten free-throws over and over till you lastly hit ten in a row. In fact, that’s the video you add to YouTube.

You’re merely performing the identical check over and over till you get the outcome you need. In fact, that’s not consultant of your precise basketball expertise. 

One technique to mitigate this bias is by utilizing out-of-sample information. That is if you check and optimize your information on say, market information from 2007-2012, then when you’re carried out optimizing it, you see in the event you can repeat comparable efficiency on information that your mannequin hasn’t seen, like, 2013-2018.

It’s sort of like endlessly making an attempt to get ten free throws in a row, then, when you’ve carried out it, you check out your new techniques fully recent, to see in the event you can repeat the outcomes. If you happen to hit 4 of ten, your earlier check was most likely an anomaly. 

In fact, there are authentic strategies of optimizing backtest outcomes.

Many closely make the most of walk-forward optimization, which checks the technique on unseen information, stopping you from that snooping bias that we’ve labored so closely on this article.

There are various professionals that don’t use optimization strategies in any respect, and as an alternative follow very tough backtests that check the core concept, quite than particular parameters.

Learn the works of Ernie Chan and Kevin Davey for some good commentary on this situation.

Treating Backtesting As An Engineering Downside

The first impediment to start out backtesting is studying to code and understanding how one can deal with monetary information. In consequence, an enormous variety of programmers hear you can check buying and selling methods by writing some Python and are rightly intrigued.

However being programmers and never skilled merchants, many undergo from treating it as an “engineering drawback,” that means they focus extra on shiny objects like machine studying, high-level math, and funky instruments, quite than determining buying and selling methods which have a logical grounding. 

Certainly not am I saying that machine studying or math aren’t vital. Under no circumstances. However they’re simply instruments. If you happen to don’t have a particular purpose to make the most of them, it’s no completely different than a brand new dealer making an attempt each mixture of technical indicators on TradeStation backtests. 

With trendy laptop {hardware}, it’s a fairly trivial process to run 1000’s of simulations towards monetary information.

You possibly can provide you with every kind of loopy outcomes. You would possibly discover that purchasing oil shares in January after the value of gold has declined three days in a row is a method with 100% win-rate. However give it some thought, why would that work in the true world, going ahead?

What are you exploiting, or what service are you offering to the market by making these trades? 

Neglecting Transaction Prices

Over time, you’ll discover that transaction prices, whereas they’ve declined considerably during the last twenty years, will nonetheless eat up an enormous chunk of your income.

Nevertheless it actually is determined by the kind of technique you’re using. If you happen to’re a hyper-aggressive day dealer who trades high-flying shares, transaction prices eat up an enormous portion of your income.

However the place dealer who holds trades in extremely liquid futures contracts for a number of weeks at a time will likely be much less affected by them.

Illiquid Shares

One “hidden” transaction price is the problem of getting crammed in an illiquid inventory. 

For instance, let’s say you’re using a share class arbitrage technique. It’s very straightforward to idiot your self with this sort of commerce as a result of if you don’t account for the transaction prices, it appears like a excessive sharpe ratio technique. 

We’ll use Lennar’s two share courses for example.

You will have Lennar’s A-class shares, buying and selling underneath the ticker $LEN, which is the extremely liquid aspect of the pair, then you’ve gotten the B-class shares, buying and selling underneath $LEN.B, which trades very seldomly all through the day and commonly has a really huge bid-ask unfold.

The straightforward technique to run a backtest is to imagine you can commerce at or close to the final worth traded.

For devices just like the S&P 500 or mega-cap shares, that is an okay assumption to make if you’re buying and selling in trivial dimension. However, in the event you go and attempt to commerce $LEN.B or a similliary illiquid inventory, you’ll discover that it’s usually tough to get crammed on the final worth, and if you wish to get crammed immediately, you’ll must cross the unfold, and take liquidity out of the market. 

And, more often than not, this crossing of the unfold negates the profitability of the commerce altogether.

There’s a number of different pesky points (which you can argue are additionally hidden transaction prices which can be tough to mannequin for) that get in the best way of executing this technique efficiently, however these are past the scope of this part.

Normally, transaction prices are probably the most influential to some forms of buying and selling methods, making these way more tough to get a practical backtest:

  • Multi-legged trades, like pairs trades, or arbitrage trades.
  • Increased-frequency buying and selling, like intraday buying and selling.
  • Buying and selling much less liquid merchandise, like microcap and small-cap shares.

In pairs trades, you must cross the unfold or present liquidity in two completely different shares, and also you’ll usually get crammed at completely different occasions for every.

A better frequency of trades means you’re paying commissions and crossing the unfold extra usually (or, in the event you’re passive, you usually received’t get the identical worth in stay buying and selling that your mannequin assumes you’ll get). 

And also you sort of must throw the e book away when buying and selling much less liquid merchandise as a result of a historic backtest can’t let you know when somebody will come round to commerce with you, and at which worth. 

How Do I Study to Backtest Buying and selling Methods?

There’s but to be a streamlined platform like CodeAcademy for studying to backtest buying and selling methods that I do know of.

And most tutorials are tailor-made to a particular library or supply of knowledge. Many examples, for instance, depend on Zipline, the backtesting library for Python that now-defunct Quantopian developed. However the firm stopped sustaining the library after they had been acquired by Robinhood. 

In order of now, since there’s no “official” path to be taught backtesting, your studying path actually is determined by your stage of competency with programming.

If you happen to’re model new, you’re greatest off beginning with Python, as a result of it’s the most well-liked language for manipulating monetary information and is fairly straightforward to get began with. DataCamp has some respectable programs that train you to fiddle with monetary information, however it’s not very trading-focused. 

A good e book with some code “recipes” is Andreas Clenow’s Buying and selling Developed. Sure, it makes use of the Zipline library which is now not maintained, however in the event you’re model new, a very powerful side is knowing how one can manipulate monetary information.

When you’ve discovered the important libraries for this: numpy and pandas, it turns into loads simpler to go and piece collectively your individual algorithms from StackOverflow and associated web sites. 

Backside Line

It’s approach too straightforward to be fooled by a backtest. Refined high-net price buyers are regularly fooled into investing into questionable hedge funds primarily based on fancy backtests and simulations.

It’s wholesome to be cynical if you see any backtest outcomes, however don’t be a defeatist. There’s a purpose that backtesting is an important instrument within the skilled dealer’s toolbox. The issue isn’t with backtesting, it’s with the misunderstanding {that a} profitable backtest “proves” something.

It’s a analysis instrument, and when approached that approach, it’s rewarding.

Add Comment

Click here to post a comment

Leave a Reply