Back testing

From Traderpedia

Definition: Testing a strategy on historical data in order to see if it would have been profitable.

Contents [hide] 1 What is a backtest? 2 The data mining problem 2.1 How to combat the data mining problem 3 Results of a backtest

What is a backtest?

A backtest is a simulation of how a trader with a mechanical trading system would have responded to market data, i.e by buying, selling, or forgoing any transaction at all. Backtesting is a way of testing the signals given by a trading system in order to see whether it would have been profitable in the past.

As with any simulation, the more realistic the backtest, the more useful it is. Obviously, the goal of any trader is to maximize return after slippage and transaction costs (while avoiding unnecessary risk) and these costs should of course be included in the test.

The data mining problem

Just because a strategy would have worked in the past had a traderstumbled upon it does not necessarily mean it is reasonable to think it will continue to work in the future. The Super Bowl Indicator (NFC team wins, market goes up) and the hemline indicator (skirts long, sell short) are obvious examples, but don't be lulled into thinking that any strategy based on more sensible criteria is sure to work. Backtesters must be careful to avoid hyping results based on a limited set of data. There may be something special about the data the backtester used (e.g., it's all January data) that causes anomalous results. Or the results may simply have been caused by chance.

Data mining is even more of a hazard as backtesters use new programming techniques and greater computer horsepower. "Genetic algorithms" sometimes spit out eye-popping numbers when using historical data. It remains to be seen whether these algorithms actually help predict anything. As O'Shaugnessy said in What Works on Wall Street, "Torture the data enough and it will confess to anything."

How to combat the data mining problem

There are three primary ways to combat data mining.

The most obvious is to view with suspicion results that contradict common sense or other well-designed studies.

The second way to avoid data mining is to divide the available historical data into a "play" set and a "confirmation" set. Use the play set to experiment with, and then see if the confirmation set confirms your hypothesis. For example, if you had 5 years of data at your disposal to test an intra-day system, you might have a play set of 24 randomly selected months and a confirmation set of 24 different months. This method is almost sure to tell you whether you are really eliminating problem children or just trying too hard. If you do not have fairly large set of data to start with, you cannot use this technique. Be very cautious in interpreting your results.

The third way is to avoid over-optimising or "curve fitting" your system. By this I mean continually tweaking and adding parameters in ordre to increase the overall profitability. Do this to excess and you are likely to end up with a brilliant system for the period of time under scrutiny, but one that fails miserably as market conditions change. The best systems tend to be fairly simple.

Results of a backtest

Here are the questions most traders want a backtest to answer before putting money into a trading system:

• Does the system provide returns that are significantly better than an appropriate benchmark?
• What is the maximum drawdown of the system?
• Is the system realistic in terms of slippage costs?
• How many trades per year are necessary for the system?
• How long is the typical position kept open in the system?
• What is the ratio of the average winner to the average loser (the profit/loss ratio)?
• What is the ratio of winning to losing trades (the win/loss ratio)?
• What is the maximum adverse excursion?
• What is the system's expected value?
• What are the characteristics of the system's equity curve?
• Is it realistic to assume an entry and exit could always be made when indicated?