Whatever your opinions are about the concept of back-testing, there are few better ways to build a trading system than through the back-testing of historical data.
Back-testing allows us to answer critical questions like ‘how many positions should I hold in a portfolio?’, ‘how much risk should I take?’ or ‘how effective was this strategy in the past?’
Clearly, back-testing historical data is an integral and important part of becoming a systematic trader.
The Problem With Back-Testing
The problem is that back-testing is an extremely tricky business. When it comes to running simulations on historical data there are numerous biases and limitations that can cause even experienced traders a lot of headaches.
Problems like survivorship bias, lookahead bias, data mining bias, overfitting, all can creep into the development process.
How do you know your system is not curve fit? How do you know your data is accurate? And…
How do you know what you don’t know?
No doubt that the wealthiest traders working for hedge funds and banks have access to the best data and the best tools while most of us have to make do with cheaper alternatives.
The cost of a Bloomberg Terminal, for example, runs at around $2000 a month which is beyond the budget of most retail investors.
Having the most accurate data is not always such a big problem but it does become so when dealing at higher frequencies.
Consider that intraday stock market data is often stitched together from a network of different exchanges. Getting accurate intraday data is rarely possible without moving down to a very granular level.
Readers of the book Flash boys by Michael Lewis will be all too aware of the shenanigans that go on at those high frequencies that help to invalidate any simulation.
All this means is that any backtest you make needs to be treated with caution. Because, ultimately, we can never be 100% certain our simulations are accurate.
Back-testing is so fraught with difficulty that we need to treat it a certain way. We need to remain skeptical at all times and we need to be aware of it’s limitations.
New Traders Are The Most At Risk
Backtesting is also the most problematic for new traders. If you are new to system trading and you don’t know the perils of back-testing then you will likely make many mistakes along the way.
This can result in a worse trading experience than if you had just traded randomly without any back-testing at all.
This is because when you have a backtest you will be more confident in your trading, without knowing that it’s inherently flawed!
So those without back-testing experience are most at risk of losing money from faulty systems.
The Back-Testing Spiral
New traders will also get stuck in what I call the back-testing spiral. This is where you spend all your time running back-tests, tweaking and fine tuning your trading system until you find something that looks impressive but is simply overfit to the data.
In this situation you might run hundreds or thousands of tests looking for that perfect system. Then, when you finally start trading live, you lose money straight away because your system was data mined, not robust to the data.
A Fine Balance
In other words, you spent all your time building a trading system that worked perfectly in the past but has hardly any chance of working in the future.
As you can see, backtesting is extremely valuable but only up to a certain point. Go past that point and back-testing becomes ineffective and even dangerous.
Furthermore, since the future is never known back-testing will always have some limitations.
The solution to this problem is the System Traders Feedback Loop where we combine the two phases of back-testing and paper trading to create a smoother transition from our original idea to real trading.
The key is we don’t get stuck in the back-testing spiral. We don’t spend too long tweaking our rules and we don’t rely too heavily on a back-test that might have some flaws.
That means, we do our best to make our back-tests as accurate as possible and as soon as we have found a trading idea that seems good enough in a backtest we move quickly to paper trading so that we can simulate that strategy in the live market and see it’s true potential.
We then use the results we get from live demo trading to feed back into our analysis. Once we have a decent enough sample size we can compare our data from demo trading to our back-testing results.
(We can also use the information from this stage to improve our back-testing processes so we can make more accurate back-tests in the future).
This is going to give us a more realistic view of our trading idea. And it’s also going to provide us with a lot more information than we would gain from a simple back-test.
For example, when we trade the system in the market, we’ll learn subtle nuances like how the system reacts to news events, how the system performs in certain periods or times of day and how the system responds to different order types.
This is all valuable information that we might not get from a back-test alone. So it’s important to track as much of this information as possible and feed it back into the process.
Once we have a system idea that’s working well in our back-test and working well in our demo trading account (and we have a large enough sample – perhaps 30, 50, 100 trades) that’s when we can then make the transition to live trading.
Live trading will be different again, because now we are making real trades with real money. So, all the time we should be analysing all the inputs and outputs of the system. Feeding them back into the loop so that we can improve and assess the strategy as we gain more information.
This is a much more effective approach to creating a trading system because it means we don’t get stuck in the back-testing phase which can be counter productive. And it ensures a better understanding of our trading model.
The Worse Alternative
The alternative to this approach is what most traders do. They run hundreds or thousands of backtests until they find something that worked in the past and then proceed straight to live trading. More often than not this fails because of the many backtesting perils that exist.
Instead, next time you have a trading idea, think first about how you can properly transition from analysis into paper trading and then into live trading so that you can get a better understanding of its potential.
Don’t rely solely on a back-test that may be flawed but think about how you might deploy the system first with a demo account so that you can understand how and whether it works.
This is the whole point of the system traders feedback loop. It gives you the ability to mould a trading strategy and find a solution that really works. A solution that can move in tune with the market and perhaps even get better with time.
Thank You For Reading
Joe Marwood is an independent trader and the founder of Decoding Markets. He worked as a professional futures trader and has a passion for investing and building mechanical trading strategies. If you are interested in more quantitative trading strategies, investing ideas and tutorials make sure to check out our program Marwood Research.
This post expresses the opinions of the writer and is for information, entertainment purposes only. Joe Marwood is not a registered financial advisor or certified analyst. The reader agrees to assume all risk resulting from the application of any of the information provided. Past performance is not a reliable indicator of future returns and financial trading is full of risk. Margin trading can lead to losses more than in your account. Mistakes in backtesting and presenting of analysis regularly occur. Please read the Full disclaimer.