Whatever your opinions are about the concept of back-testing, there are few better ways to build a trading system than through the back-testing of historical data.
Back-testing allows us to answer critical questions like ‘how many positions should I hold in a portfolio?’, ‘how much risk should I take?’ or ‘how effective was this strategy in the past?’
Clearly, back-testing historical data is an integral and important part of becoming a systematic trader.
The Problem With Back-Testing
The problem is that back-testing is an extremely tricky business. When it comes to running simulations on historical data there are numerous biases and limitations that can cause even experienced traders a lot of headaches.
Problems like survivorship bias, lookahead bias, data mining bias, overfitting, all can creep into the development process.
How do you know your system is not curve fit? How do you know your data is accurate? And…
How do you know what you don’t know?
No doubt that the wealthiest traders working for hedge funds and banks have access to the best data and the best tools while most of us have to make do with cheaper alternatives.
The cost of a Bloomberg Terminal, for example, runs at around $2000 a month which is beyond the budget of most retail investors.
Having the most accurate data is not always such a big problem but it does become so when dealing at higher frequencies.
Consider that intraday stock market data is often stitched together from a network of different exchanges. Getting accurate intraday data is rarely possible without moving down to a very granular level.
Readers of the book Flash boys by Michael Lewis will be all too aware of the shenanigans that go on at those high frequencies that help to invalidate any simulation.
All this means is that any backtest you make needs to be treated with caution. Because, ultimately, we can never be 100% certain our simulations are accurate.
Back-testing is so fraught with difficulty that we need to treat it a certain way. We need to remain skeptical at all times and we need to be aware of it’s limitations.
New Traders Are The Most At Risk
Backtesting is also the most problematic for new traders. If you are new to system trading and you don’t know the perils of back-testing then you will likely make many mistakes along the way.
This can result in a worse trading experience than if you had just traded randomly without any back-testing at all.
This is because when you have a backtest you will be more confident in your trading, without knowing that it’s inherently flawed!
So those without back-testing experience are most at risk of losing money from faulty systems.
The Back-Testing Spiral
New traders will also get stuck in what I call the back-testing spiral. This is where you spend all your time running back-tests, tweaking and fine tuning your trading system until you find something that looks impressive but is simply overfit to the data.
In this situation you might run hundreds or thousands of tests looking for that perfect system. Then, when you finally start trading live, you lose money straight away because your system was data mined, not robust to the data.
A Fine Balance
In other words, you spent all your time building a trading system that worked perfectly in the past but has hardly any chance of working in the future.
As you can see, backtesting is extremely valuable but only up to a certain point. Go past that point and back-testing becomes ineffective and even dangerous.
Furthermore, since the future is never known back-testing will always have some limitations.
The Solution
The solution to this problem is the System Traders Feedback Loop where we combine the two phases of back-testing and paper trading to create a smoother transition from our original idea to real trading.
The key is we don’t get stuck in the back-testing spiral. We don’t spend too long tweaking our rules and we don’t rely too heavily on a back-test that might have some flaws.
That means, we do our best to make our back-tests as accurate as possible and as soon as we have found a trading idea that seems good enough in a backtest we move quickly to paper trading so that we can simulate that strategy in the live market and see it’s true potential.
We then use the results we get from live demo trading to feed back into our analysis. Once we have a decent enough sample size we can compare our data from demo trading to our back-testing results.
(We can also use the information from this stage to improve our back-testing processes so we can make more accurate back-tests in the future).
This is going to give us a more realistic view of our trading idea. And it’s also going to provide us with a lot more information than we would gain from a simple back-test.
For example, when we trade the system in the market, we’ll learn subtle nuances like how the system reacts to news events, how the system performs in certain periods or times of day and how the system responds to different order types.
Valuable Feedback
This is all valuable information that we might not get from a back-test alone. So it’s important to track as much of this information as possible and feed it back into the process.
Once we have a system idea that’s working well in our back-test and working well in our demo trading account (and we have a large enough sample – perhaps 30, 50, 100 trades) that’s when we can then make the transition to live trading.
Live trading will be different again, because now we are making real trades with real money. So, all the time we should be analysing all the inputs and outputs of the system. Feeding them back into the loop so that we can improve and assess the strategy as we gain more information.
This is a much more effective approach to creating a trading system because it means we don’t get stuck in the back-testing phase which can be counter productive. And it ensures a better understanding of our trading model.
The Worse Alternative
The alternative to this approach is what most traders do. They run hundreds or thousands of backtests until they find something that worked in the past and then proceed straight to live trading. More often than not this fails because of the many backtesting perils that exist.
Instead, next time you have a trading idea, think first about how you can properly transition from analysis into paper trading and then into live trading so that you can get a better understanding of its potential.
Don’t rely solely on a back-test that may be flawed but think about how you might deploy the system first with a demo account so that you can understand how and whether it works.
This is the whole point of the system traders feedback loop. It gives you the ability to mould a trading strategy and find a solution that really works. A solution that can move in tune with the market and perhaps even get better with time.
Thank You For Reading
—
At some point I certainly want to explore the accuracy of back testing in a particular plateform known by the name of ,trackntrade by lan turner but you would need to do the fourteen day test period in order to see my point about a disconnect in results in softs futures for the past ten years. you will be able to set the auto trader on fifteen minute trading in selected softs future and a back trade in the Dow emini for the last month period of time. Use a fifteen minute crossover moving averages. And set the time period only trading hours only for open and close during eastern time zone and see if that coincided with other systems you know about for training call options or day trading. I am very curious about what you find. Your value should be a modest dollar value for profit and losses.
William
Thanks for the comment, I haven’t heard of trackntrade before so will take a look.
Thanks for the post JB. This is really interesting, and I appreciate you walking through some of the finer points of backtesting (especially for new traders!) I totally agree with your approach of paper-trading to collect sample sizes of trades, before moving into the live market.
In my experience, I prefer to use trading systems that intuitively make sense to me. That’s why I like trend following: because I understand under what market conditions it works, and when it does’t. On the other hand, I’ve found when backtesting too extensively I sometimes lose the forest for the trees and come up with esoteric trading systems that might work well in-sample on paper, but I don’t really understand them and thus can’t stick to these less-robust systems.
Thanks again for sharing your thoughts on this and hope you have a great weekend!
Hi Jay.
Absolutely, I have definitely experienced that too where a trading system can work well on paper but it’s hard to follow when you don’t quite understand the logic. That may well be a weakness of mine that I need to work on I think. I’ve found that it can really take some time to get comfortable with a new system.