• Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Decoding Markets

  • Top Posts
  • Courses
  • Resources
    • Books
  • Free Class
  • Blog
You are here: Home / Amibroker / The System Traders Feedback Loop – Don’t Get Stuck In The Back-Testing Spiral

April 28, 2017 By Joe Marwood 6 Comments

The System Traders Feedback Loop – Don’t Get Stuck In The Back-Testing Spiral

Whatever your opinions are about the concept of back-testing, there are few better ways to build a trading system than through the back-testing of historical data.

Back-testing allows us to answer critical questions like ‘how many positions should I hold in a portfolio?’, ‘how much risk should I take?’ or ‘how effective was this strategy in the past?’

Clearly, back-testing historical data is an integral and important part of becoming a systematic trader.

The Problem With Back-Testing

The problem is that back-testing is an extremely tricky business. When it comes to running simulations on historical data there are numerous biases and limitations that can cause even experienced traders a lot of headaches.

Problems like survivorship bias, lookahead bias, data mining bias, overfitting, all can creep into the development process.

How do you know your system is not curve fit? How do you know your data is accurate? And…

How do you know what you don’t know?

No doubt that the wealthiest traders working for hedge funds and banks have access to the best data and the best tools while most of us have to make do with cheaper alternatives.

The cost of a Bloomberg Terminal, for example, runs at around $2000 a month which is beyond the budget of most retail investors.

Having the most accurate data is not always such a big problem but it does become so when dealing at higher frequencies.

Consider that intraday stock market data is often stitched together from a network of different exchanges. Getting accurate intraday data is rarely possible without moving down to a very granular level.

Readers of the book Flash boys by Michael Lewis will be all too aware of the shenanigans that go on at those high frequencies that help to invalidate any simulation.

All this means is that any backtest you make needs to be treated with caution. Because, ultimately, we can never be 100% certain our simulations are accurate.

Back-testing is so fraught with difficulty that we need to treat it a certain way. We need to remain skeptical at all times and we need to be aware of it’s limitations.

New Traders Are The Most At Risk

Backtesting is also the most problematic for new traders. If you are new to system trading and you don’t know the perils of back-testing then you will likely make many mistakes along the way.

This can result in a worse trading experience than if you had just traded randomly without any back-testing at all.

This is because when you have a backtest you will be more confident in your trading, without knowing that it’s inherently flawed!

So those without back-testing experience are most at risk of losing money from faulty systems.

The Back-Testing Spiral

New traders will also get stuck in what I call the back-testing spiral. This is where you spend all your time running back-tests, tweaking and fine tuning your trading system until you find something that looks impressive but is simply overfit to the data.

In this situation you might run hundreds or thousands of tests looking for that perfect system. Then, when you finally start trading live, you lose money straight away because your system was data mined, not robust to the data.

A Fine Balance

In other words, you spent all your time building a trading system that worked perfectly in the past but has hardly any chance of working in the future.

As you can see, backtesting is extremely valuable but only up to a certain point. Go past that point and back-testing becomes ineffective and even dangerous.

Furthermore, since the future is never known back-testing will always have some limitations.

The Solution

The solution to this problem is the System Traders Feedback Loop where we combine the two phases of back-testing and paper trading to create a smoother transition from our original idea to real trading.

system traders feedback loop

The key is we don’t get stuck in the back-testing spiral. We don’t spend too long tweaking our rules and we don’t rely too heavily on a back-test that might have some flaws.

That means, we do our best to make our back-tests as accurate as possible and as soon as we have found a trading idea that seems good enough in a backtest we move quickly to paper trading so that we can simulate that strategy in the live market and see it’s true potential.

We then use the results we get from live demo trading to feed back into our analysis. Once we have a decent enough sample size we can compare our data from demo trading to our back-testing results.

(We can also use the information from this stage to improve our back-testing processes so we can make more accurate back-tests in the future).

This is going to give us a more realistic view of our trading idea. And it’s also going to provide us with a lot more information than we would gain from a simple back-test.

For example, when we trade the system in the market, we’ll learn subtle nuances like how the system reacts to news events, how the system performs in certain periods or times of day and how the system responds to different order types.

Valuable Feedback

This is all valuable information that we might not get from a back-test alone. So it’s important to track as much of this information as possible and feed it back into the process.

Once we have a system idea that’s working well in our back-test and working well in our demo trading account (and we have a large enough sample – perhaps 30, 50, 100 trades) that’s when we can then make the transition to live trading.

Live trading will be different again, because now we are making real trades with real money. So, all the time we should be analysing all the inputs and outputs of the system. Feeding them back into the loop so that we can improve and assess the strategy as we gain more information.

This is a much more effective approach to creating a trading system because it means we don’t get stuck in the back-testing phase which can be counter productive. And it ensures a better understanding of our trading model.

The Worse Alternative

The alternative to this approach is what most traders do. They run hundreds or thousands of backtests until they find something that worked in the past and then proceed straight to live trading. More often than not this fails because of the many backtesting perils that exist.

Instead, next time you have a trading idea, think first about how you can properly transition from analysis into paper trading and then into live trading so that you can get a better understanding of its potential.

Don’t rely solely on a back-test that may be flawed but think about how you might deploy the system first with a demo account so that you can understand how and whether it works.

This is the whole point of the system traders feedback loop. It gives you the ability to mould a trading strategy and find a solution that really works. A solution that can move in tune with the market and perhaps even get better with time.


Thank You For Reading

joe marwood profile pictureJoe Marwood is an independent trader and the founder of Decoding Markets. He worked as a professional futures trader and has a passion for investing and building mechanical trading strategies. If you are interested in more quantitative trading strategies, investing ideas and tutorials make sure to check out our program Marwood Research.


Disclaimer

This post expresses the opinions of the writer and is for information, entertainment purposes only. Joe Marwood is not a registered financial advisor or certified analyst. The reader agrees to assume all risk resulting from the application of any of the information provided. Past performance is not a reliable indicator of future returns and financial trading is full of risk. Margin trading can lead to losses more than in your account. Mistakes in backtesting and presenting of analysis regularly occur. Please read the Full disclaimer.

Filed Under: Amibroker, How to, Strategies/ Systems, Top Posts Tagged With: amibroker, trading systems

Recommended Educational Resources:


  • Discover New Trading Strategies
  • Learn Japanese Candlesticks (Free)
  • Mental Models For Trading (Free)
  • Indicators For Amibroker (Free)
  • Tools For Options Traders (Free)
  • Historical Data For 8000 Stocks (Free)

Reader Interactions

Comments

  1. William says

    April 29, 2017 at 9:48 am

    At some point I certainly want to explore the accuracy of back testing in a particular plateform known by the name of ,trackntrade by lan turner but you would need to do the fourteen day test period in order to see my point about a disconnect in results in softs futures for the past ten years. you will be able to set the auto trader on fifteen minute trading in selected softs future and a back trade in the Dow emini for the last month period of time. Use a fifteen minute crossover moving averages. And set the time period only trading hours only for open and close during eastern time zone and see if that coincided with other systems you know about for training call options or day trading. I am very curious about what you find. Your value should be a modest dollar value for profit and losses.
    William

    Reply
    • JB Marwood says

      May 1, 2017 at 10:38 am

      Thanks for the comment, I haven’t heard of trackntrade before so will take a look.

      Reply
  2. Jay says

    April 30, 2017 at 2:18 am

    Thanks for the post JB. This is really interesting, and I appreciate you walking through some of the finer points of backtesting (especially for new traders!) I totally agree with your approach of paper-trading to collect sample sizes of trades, before moving into the live market.

    In my experience, I prefer to use trading systems that intuitively make sense to me. That’s why I like trend following: because I understand under what market conditions it works, and when it does’t. On the other hand, I’ve found when backtesting too extensively I sometimes lose the forest for the trees and come up with esoteric trading systems that might work well in-sample on paper, but I don’t really understand them and thus can’t stick to these less-robust systems.

    Thanks again for sharing your thoughts on this and hope you have a great weekend!

    Reply
    • JB Marwood says

      May 1, 2017 at 10:37 am

      Hi Jay.
      Absolutely, I have definitely experienced that too where a trading system can work well on paper but it’s hard to follow when you don’t quite understand the logic. That may well be a weakness of mine that I need to work on I think. I’ve found that it can really take some time to get comfortable with a new system.

      Reply
  3. Richard Johnson says

    November 21, 2020 at 11:35 am

    These kinds of articles are silly. If you have a system that backtested successfully for 6 months and you then spend 1 month ‘paper trading’ all you’re doing is backtesting your system for 7 months. Even after 7 months there’s no guarantee it’ll work for another month. So you spend another month paper trading.. then you have 8 months data but there’s no guarantee it’ll work for another month. So you might as well go straight to live. If you are too afraid to do it then either a) scrap your system and accept if you’re frightened to try a profitable system live you haven’t got the heart to be a trader or b) don’t waste anymore time and go get a job in an office.

    Reply
    • Joe Marwood says

      November 21, 2020 at 10:26 pm

      Thank you for your comment. However, it has very little to do with the number of months being backtested or being ‘frightened’. It is much more about gathering an adequate sample size and building confidence in the backtest results.

      Reply

Leave a Reply Cancel reply

Your email address will not be published.

Primary Sidebar

Access Marwood Research image

Popular

The Tycoon Barbell from The Safe Investor

How To Limit Risk And Capture Upside With A Barbell Trading Strategy

21 Useful Ways To Improve Your Trading System

beware youtube trading videos

Beware YouTube Trading Videos

does technical analysis work? one in 5 hedge funds use it

Does Technical Analysis Work? Evidence From The Social Sciences

easy-money-trading

The Easiest 100 Ticks I Ever Made (And Worst)

Performance Of Our Trading Systems In 2019

Investing With Intelligent Machines

RSI trade divergence example

Testing 3 Simple Divergence Trading Strategies

Search

Recommended Resources

  • Discover New Trading Strategies
  • Learn Japanese Candlesticks (Free)
  • Mental Models For Trading (Free)
  • Indicators For Amibroker (Free)
  • Tools For Options Traders (Free)
  • Hist. Data For 8000 Stocks (Free)

Financial Disclaimer

Financial trading is risky and you can lose money. Joe Marwood is not a registered investment advisor and nothing on this site is to be regarded as personalized investment advice. Past performance is not indicative of future results. Data errors and mistakes do occur. Please see the full disclaimer.

Footer

About The Author

Joe Marwood is an independent trader and investor specialising in financial market analysis and trading systems. He worked as a professional futures trader for a trading firm in London and has a passion for building mechanical trading strategies. He has been in the market since 2008 and working with Amibroker since 2011.

Joe Marwood Profile PIc

Recent Posts

  • Investing With Intelligent Machines
  • Trend Following With ETFs and Large Cap Stocks
  • Should You Trade This Earnings Pullback Strategy?
  • Simple Algo Trading With Alpaca And Streak
  • Is Value Investing Dead? Dissecting The Value Investing Narrative
  • How To Screen For Stocks With Norgate Data
  • Do Stocks Have A Floor?
  • What Trading Signals Work Best For Long-Term Investors?

Search

Privacy policy | Contact | Disclaimer | All Rights reserved. JB Marwood.