Auto Trade System Back Testing


Introduction

When you are using the Spreadsheet System for Trading study or you have created an ACSIL (Advanced Custom Study and Interface Language) trading system study that uses the ACSIL Trading functions, then you can perform Back Testing by using Trade Simulation Mode and replaying the chart, or by performing a Bar Based Back Test.

In order to evaluate how your trading system has performed historically and to see the historical trades for past data, it is necessary to run a Back Test.

A Bar Based Back Test uses the Open, High, Low, Close data of the loaded chart bars themselves to evaluate the automated trading system. It is this data which is incrementally added to the chart during the Back Test. This method of Back Testing is faster compared to a Replay Back Test, however it is less precise than a Replay Back Test.

A Replay Back Test uses the underlying data the Intraday data file to evaluate the automated trading system. This underlying data has a timeframe per record anywhere from 1 Tick to 1 Minute (depends upon various factors, including Data/Trading service used, historical data available and Sierra Chart settings). It is this more detailed underlying data in the Intraday data file which is incrementally added to the chart during the Back Test. This is a slower method of back testing compared to a Bar Based Back Test, however it is more precise.

With either method of Back Testing, a complete Trade Statistics report can be viewed based upon all of the order fills generated during the Back Test. Refer to Viewing Back Test Results.

Bar Based Back Testing

Supported for Historical and Intraday charts.

Back testing on loaded bars is a back testing method using the Open, High, Low, and Close values of the loaded bars. The underlying data in the data file is not used directly, only the loaded bars themselves. This usually does not perform as high accuracy of a Back Test compared to a Replay Back Test. However, it is usually faster and the recommended method if fast back testing is important.

  1. Make sure there is a check mark by Trade >> Auto Trading Enabled if there is not one already. Otherwise, the automated trading system will not generate any trades.
  2. Disconnect from the data and trade server if not already, by selecting File >> Disconnect. It is not possible to perform a Bar Based back test while connected to the data or trade server.
  3. To perform a high-speed Back Test based on loaded bars, select Trade >> Auto Trade System Bar Based Back Test.
  4. You will be prompted for the Bar Processing Increment . The default is 250. This specifies the number of bars that are processed at once before user interface input is processed. Which means if you specify 250, then 250 bars be processed at once and then another 250. Between each of these blocks of processing, you are able to interrupt the Back Test by pressing the Escape key on the keyboard. The larger the number that is specified, the longer the time that the user interface will remain busy.
  5. To view detailed results of the Back Test, refer to Viewing Back Test Results.

Bar Based Back Testing Notes

The studies are calculated on the chart 4 times for each bar. For each bar in the chart, the studies are calculated first with the Open price, then the High and Low of the bar. Either the High is used first or the Low is used first, depending upon the determined direction of price movement.

The Low is used first when the Close of the bar is closer to the High of the bar. The High is used first when the Close of the bar is closer to the Low of the bar, or the Close is an equal distance to the High and Low. Finally, the Close price of the bar is processed.

The fill prices of orders are going to be based upon the Open, High, Low, and Close bar values or within 1 tick of them, unless the orders are resting orders (nonmarket orders that did not immediately fill). For more information, refer to How Orders Are Filled. Keep this in mind when looking at the fill prices. The timestamp of an order fill will be the starting time of the bar that the fill occurred at.

There is nothing special you need to do with development of an ACSIL trading system or a Spreadsheet Trading system when performing bar based back testing.

If you want to make a trade at the close of the bar when your rules are met, you will know that your system is being evaluated against the closing price when you see that the sc.LastTradePrice, in the case of an ACSIL trading system, is equal to the bar close value or within 1 tick of it.

Bar based back testing is a fast method of back testing unless you need high precision tick by tick replay based back testing.

Replay Back Testing - Automatic

Supported for Intraday charts only.

This section documents how to perform a Back Test by using the Trade >> Auto Trade System Replay Back Test command. This simplifies the process of performing a Replay Based Back Test.

  1. Disconnect from the data feed by selecting File >> Disconnect on the menu.
  2. Select Chart >> Chart Settings and set the Days to Load to the number of days you want to perform your Back Test over. Press OK. Once this is set, it does not have to be changed again.
  3. Make sure there is a check mark by Trade >> Auto Trading Enabled if there is not one already. Otherwise, the automated trading system will not generate any trades.
  4. Select Trade >> Auto Trade System Replay Back Test on the menu. This command clears all simulated trade data for the chart's symbol and replays the entire chart from the beginning at a high speed.
  5. Sierra Chart will be busy during the Back Test. You can stop the Back Test though by pressing the Stop button on the Chart Replay window. You will have an opportunity to press this button every few seconds.
  6. To view detailed results of the Back Test, refer to Viewing Back Test Results.

Replay Back Testing - Manual

Supported for Intraday charts only.

Manual back testing is used when you want to run the Back Test at a slower speed to observe what it is doing and it may be necessary to use this type of Back Test when you are back testing a trading system that involves multiple charts.

  1. Make sure there is a check mark by Trade >> Trade Simulation Mode On if there is not one already. However, you will be placed into this mode anyway when you start a Back Test.
  2. Make sure there is a check mark by Trade >> Auto Trading Enabled if there is not one already. Otherwise, the automated trading system will not generate any trades.
  3. Replay the chart from the point where you want to begin your backtest. To do this, select Chart >> Replay Chart to display the replay window. On the Replay Window you must select Accurate Trading System Back Test Mode. Scroll to the point in the chart where you want to begin the replay and press the Play button on the Replay Window.
  4. When the replay is started, the current simulated position for the symbol, if any, the simulated orders for the symbol, if any,and all of the simulated order fills for the symbol are all cleared.
  5. To view detailed results of the Back Test, refer to Viewing Back Test Results.

Performing Back Testing on a Trading System That Uses Multiple Charts

This information has been updated and applies to version 892 and higher.

When you have a trading system that involves multiple charts, the question is whether you need to replay or Back Test only the chart the trading system study is on, or all of the charts that the trading system makes reference to.

You will want to replay all charts that a trading system is referencing by using the Replay Back Testing - Manual method if you want to have your trading system use the price and study values from bars that are in the process of being formed rather than using study and price values in the source charts which have complete and finalized values.

For example, consider a 5 minute per bar chart. In the source chart that the trading system is referring to, if it is not replaying at the same time it will have complete 5 minute bars. Whereas if you are replaying it, gradually over the five-minute timeframe, a 5 minute bar will build and the study values will change. If it is important to have changing bar values and study values for your trading system, and only you can answer this question, then you do want to replay all charts the trading system uses.

You will want to not replay all the charts that a trading system is referencing and instead only replay or perform a back test on the particular chart that the trading system is on, if it is acceptable for you to have complete and finalized bar and study values in the source chart or charts.

When a trading system is making references to other charts, it needs to use one of two methods to do so. It will need to use the Study/Price Overlay study. Or in the case of an ACSIL trading system it can make references to other charts by using the method documented on the Referencing Other Time Frames and Symbols When Using the ACSIL page.

When using the Study/Price Overlay to overlay price or study data onto the chart that the trading system is on, and the chart bars are based on Volume, Number of Trades, Range, Renko, Reversal, Delta Volume, or Price Changes, then you need to set the Data Copy Mode study Input to Use Earliest Value from Corresponding Timeframe.

Otherwise, in the case of when the other chart being referenced is not replaying and is not synchronized timewise to the destination chart, data from the last bar in the chart being referenced will be used for the current bar being back tested. So the referenced data would not be correct.

When you are replaying multiple charts at the same time to perform a back test, then you need to use the Replay Back Testing - Manual method. On the Replay window you need to enable the For All Charts in Chartbook option. And on the Replay window, select Accurate Trading System Back Test Mode or Calculate at Every Tick/Trade. When you do this, a synchronized Back Test will be performed. For this type of Back Test, it is recommended that you disconnect from the data and trade servers by selecting File >> Disconnect.

With a synchronized Back Test, the dependency between the charts is determined and the calculations are done in the proper order. This ensures consistency and accuracy. You can use any Replay Speed.

When you start the synchronized Back Test replay you will be asked for the time step increment to perform the back test in (Enter Processing Step in Seconds). The default is 60 seconds. The smaller the timeframe, the more accurate the Back Test, however it will take longer to complete. The larger the timeframe, the faster the Back Test will be but it will be less accurate. However, the accuracy really depends upon the timeframe of the source and destination chart bars. You would not want to use a time step increment that is longer than the time frame of the bars being involved in the trading system. You may want to choose a timeframe which is about 25% of the average bar time length.

Viewing Back Test Results

You are able to view detailed results of your trading system by selecting Trade >> Trade Activity Log >> Trade Statistics from the menu.

At the top of the Trade Activity Log select the symbol that the back test was run on. You need to choose the symbol that has [SIM] in front of it.

At the top of the Trade Activity log you must select Simulated from the list of the Order Activity Sources, to display the order and fill activity for the Back Test.

For complete instructions, refer to Trade Statistics Tab. There are many fields displayed which provide detailed results of the trading.

For a Trade by Trade log, select Trade >> Trade Activity Log >> Trades from the menu. For complete instructions, refer to Trades Tab.

To view trade order fills visually on the chart, enable Trade >> Show Order Fills on the menu of the main window or the chart window.

To know exactly when an order was filled, you need to look at the order fills on the chart rather than looking at the arrows placed by a trading system on the chart because those arrows may not necessarily represent actual orders accepted and filled. For additional information about this, refer to Ignored Signals With Spreadsheet Systems or Alerts (This applies to the Spreadsheet System for Trading study).

Improving Back Test Performance

There are some things that can be done to improve back test performance.

The fastest back testing is going to be when using the Bar Based Back Testing back testing method.

In the case of when using the Spreadsheet System for Trading, refer to Improving Back Test Performance of Spreadsheet Trading Systems in addition to the items below.

Developing an automated trading system using the Advanced Custom Study Interface and Language is going to always give the very best performance compared to using Spreadsheets.

When running a replay type of back test, if the data in the Intraday chart data file contains 1 Tick or 1 Second data records, back testing will be slower compared to higher timeframe data records in the Intraday data file. Therefore, increase the Global Settings >> Data/Trade Service Settings >> Intraday Data Storage Time Unit. After this is done, re-download the Intraday chart data by going to the Intraday chart and select Edit >> Delete All Data and Download. This only needs to be done once per symbol. This will help reduce the time for back testing.

The next thing to check to determine the source of slow back testing is to remove all of the studies from the chart or charts that are being replayed, or from the chart a bar based back test is being performed on. Run the back test again and see if you get better performance. If so, then it is one of those particular studies causing the problem. You can then add studies back gradually one by one to see which one is taking a longer time to calculate.

If a Custom Study is causing a performance problem, then the developer of that study needs to improve the performance of that study. The custom study must set sc.FreeDLL = 0 in the sc.SetDefaults code block to improve performance.

Improving Back Test Performance of Spreadsheet Trading Systems

In the case of when using the Spreadsheet System for Trading study, Back Testing is usually not very fast. The reason for this is because all of the chart data is outputted to a separate Spreadsheet object.

Every time there is a new bar added to the chart during a Back Test, all of the chart data has to be re-outputted to the spreadsheet. This causes a lot of calculations to occur. Inherently using ACSIL is much faster.

However, you can improve performance by following the steps below.

  1. Look at your formulas in cells K-Z (last formula column when using 16 formula columns). Determine how many rows from the current row 3, they refer back to. For example, if the formulas refer down no further than row 12, then the Spreadsheet Study only needs to output 10 rows of data to the Spreadsheet.
  2. Reduce the Number of Rows input in the Study Settings window for the Spreadsheet System for Trading study to the minimum number of rows required.
  3. Clear the data from the Sheet rows that are now no longer used on the Spreadsheet. To do this, highlight those rows by selecting the row numbers on the left side of the Sheet of the Spreadsheet and press the Delete key on the keyboard or select Spreadsheet >> Clear Selection.
  4. Remove any unused Sheets within the Spreadsheet. At the top left of the Spreadsheet window, select the particular Sheet# that is not used, and select Spreadsheet >> Delete Sheet.
  5. If there is also an Advanced Custom Study on the chart which you have developed yourself, make sure it has sc.FreeDLL set to 0. This is essential for performance.
  6. Perform the Back Test using one of the methods described on this page. The fastest back test will be a Bar Based Back Test.

Differences Between Back Testing and Real-Time Auto Trade System Evaluation

The BuyEntry, BuyExit, SellEntry, SellExit Order Actions are evaluated when the trading study is calculated. During normal real-time chart updating this calculation happens at the Chart Update Interval set in Global Settings >> General Settings. However, when the Chart Update Interval elapses, the trading studies will only be calculated when there is data being received from the data feed, or there is new/updated trade orders or position data. During a Back Test, there are definite rules that control when trading studies are calculated.

For a Bar Based Back Test , the trading studies are calculated for each of the Open, High, Low, Close values in the bar.

During a Replay Back Test, the trading studies are calculated when there is a new bar added to the chart and any time there is a change with the High, Low, Close values of the last bar in the chart as the data is processed through the Intraday chart data file.

During real-time updating, between the times the trading studies are calculated which is at the Chart Update Interval, there potentially may be more than one data record added to the chart. This is especially true with tick by tick data. Therefore, the study is not calculated at every single tick. If you want your Order Actions evaluated as fast as possible during real-time chart updating, then you may want to decrease the Chart Update Interval. We do not recommend going lower than 50 to 100 ms.

The point of this discussion is that there can be some inconsistencies between a trading system performing in real-time and during a Back Test due to the conditions and frequency for when the studies including the trading study are calculated.

You will have the greatest consistency between back testing and real-time auto trade system evaluation when you are performing a Replay Based Back Test on tick by tick data. To ensure you have tick by tick data in the Intraday data files, refer to Tick by Tick Data Configuration.

In the case of the Spreadsheet System for Trading study, if you want to evaluate your condition formulas only on a bar close, then set the corresponding Signal Only on a Bar Close inputs with the Spreadsheet System for Trading study to Yes. This will give your system greater stability and not be affected by the Chart Update Interval. In the case of an ACSIL trading system, you will want to use sc.GetBarHasClosedStatus().

In the case of a trading system which use multiple charts, during real-time updating of charts, to control the calculation order of the charts based upon the dependency between them you need to enable Global Settings >> General Settings >> Use Controlled Order Chart Updating. This provides higher consistency between back testing and real-time auto trade system evaluation for trading systems that use multiple charts.

Back Testing and Continuous Futures Contract Charts

It is supported to perform Back Testing when using one of the Chart >> Chart Settings >> Advanced Settings >> Continuous Contract options.

When performing back testing across a Continuous Futures contract chart, although the expired futures contracts are loaded into the chart, the chart only has one symbol which is used for trading even though you are Back Testing across historical data. So the trades are occurring on expired contract prices, but being specified with the current symbol. Therefore, you will only see one symbol when reviewing the trading results in the Trade Activity Log.

Consistency Between Back Tests

When performing Back Testing, you must use one of the methods documented on this page and follow the procedures as given.

When performing a Replay Based Back Test, on the Replay Window you must select Accurate Trading System Back Test Mode in order to have consistency when Back Testing the same automated trading system under the same conditions.

During a Back Test, when Sierra Chart is processing through the underlying data records for a Replay Based Back Test or through the bar data for a Bar Based Back Test, the Bid and Ask prices are set from the underlying data records in the Intraday data file or are derived from the Open High Low Close bar data. These Bid and Ask prices are used to fill the orders which are submitted by an automated trading system.

There is always a 100% consistency with the setting of the Bid and Ask prices between Back Tests on the same chart data as the data is processed.

If your trading system gives inconsistent results between Back Tests without making any changes to the chart data, the Chart Settings or to Study Settings, you must look at your own trading system code/formulas for a cause.

When there are inconsistencies, this can only be attributed to your own code. It is very unlikely to be the result of how Sierra Chart is processing the back test. Inconsistencies reveals that your own trading system produces inconsistent results based on the way that it is designed and working.

Even if you do not get a consistent result between Back Tests, generally this does not necessarily mean very much because although you may want to see a consistent result every time you run a Back Test, assuming that your study is working consistently and Sierra Chart does not have any control over that, it does not change the fact that with live trading you will get yet another different result from your trading system.

Try one of the Sierra Chart provided trading systems. Run it many times. You will get a consistent result every time. If you do get a consistent result every time, and with your own trading system you do not, then this proves most likely the inconsistency problem is within your own trading system.

The Sierra Chart example trading systems can be found in Analysis >> Studies >> Add Custom Study >> Sierra Chart Custom Studies and Examples >> Trading Example: *.

If you are Back Testing a trading system that involves replaying multiple charts at the same time, then this is something which involves a lot more complexity. This is something to consider if you have inconsistencies between Back Tests.

Using Actual Bid and Ask Prices During Back Test

Sierra Chart supports using the actual Bid and Ask prices during a Replay Based Back Test. These Bid and Ask prices are used to fill orders. This provides a very high degree of accuracy when Back Testing.

Historical Bid and Ask prices are available when using historical data provided by the Sierra Chart Historical Data services. In most cases this service is being used for historical data for charts. This historical Bid and Ask price data begins on June 23, 2014.

Follow these steps to use actual Bid and Ask prices during back testing.

  1. Set Sierra Chart to store data tick by tick. Refer to Tick by Tick Data Configuration. This is an essential step.
  2. Perform an automatic or manual Replay Back Test. It must be a Replay Back Test.

*Last modified Wednesday, 14th September, 2016.