Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Thu, 02 May 2024 01:12:43 +0000
[Locked] - Replay Backtest Issues
View Count: 1689
| [2020-06-06 18:11:04] |
| User994498 - Posts: 27 |
|
Hi again, So, I have been running bar backtests with my algo this whole time thinking bar backtests were the correct backtest I should be running, totally ignorant to the fact that apparently bar backtests are there to make the algo look much better then it really is. I found out today that replay backtests were the most realistic. So started running replay accurate and Im getting such drastically different results, its kind of ridiculous actually. How can bar backtests be so grossly inaccurate compared to the replay backtets? And it seems that when im running the replay backtests it takes trades at the wrong time. So it will show that it took a long trade at 3101 in ES for example when ES is trading at 3096 it seems; how is this possible? What is the point of bar backtests if they are so grossly inaccurate? I read the support page on it and I get that its more efficient as far as time it takes to run, but if its so grossly inaccurate, it puts users at a huge disadvantage regardless of difference in time. Am I running the replay backtest wrong? Please let me know! Jordan |
| [2020-06-06 19:51:01] |
| User994498 - Posts: 27 |
|
http://www.sierrachart.com/image.php?Image=1591472944630.png See above. It is clear that the trades are being taken at prices that weren't even touched yet. There were no problems with the bar backtest but it seems like the trades it says the algo is taking is impossible. The prices make no sense. |
| [2020-06-06 20:01:34] |
| User994498 - Posts: 27 |
|
I saw someone elses support board post and they seemed to have a very similar issue. It seems like I am a subscribed to SC data services but the data im currently using the CQG Web API. Do I need to be on SC data to run the backtest and then switch back over to CQG Web API when I go live?
|
| [2020-06-07 01:38:42] |
| |
|
First neither of you are using the Denali Exchange Data Feed provided by Sierra Chart. You are using the CQG data feed. To use the Denali Exchange Data Feed, refer to: Denali Exchange Data Feed You will also want to update to the very latest prerelease: Software Download: Fast Update And enable this option to avoid any problems with the bid and ask prices, which are used to fill orders, being at unexpected values : Chart Settings: Always Simulate Bid and Ask Prices During Replay (Chart >> Chart Settings >> Trading menu) Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-06-07 01:39:13
|
| [2020-06-07 02:11:47] |
| User994498 - Posts: 27 |
|
I am following the directions from the index and its not under advanced settings 3. I do not see the box to check off "Always simulate bid and ask prices during Replay". Please clarify where this is! Thank you! |
| [2020-06-07 02:38:10] |
| User994498 - Posts: 27 |
|
Also, Can you guys see which data feeds I paid for... I'm pretty sure I purchased the Denali feed??
|
| [2020-06-07 02:40:33] |
| User994498 - Posts: 27 |
|
I see the [M] on my chart but its delayed. Does this mean I have some version of it?
|
| [2020-06-07 03:05:53] |
| User994498 - Posts: 27 |
|
And as a side note, I very much appreciate how much you guys have added to the software and the amount of advanced features the software has, but it would have been nice to know ahead of time that there was only one data feed that enabled accurate backtesting and that only 1/3 backtesting engines on sierra chart are even remotely accurate. I put in numerous hours backtesting and trying to perfect my algo to take live this coming Monday, and to find out today that the algo has been optimized and tweaked off phony backtest results is disheartening to say the least. With all due respect, what is the point of the two, bar based backtesting engines if both totally fabricate the profit making ability of your algo? Also, if you guys are well aware of the fact that this "simulate bid ask prices during replay" setting must be turned on manually for backtesting to work properly, why isn't a default that its turned on?
|
| [2020-06-07 20:46:19] |
| User994498 - Posts: 27 |
|
Also, it asks for a clearing firm on one of the agreements. What should I put down?
|
| [2020-06-08 01:58:15] |
| |
|
Regarding post #5 you need version 2118. I see the [M] on my chart but its delayed. Does this mean I have some version of it? Yes you must be receiving the Delayed Exchange Data Feed then.but it would have been nice to know ahead of time that there was only one data feed that enabled accurate backtesting
Regarding bid and ask prices during a replay, there is not any known issue with using CQG data. That should not be a problem. Based on one of the questions we were just pointing out that the Denali Exchange Data Feed was not being used. Not that there is a problem with CQG data. The known issues are that sometimes the bid and ask data can lag behind the last trade price but this is an exchange level behavior and is not common. It is resolved with this setting: Chart Settings: Always Simulate Bid and Ask Prices During Replay (Chart >> Chart Settings >> Trading menu) Also this setting is new. Also there were some recent versions, not the current versions, that when market depth data is also being replayed, under some conditions the best bid and ask prices were not getting set properly. This has been resolved. And finally regarding bar based back testing, that is provided for efficiency and it is well documented how it works. You have to make your own judgment as to the usefulness of that based upon your use case. Certainly it is of value. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-06-08 01:59:14
|
| [2020-06-08 02:42:21] |
| User994498 - Posts: 27 |
|
Hi, Thanks for making clear what version I need and for making clear that Denali is not required for backtesting. I thought when you update the software, it automatically goes to the very latest version. With all due respect, how can it be of value if it totally overexaggerates results and you guys even have steps on how to make a better looking backtest, using bar backtest being one of the steps, if I remember correctly. If its not as precise as the replay and wholly inaccurate, the fact that its faster seems a moot point to me. The point of a backtest is to have a good idea of how your automated system will perform live, not how quickly you can have or show results. I am pretty sure I am not speaking for just myself here but I rather have a backtesting system that is as precise as reasonably expected and within parameters of engineering abilites that takes 2 days to complete, rather then one that takes 10 minutes to complete, is far less precise and has results that can't be expected by a long shot. You guys say it is less accurate (this may be an unanswerable question)- but, on a scale of 1-10, 10 best most precise, 1 be totally and completely inaccurate- where would you rank the replay backtest and where would you rank the bar backtest? Maybe its the specific algo im using but this is why I am asking questions. |
| [2020-07-29 19:00:34] |
| Marion2025 - Posts: 13 |
|
I'm having the same problem. Have you checked that your algorithm references any cells at the current timestep? This will cause a very large divergence in the backtests as the reply backtest will pull tick data but the bar backtest will estimate a value based on the current bar. However, even if you only reference the prior bar, you will still likely see a large divergence. This is the issue I am facing and cannot explain it. My bar backtest still shows my strategy performs 10x better than in a replay back test. I have even enabled "always simulate bid/ask." I cannot explain the discrepancy as it makes no sense, nor can I find a strategy that runs consistently on both backtest methods. In my strategy, the trade should only execute at the absolute beginning of the next bar, referencing only values from the previous bar. These two methods should produce similar results under those conditions This an extremely frustrating result. Some divergence would be tolerated, as it would allow users to iterate on the quicker method and validate on the slower method. But when the bar based backtest is wholly inaccurate even under these restrictions it renders it useless as a feature, or worse, a time waster. The runtime on the reply test makes it near impossible to optimize strategies in a time efficient matter. I don't see a solution in sight beyond building my own backtesting framework |
| [2020-07-29 19:14:00] |
| User994498 - Posts: 27 |
|
Marion 2025- I tried explaining the same thing to them.
|
| [2020-07-30 19:48:11] |
| |
|
nor can I find a strategy that runs consistently on both backtest methods. There should be consistency with each back test. If there are inconsistencies it would seem to be something about the automated trading system itself causing this. Let us consider bar based back testing. So you get a different result in regards to the number of order fills or the fill prices every time you do a bar based back test of your automated trading system?
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-07-30 19:48:59
|
| [2020-07-30 20:21:07] |
| User994498 - Posts: 27 |
|
There should be consistency with each back test. How can there be consistency when the process by which an order is filled differs? If I remember correctly, for a bar backtest, there is only 4 prices your order can be filled at (open, high, low, close) and the engine somehow picks the best one. With replay, it fills at the price the signal occurs if i remember correctly? And what does consistency matter if the backtest is wholly inaccurate and thus seemingly useless as Marion alluded to. And the concept of it filling at one of four prices, is kind of unrealistic because it accounts for no slippage. Can you guys please explain the realistic use of bar backtest? What was the point of its creation? |
| [2020-08-14 11:04:08] |
| |
|
and the engine somehow picks the best one. No. Each of the four prices are always used per bar.And your questions are already thoroughly documented on the Back Testing and Trade Simulation Mode pages. Read through those pages. This is why the documentation exists. It is there for you to read. You should not be asking these questions here when this is already documented. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-08-14 11:45:23
|
To post a message in this thread, you need to log in with your Sierra Chart account: