Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Thu, 25 Apr 2024 13:00:52 +0000
Back Test Speed
View Count: 1926
| [2018-02-24 01:21:05] |
| joer - Posts: 41 |
|
I haven't used Sierra for about a year. Back then I converted my spreadsheets to the new format since it was clear that was the direction Sierra was going in, and began experiencing unacceptably slow back testing speeds. Thought I'd come back and see if this issue has been resolved but I am still experiencing this problem. I use spreadsheet system for trading, have done all the things in your documentation to optimize the back testing but still very, very slow. I realize from request I made in the past about this problem that you aren't big on fixing it, but a couple of things worth noting: I notice Sierra is not using the computer resources that are available. During a back test, I show less than 30% cpu utilization and about 40% of memory being used. Disc utilization is minimal (and I have a solid state disc anyway), so just wondering if somehow you couldn't take better advantage of the resources available. What is the bottleneck causing this back testing to be so slow? As it stands, it takes more than a day to backtest 1 month worth of data. That's ridiculous, and a shame because everything else about your software is so robust and efficient. It wasn't like this with the old spreadsheets, I used to backtest a month and it took a couple of hours not a couple of days. Won't you please help with this? |
| [2018-02-24 01:41:45] |
| |
|
I realize from request I made in the past about this problem that you aren't big on fixing it This is actually not true. This is actually one reason, why we have de-prioritized some support, and ensuring that users do not waste our time on the support board, in order so we can focus more on development.Unfortunately, the spreadsheet optimization task just has not yet been completed. But we are actually actively working on it these last few days. It is a priority for us to complete, but other things have come up. We wanted to involve another programmer as well on this task. Allow us about another 30 days. What you can do for now is use the 64-bit version of Sierra Chart. Update to prerelease 1706: Software Download: Fast Update This will at least give some better performance. If you also attach your Chartbook and Spreadsheet, we can test it as part of the performance testing. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2018-02-24 01:43:38
|
| [2018-02-24 01:46:11] |
| |
|
As it stands, it takes more than a day to backtest 1 month worth of data. If it is taking this long, there must be something unusual about the spreadsheet that involves a lot of calculations and you must not be actually following the advice we gave to improve performance. There is no reason it should take this ridiculously long if you follow these instructions here:Auto Trade System Back Testing: Improving Back Test Performance of Spreadsheet Trading Systems You need to perform a bar based back test. We would normally expect a back test of a month of data to be done in a matter of minutes. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2018-02-24 01:46:33
|
| [2018-03-26 05:26:14] |
| joer - Posts: 41 |
|
I decided the spreadsheet I was working with was too large and complicated to try and trouble shoot this replay problem with. So I created a new one leaving out most of the formulas, leaving me with a very basic, small spreadsheet. I upgraded to the latest version (1725)and started using the bar based back test as you recommended. A month of data now takes only a minute or two so I am able to test a year worth of data in a reasonable amount of time. I am quite pleased with this and was producing the type of results profit wise I was looking for. FYI I am using your Sierra data feed and 1 tick interval for all this testing. As a final check before actually trading with this system, I ran a system replay back test. This takes about 15 minutes for a month of data. Not too bad. But here is the problem. The back test results for the bar based back test for 3/1/2018 thru 3/23/2018 showed a profit of $4,279. But when I ran the replay back test the results were a loss of -$3,164. I ran both types several times and get the exact same results each time. I also ran for other months and saws similar results. My question is how can there be such a huge and totally opposite result from these two methods? And sadly, it appears the method you said I need to use, the bar based back test, is the one that is wrong. This has all been very frustrating. Date Time Of Last Edit: 2018-03-28 19:02:26
|
| [2018-03-28 19:02:12] |
| |
|
My question is how can there be such a huge and totally opposite result from these two methods? This is impossible for us to answer from our perspective since we do not know your trading system and we do not have the time to analyze your trading system and this would not be productive either.If you think that the bar based back test is not giving an accurate result, then just use the replay back test. There really is not any further help we can provide. How each of those work is fully documented here: Auto Trade System Back Testing The concept that there is something wrong and that we need to solve it is not correct. There are different methods of back testing and only you can answer the question as to why you are getting a different result. Both methods are very well documented. Ultimately this is for you to analyze and make a determination as to what method is most logical. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2018-03-28 19:04:07
|
| [2018-03-28 19:16:01] |
| joer - Posts: 41 |
|
Based on the fact that the replay back test results are fairly accurate and the bar based back test results are totally bogus, I can logically determine the replay back test is the one to use. Unfortunately, it takes 10 hours or more to back test a year which is the same issue I reported a couple of months ago. Are you saying this is acceptable performance from your point of view?
|
| [2018-04-16 13:01:45] |
| |
|
If you were to use ACSIL instead of Spreadsheets you will improve performance. Refer to: Automated Trading From an Advanced Custom Study And we did say we are working on improving Spreadsheet performance, to the extent we can. Two of our programmers are working on this full-time right now. To what extent we can do this, still remains to be seen. The point we were trying to make, there is not something wrong with bar based back testing in regards to how it works according to the documentation. Now whether it is accurate or not, you have to make that judgment. But what you can conclude is that obviously if you want accurate results it is going to be time-consuming. There is not any other way to do this. If you still get poor performance even with ACSIL, then you just have to realize that accurate results is time-consuming. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2018-04-16 13:01:55
|
| [2018-04-17 18:12:49] |
| joer - Posts: 41 |
|
I don't see how you can say there is nothing wrong with bar based back testing when it gives results that are so completely different from the accurate results given from replay back testing. Be that as it may, I understand I will have to use the replay back test if I want results with some degree of reliability. I also understand it will be time consuming, but I have a problem with it taking 2 or 3 times longer with your new spreadsheets compared to the old. You basically forced conversion to the new spreadsheets; you shouldn't have done this until performance issues were better or at least comparable to the old spreadsheets. As someone with 20 years of IT systems design and development experience, I'd like to point out that during back testing the total cpu utilization rarely goes above 30% and memory utilization stays below 50%. I have a solid state hard drive and plenty of memory, so if I were trying to address the back testing performance issues I would start by fixing whatever is preventing your software from using the resources that are available. |
| [2018-04-17 18:56:37] |
| |
|
I don't see how you can say there is nothing wrong with bar based back testing when it gives results that are so completely different from the accurate results given from replay back testing. There definitively is nothing wrong with it as it is designed. What it only demonstrates is that skipping prices, does result in less accuracy. Based on how the trading system is designed, some trading systems are go to be affected more than others.So this does affirm that it is time-consuming to get an accurate test. you shouldn't have done this until performance issues were better or at least comparable to the old spreadsheets. In many cases the performance is comparable and even better. And in any case we are only doing our best.so if I were trying to address the back testing performance issues I would start by fixing whatever is preventing your software from using the resources that are available. This is a technical impossibility. Back testing must be done in sequence, and on a single thread of execution. It is not possible to do this differently for general back testing functionality.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2018-04-17 18:58:30
|
To post a message in this thread, you need to log in with your Sierra Chart account: