Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Wed, 24 Apr 2024 22:03:03 +0000
Volume By Price causes Replay to be so slow it is unusable
View Count: 1004
| [2021-04-26 10:33:48] |
| Ernie Doo - Posts: 46 |
|
Dear SC team, I will do my best effort to explain as clearly as possible. The bottom line is, when I load the VbP study with a Volume Graph Period Type other than Visible Variables and I let the Replay run, it gets very slow eventually (usually after replaying 3 months). I noticed that if I stop the replay, modify the "From" field in Chart Settings, "Use Date Range" and test the next 3 months the speed goes back to normal slowing down again, after +/- 3 months. Setup: - I have a 1 minute Candle stick chart for S&P (ES) - In advanced settings of this chart I have Continuous Contract: Continuous Futures Contract- Date Rule Rollover, Back Adjusted - I ONLY load 1 study on this chart: VbP (Volume by Profile) - My chart window shows 120 bars (2 hours of market data) if this is at all relevant - I have Intraday Data Storage Time Unit set to 1 Second - The "Use Date Range" from the Chart Settings is set to: from 2020-01-02 to 2021-03-31 - Replay settings - Charts to Replay = Single Chart. Replay Mode: Accurate Trading Systems Back Test Mode. Speed: 30720 - I start replaying from 2020-01-02 until 2021-03-31 1st Test (Volume Graph Period Type = Visible Bars) Here there is no problem. Replay is super fast. Sierra Chart will replay the whole period (the full 15 months) in 8 minutes. 2nd Test If I change the Volume Graph Period Type = Session Start to End the replay becomes extremely slow to the point of being unusable. Just to give you an idea: after 8 minutes the replay reached barely March 2, 2020 (2 months in 8 minutes) after 26 minutes April 1, 2020 (3 months in 26 minutes) after 1hr 10min August 3, 2020 (7 months in 1hr 10 mins; I stopped the replay after this test) IMPORTANT TO MENTION: I noticed that in January, at the beginning of the test, the speed was fast, just like the 1st test; the speed dropped after a couple of weeks 3rd Test I started the replay on April 1st, 2020; the test was still slow: after 10 minutes May 1, 2020 (1 month in 10 minutes) after 20 minutes June 1, 2020 (2 months in 20 minutes) after 31 minutes July 1, 2020 (3 months in 30 minutes) 4th Test (HERE is when it starts getting interesting) In Chart Settings, "Use Date Range" I changed the "From" field from 2020-01-02 to 2020-04-01 and replayed starting at April 1, 2020. I saw immediately how the test's speed increased considerably compared to the 3rd test with these results: After 2 minutes May 1, 2020 (1 month in 2 minutes instead of 10 mins in the previous test) After 5 minutes June 1, 2020 (2 months in 5 minutes instead of 20 mins) After 11 minutes July 1, 2020 (3 months in 11 minutes instead of 31 minutes) ADDITIONAL NOTE: I also tried Volume Graph Period Types: "From Session Start Time To End", "One Period from Start Time To End Time" and "From Start Time to End" with the same slow effect. I also noted that the CPU load and Memory load was similar in all tests: CPU an average of 12% and Memory between 750-805 MB. I also deleted all data and downloaded it again. This helped improve the speed a little but not completely. Since I cannot test continuously at an acceptable speed, my solution has been to test every 3 months independently and consecutively and adding my results at the end of all tests. I mean, I test from January 2020 to March 2020. Then I modify in Chart Settings, "Use Date Range" the "From" field from 2020-01-02 to 2020-04-01 and run the next 3 months, from April to June, and so on and so forth. A very inefficient and impractical solution but it will have to work until I receive a response from you guys. What I don't understand is: 1) Why is it that the longer a Replay runs the slower it gets? I have limited the profile period to maximum 1 day so I see no reason for the Replay to slow down. 2) Why, if I move in Chart Settings, "Use Date Range" the "From" field to a later date and start the replay at this date the speed is again very fast but then drops again? 3) What can I do to be able to test continuously the whole period at an acceptable speed? |
| [2021-04-26 21:41:35] |
| |
|
Well in general, maybe as there is more memory being used by the chart, the speed of accessing the chart data becomes slower. Regarding this: "2nd Test", is there just one By Profile in the entire chart? Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2021-04-26 21:41:45
|
| [2021-04-27 06:54:49] |
| Ernie Doo - Posts: 46 |
|
Well in general, maybe as there is more memory being used by the chart, the speed of accessing the chart data becomes slower.
If this was the case why is the 1st test so fast? It accesses 15 months worth of data in a 1 Minute Chart in 8 minutes.Regarding this: "2nd Test", is there just one By Profile in the entire chart?
No. There is one profile per day. And according to SC Study instructions this is the way it should work: https://www.sierrachart.com/index.php?page=doc/StudiesReference.php&ID=141&Name=Volume_by_Price#SettingsBesides, I also tested other Volume Graph Period Types which only use 1 Profile per day: I also tried Volume Graph Period Types: "From Session Start Time To End", "One Period from Start Time To End Time" and "From Start Time to End" with the same slow effect.
If you could just test it you would find out that the replay gets slow over time as if either the study is not releasing memory (which I doubt it) or it is accessing some older dates (older than the current day) to calculate, which shouldn't even be necessary. But like I said, it makes no sense to me that if I stop the replay when it gets slow and just restart it at the same date where I stopped it, the speed is fast again. And even it gets much faster if I modify in Chart Settings, "Use Date Range" the "From" field to the same date I will start testing. Please, I really need your help. It has been very time consuming to test this way. Thanks Date Time Of Last Edit: 2021-04-27 13:46:47
|
| [2021-04-28 01:03:40] |
| |
|
We are still looking at this. The one area of concern with efficiency, did not identify any problem. But we need to look more closely at that. However, this confirms, there is not a performance issue related to searching for a specific date-time: If this was the case why is the 1st test so fast? It accesses 15 months worth of data in a 1 Minute Chart in 8 minutes.
If this was the case why is the 1st test so fast? I
Probably you are dealing with an operating system and hardware performance issue.Would be very unlikely to be a performance issue within Sierra Chart itself. Would not even make sense. So you are asking a question we cannot answer and we do not know the answer to. Sierra Chart is going to work consistently. So I think it is safe to say you are dealing with operating system and hardware performance issues which are out of our control. However, we need more time to look into this. What you have said here, we have heard more than once. There really would not be anything within Sierra Chart which would progressively create a slower performance issue during a chart replay. Other than more memory being allocated in the chart as it gets larger but if you are already loading a lot of data in the chart and you notice that the performance is good as compared to running a replay for a while, which at that moment is loading the equivalent amount of data, then it's not the amount of data. If your hardware is very fast and you have a lot of memory you really should not notice an issue. In the end, we think it is very unlikely with what we know, at this point, that we can do anything to help. All evidence at this point, points to the operating system and hardware. That really is the only logical conclusion. However, we do not want to give you the impression that spending a lot of money on new hardware is going to help because we really do not know. But it would be interesting to find out. Maybe we could test on one of our high-performance servers that have about 400 gigabytes of memory. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2021-04-28 01:06:23
|
| [2021-04-28 01:10:19] |
| |
|
Really thinking through this some more, we do really think it is an operating system and hardware level issue. Really makes no logical technical sense for it to be a problem within Sierra Chart. Perhaps there is some issue with the message queue, perhaps there is something with the location of memory allocations and caching. Sierra Chart during a replay is going to work consistently. That is 100% certain. Another thought is that as the chart is replaying Sierra Chart continuously is reading from the same file without reopening it and the operating system may be buffering data and having degraded file read performance. But this would be strictly a Windows issue. If we have to reopen the file to improve performance, that would be something we should not have to do. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2021-04-28 02:04:14
|
| [2021-04-28 13:56:16] |
| Ernie Doo - Posts: 46 |
|
I completely agree. Really thinking through this some more, we do really think it is an operating system and hardware level issue.
Really makes no logical technical sense for it to be a problem within Sierra Chart. Perhaps there is some issue with the message queue, perhaps there is something with the location of memory allocations and caching. I even think the problem might be more in the line of what you said next: Another thought is that as the chart is replaying Sierra Chart continuously is reading from the same file without reopening it and the operating system may be buffering data and having degraded file read performance. But this would be strictly a Windows issue.
However, wouldn't it be a good idea to look a little into this problem? Could a solution be to open only two files for the replay, close them, and then open the next two? In other words, open ESH20-CME and ESM20-CME, then close them and open the next two ESU20-CME and ESZ20-CME, and so on and so forth? I am just giving some ideas here. By no means do I pretend to be an expert in SC, but I have read some people have this problem too and if something like this could solve it then it wouldn't probably matter that much which operating system or hardware is everyone running (unless they are running a very old operating system in a very old hardware of course). Just some food for thought. Sierra Chart is the best platform I found in the market based on its performance, speed, reliability and programmability. It is actually my bread and butter. I would love for SC to continue growing and thriving as it has until now, or even more. Date Time Of Last Edit: 2021-04-28 14:23:55
|
| [2021-05-05 19:37:02] |
| Sierra_Chart Engineering - Posts: 14076 |
|
For a Continuous Futures Contract chart, there is only one chart data file open at a time. When the next contract month begins the prior contract month file is closed. We have to think about the problem more. We have marked this as a follow-up. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2021-05-05 19:37:42
|
| [2022-02-09 21:07:05] |
| |
|
Another thought is that as the chart is replaying Sierra Chart continuously is reading from the same file without reopening it and the operating system may be buffering data and having degraded file read performance. But this would be strictly a Windows issue. If we have to reopen the file to improve performance, that would be something we should not have to do. We have implemented a solution to this. During a replay, when 100 MB are read from an Intraday data file, it is closed and reopened.This is in the latest Sierra Chart release. See if that helps. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: