Support Board
Date/Time: Thu, 17 Jul 2025 10:32:13 +0000
Backtesting data - loading times of charts
View Count: 61
| [2025-07-15 12:00:20] |
| Cell2985 - Posts: 3 |
|
Dear Engineering legends, I have the following issue; Due to having an old computer and being poor, I thought I was cleaver to start a separate VPS specifically for back testing. But for some reason, and I cannot figure out why, loading times for NQ charts (specifically two years of tick data) loads very slow. We are talking sub 60mb/s where bursts of 8gb/s are available and tested with a disk checker. I can work around this by using second based chart updates and limiting the size of the data files instead of tick data. But I would rather use tick data. Is there a setting that I overlooked? Should I use a dedicated server? I have 4 vCPU's, 16GB of DDR5 RAM and a Gen4 NVMe SSD running up to 5000mb/s available (which has 8000mb/s bursts). Is that enough? How am I being bottle necked and how can I fix this? Thank you in advance for your efforts! Kind regards, K. |
| [2025-07-15 15:31:52] |
| John - SC Support - Posts: 40942 |
|
This is something that we really can not comment on, as there are too many variables and we are not there. Your setup does sound fine, but 2 years of 1 tick data is going to take a bit to load, no matter what setup you are using. It's simply a lot of data. And keep in mind that there are two different items that you need to deal with - one is the downloading of the data from our servers, which should only occur once and then update with just any missing data; the other is the loading of the data from disk into the system. It sounds like you have looked into most of what can be done. The information we have on how to improve performance (including loading data) is at the following link: High CPU Usage | Inactive User Interface | Poor Performance | Long Time to Load Chart Data | Charts Reloading Often For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2025-07-15 19:20:29] |
| Cell2985 - Posts: 3 |
|
Hi John, Thanks for the response. To clarify: all data is already downloaded. The issue is loading it from disk, specifically with tick data. I ran a disk speed test and my VPS shows ~33 MB/s random read speed (screenshot attached). Interestingly, Sierra Chart seems to cap out around that same speed during replay and chart loading. The VPS itself uses a Gen4 NVMe SSD capable of up to 8000 MB/s bursts, but Sierra Chart doesn’t seem to utilize that. Even when loading just 1 or 2 symbols, performance is slow — almost like the system is stuck using low-level SATA speeds. My question is: Is there a way to increase the speed at which Sierra Chart reads data from disk? Or: Can I force SC to use higher I/O throughput, closer to the actual 8000 MB/s that the hardware offers? Thanks again for your help, K. |
Private File Attachment Deleted. |
| [2025-07-15 22:15:32] |
| Sierra_Chart Engineering - Posts: 20217 |
|
This is definitively not a limitation within Sierra Chart and is fully impossible: Sierra Chart seems to cap out around that same speed during replay and chart loading. Sierra Chart has no such limits at all. None.The issue is with the hardware and it can relate to the SSD, the CPU, and RAM and also whether the data file exists sequentially on the SSD. It may not. And we would never use a VPS. Never. Those types of systems can never be trusted. You should just simply build your own computer, and not involve anyone else. No one. Put your trust in absolutely no one. One type of persistent memory, that delivers very fast random read performance is Intel Optane. This can be configured, as a drive. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-07-15 22:18:59
|
| [2025-07-15 23:15:32] |
| Sierra_Chart Engineering - Posts: 20217 |
|
We were thinking about this some more, being you are using a VPS, you can never be assured of, and it is very unlikely, that a large file (Sierra Chart intraday data files), would ever span a continuous block of memory within the SSD. It probably is very broken up. Therefore, the random read speed would apply. You should not be using a VPS and you should control your own solid-state drives completely with ones that you purchase and fully control. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2025-07-15 23:15:55
|
| [2025-07-16 08:10:13] |
| Cell2985 - Posts: 3 |
|
Dear legends, Understood! Not the answer I wanted to hear, but fully understand why and what is happening. Anyway, it is what it is. Thanks again for your reply and keep up the good work! Best, K. |
To post a message in this thread, you need to log in with your Sierra Chart account: