Support Board
Date/Time: Fri, 31 Jul 2026 17:47:53 +0000
Non Deterministic Downloads Historical Tick data.
View Count: 18
| [2026-07-31 14:52:02] |
| skalaydzhiyski - Posts: 128 |
|
Hi SC, I noticed something worrying in the historical downloaded data and i was hoping you can explain this Repeating Chart >> Edit >> Delete All Data and Download on CLM26-NYMEX nine times returned three different record counts. The Message Log records which server served each request, and the counts partition perfectly by server: ``` HD #290 ds28 7,198,824 HD #295 ds37 7,198,779 HD #291 ds37 7,198,779 HD #296 ds32 7,198,825 HD #292 ds32 7,198,825 HD #297 ds26 7,198,825 HD #293 ds26 7,198,825 HD #298 ds28 7,198,824 HD #299 ds37 7,198,779 ``` Every server returns the same count every time — the downloads are deterministic. Setting "CME Exchange Data Feed Historical Server Address Override" to ds32 and downloading three more times gave 7,198,825 each time, and the resulting .scid files were byte-identical (same MD5, 287,953,056 bytes). So the variation is entirely between servers, not between requests. Comparing the files: - **ds26 vs ds32** agree on every record: same timestamps, prices, volumes and bid/ask volumes across all 7,198,825 records. The only difference is 12 records where the Open field holds FIRST/LAST_SUB_TRADE_OF_UNBUNDLED_TRADE_VALUE on one and SINGLE_TRADE_WITH_BID_ASK on the other. - **ds37 is a strict subset.** Zero records are unique to it. It is missing 46 records that ds26 and ds32 both have. The same pattern appears on ZBM26-CBOT over an identical requested range: ``` HD #280 ds37 9,011,042 HD #286 ds32 9,013,817 ``` On 2026-05-18 alone, ds37 returns 157,565 records / 666,231 contracts against ds32's 158,263 records / 669,627 contracts. The 698 missing records carry 3,396 contracts of volume, so this is not a bundling or encoding difference — that volume is simply absent from ds37. 585 of the 698 are ordinary SINGLE_TRADE_WITH_BID_ASK records, and both files are internally balanced on unbundled-trade markers (ds37 16,556 FIRST / 16,556 LAST, ds32 16,611 / 16,611), so neither is truncated mid-bundle. Questions: 1. Why do ds26/ds32, ds28 and ds37 hold different amounts of data for the same expired contract? 2. Which server should be treated as authoritative? Is ds37 known to be incomplete, or are ds26/ds32 holding records they should not? 3. Since the server is selected automatically per request, is there a supported way to ensure a download is complete without pinning a specific server? 4. Are the servers expected to converge over time, or is a copy that is short permanently short? Both .scid files, the log excerpts and the full list of differing records are available on request. I find this really strange and worrying, but maybe my analysis is wrong, looking forward to hearing your 2 cents. |
To post a message in this thread, you need to log in with your Sierra Chart account: