Support Board
Date/Time: Fri, 03 Apr 2026 23:27:00 +0000
Intra-millisecond tick sequencing
View Count: 201
| [2026-02-16 10:34:17] |
| skalaydzhiyski - Posts: 126 |
|
Hi SC, I am training ML models on the tick data exported from Sierra Chart and as part of my data engineering pipeline I need to figure out one thing relating the sequencing of the data I believe to be very important. From what I have seen, Sierra saves the data (when 1 tick storage is selected) into millisecond level entries in the .scid file, but if we have multiple trades happening inside a millisecond, then Sierra increments the microsecond portion of the timestamps to allow for uniqueness. This all makes perfect sense to me, but the important clarification I need is - apart from uniqueness, does the increment provide ordering/sequencing too ? If we were to constrain the question to CME, CBOT, NYMEX, COMEX data I believe the exchange provides sequence numbers as part of the data feed, is the ordering derived from those sequence numbers or is there no real ordering of the intra-millisecond ticks when they are stored in .scid files? NOTE: This can also have implications on how Volume Bars are constructed - if t1, t2, t3 are 3 unique ticks with volumes 10,20,30 happening inside the same millisecond WITHOUT ordering and I have the following setup: 1. I have Split Data Records disabled (I need this to keep my data realistic for training my models) 2. I have volume bars of size 100 on my chart. 3. My current bar is at volume 80. Then -> when the 3 trades happen at the same time it really matters whether the 10 volume trade happens first or the 30 volume trade happens first in determine the bar cutoff. Separately, is the some kind of ordering maintained in the .depth files, i.e. do we have a guarantee that 2 rows in .depth file happened 1 after the other? I would love to go deeper on this topic because I feel it is very important for anyone doing serious ML research using Sierra Chart. All the best Date Time Of Last Edit: 2026-02-16 12:38:37
|
| [2026-02-16 23:18:26] |
| Sierra_Chart Engineering - Posts: 23425 |
|
The ordering of trades within Intraday data files is exactly the same as provided by the exchange. This is guaranteed. There is no other possibility. In the case of market depth data, that is not an exact representation of the data from the exchange when it comes to ordering during a batch of market depth and order by order updates from the exchange. The updates that you see within a very short span of time, a millisecond, may not exactly represent the same order as from the exchange. It is too complex of a subject for us to be explaining here and now, regarding the market depth data. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2026-02-17 00:03:40
|
| [2026-02-16 23:27:53] |
| skalaydzhiyski - Posts: 126 |
|
I guess I was asking in case there is a way of joining the depth data with the scid time and sales data into a unique ordering? If that is not the case i have to group by millisecond for the time being... |
| [2026-02-17 03:42:32] |
| Sierra_Chart Engineering - Posts: 23425 |
|
No there is not a way to do this: I guess I was asking in case there is a way of joining the depth data with the scid time and sales data into a unique ordering?
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2026-02-17 16:20:13] |
| skalaydzhiyski - Posts: 126 |
|
Hmm, that is what I don't understand though, Sierra Chart is replaying market depth from the downloaded historical files and it's always perfectly consistent, i.e. if I run multiple replays with the same data at the exact same millisecond and I run the replay 1 tick at a time, I can clearly see that the orderbook is updating absolutely correctly and consistently, so internally there must be some algorithm for matching the .depth and the .scid at that time level. Can you please share that with me ? I can implement everything myself that's not a problem I just need to know what Sierra does behind the scenes in pseudo-code or explained in English... I need this so that my research environment is aligned with my production environment (Sierra). Date Time Of Last Edit: 2026-02-17 16:21:17
|
| [2026-02-17 22:49:20] |
| Sierra_Chart Engineering - Posts: 23425 |
|
The method for matching, market depth data, and the chart bars is simply the timestamps. If a trade occurs at 01:00:02.300 within a chart bar, then the market depth data, with timestamps up to that time are included in the chart bar. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: