Support Board
Date/Time: Thu, 14 May 2026 11:02:03 +0000
CFE feed: PITCH data preservation in SCID/SCDD
View Count: 221
| [2026-04-27 23:13:36] |
| AndyB - Posts: 111 |
|
I've been doing detailed binary-level analysis of CFE depth and intraday data files (VXK26_FUT_CFE specifically) and have a few questions about what Sierra Chart is currently preserving versus stripping from the source PITCH feed, and whether some of the missing information could be carried in the existing record structures. Upfront context: I understand Sierra Chart operates under licensing constraints with the exchanges and with Barchart/CQG, and that the recent CME DTC access changes show how those constraints can shift. So for each item below, even an answer of "Cboe/Barchart/CQG doesn't license us to redistribute that" is useful, since it tells me whether to look elsewhere rather than wait. I'm not asking for new data Sierra Chart isn't licensed to provide. I'm asking whether information Sierra Chart already receives is being unnecessarily dropped before reaching the SCID/SCDD files. Trade Condition codes in SCID. The CFE Multicast PITCH spec defines a one-byte Trade Condition field on Order Executed (0x23) and Trade (0x2A/0x2B) messages with values S = Spread trade, B = Block trade, E = ECRP trade, D = Derived, O = Opening trade. None of this is preserved in s_IntradayRecord. For VX I've empirically confirmed zero 0.005-aligned prints in a 1.15M-record SCID (i.e., zero apparent Block or ECRP trades), which suggests these condition-tagged trades are being filtered out entirely rather than just stripped of their tags. The existing SCID format already uses the Open field as a sentinel carrier: Open=0.0 for SINGLE_TRADE_WITH_BID_ASK, plus the FIRST_SUB and LAST_SUB constants provided for CME products. Adding 4 or 5 more sentinel float values for the PITCH trade conditions would be a backward-compatible extension following the exact pattern that already exists. No format version change required, no field reinterpretation, just additional sentinel constants that existing readers safely ignore. Question: is the upstream filtering happening at the Sierra Chart layer or at Barchart/CQG, and if the former, is preserving the condition tags in the Open field as additional sentinels something you'd consider? Transaction Begin/End in SCDD. PITCH wraps causally related events in Transaction Begin (0xBC) and Transaction End (0xBD) messages. For example, one aggressive order matching multiple resting orders gets bracketed as one transaction block. The SCDD format has FLAG_END_OF_BATCH but no transaction-block markers (I have a very intimate understanding of the peculiarities of your files, FYI, so no, I was not expecting these to be present), so this causal grouping appears to be lost during the L3 to L2 aggregation. The s_MarketDepthFileRecord struct already has a 4-byte Reserved field documented as "Not used, but exists due to byte padding." A transaction batch ID written into that field, with the same ID for all records within one PITCH transaction block and zero outside, would reconstruct the grouping perfectly without changing the record size or format version. Existing readers ignore the Reserved field, so this is fully backward-compatible. Alternatively, defining new Command enum values (COMMAND_TRANSACTION_BEGIN = 8, COMMAND_TRANSACTION_END = 9) would also work and adds at most two extra records per transaction. Question: is the transaction grouping information available anywhere in the Sierra Chart pipeline, or is it stripped during the L3 to L2 aggregation? If it's available, would either of the above approaches be feasible? SCDD timestamp source. **This is the most interesting to me**. I'm seeing genuine microsecond-precision timestamps in CFE depth files (deltas of 3us, 33us, 215us, etc., not the synthetic +1us counter pattern used in SCID). Documentation states depth is millisecond-only with no microsecond adjustment. This is a clarification request, not a feature request: are the microseconds I'm observing exchange-side timestamps preserved through the Barchart/CQG path, or vendor-ingest timestamps applied somewhere downstream? This affects whether they can be used for causal ordering relative to trade events. L3 order-level access. I recognize this is a much bigger ask than items 1 through 3 and likely runs into both engineering scope and licensing questions. Is there any current or planned mechanism to expose the L3 PITCH order stream (Add/Modify/Delete/Order Executed by Order Id) for CFE? Even an acknowledgment that this isn't available and isn't planned helps me decide whether to supplement Sierra Chart with a direct CFE feed. I appreciate that CFE may not be a priority exchange given its volume relative to CME, and I understand the support burden of the Barchart/CQG symbol mapping situation that's been discussed in past threads. Items 1 and 2 in particular look like small additions that fit cleanly into existing structures, so I wanted to surface them in case they're easier than they might initially appear. To summarize the questions: 1. Is the filtering of Trade Condition codes happening at Sierra Chart or upstream at Barchart/CQG, and if at Sierra Chart, would preserving them as additional Open-field sentinels be feasible? 2. Is PITCH transaction grouping available in the Sierra Chart pipeline, and if so, would either the Reserved-field batch ID or new Command enum values be feasible? 3. Are the microsecond timestamps I'm observing in CFE depth files exchange-side or vendor-ingest? 4. Is there any current or planned L3 order-level access for CFE? Thanks. |
| [2026-04-28 21:59:41] |
| Sierra_Chart Engineering - Posts: 23752 |
|
We are very busy and we do not have time for this. We are declining support for this. Also the data comes direct from CBOE. It does not come from Barchart or CQG. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2026-04-28 22:02:45
|
| [2026-04-30 16:19:44] |
| AndyB - Posts: 111 |
|
I appreciate the correction on the data source coming direct from Cboe rather than Barchart or CQG. That alone was useful and updates my understanding. I want to push back gently on the “declining support” framing though. I’m not asking for support in the sense of help configuring the software or troubleshooting an issue. I’m asking clarification questions about your own data files and what’s preserved versus dropped from the source feed. Three of my four questions were yes/no or “exchange-side or vendor-ingest” answerable in a sentence each. The fourth was a planning question about future L3 access that “no plans” would have fully answered. Given that you’ve now confirmed the data comes direct from Cboe, the technical questions become straightforward to answer for someone with knowledge of your pipeline: 1. Are PITCH Trade Condition codes received and discarded, or filtered upstream of your ingest? 2.Are Transaction Begin/End messages received and discarded, or filtered upstream? 3. Are the microsecond timestamps in SCDD files exchange-side or applied during your ingest? 4. Any current or planned L3 access for CFE, even just yes/no? Heavy emphasis on question #3. I need to understand the time stamping convention, because you have only described behaviors relevant to CME. The dismissiveness wasn’t necessary for questions about your own data, especially considering I’ve been a paying customer for seven years. A one-line answer per question would have closed this out. I do understand that most support requests you receive pertain to topics that users could easily answer themselves if they were to read the documentation. But I’m likely one of the few who has actually done so. I only submit support requests as a last ditch effort. Thanks. PS. I am referring to the _FUT_CFE tagged /vx symbol. I have not looked at the -CFE tagged data yet and do not know if the time stamping is the same or different. Here are seven consecutive records from VXK26_FUT_CFE.2026-04-27.depth showing genuine microsecond-level timestamp deltas. Fields are DateTime, Command, Flags, NumOrders, Price, Quantity, Reserved: 2026-04-27 00:00:00.186571, 5, 1, 25, 20.75, 204, 0 2026-04-27 00:00:00.186575, 5, 1, 21, 20.799999237060547, 233, 0 2026-04-27 00:00:00.186620, 4, 1, 3, 20.700000762939453, 15, 0 2026-04-27 00:00:00.186630, 4, 1, 4, 20.700000762939453, 41, 0 2026-04-27 00:00:00.186667, 4, 1, 5, 20.700000762939453, 59, 0 2026-04-27 00:00:00.186692, 4, 1, 6, 20.700000762939453, 60, 0 2026-04-27 00:00:00.186696, 5, 1, 24, 20.75, 202, 0 Deltas of 4us, 45us, 10us, 37us, 25us, 4us. These don’t look like the synthetic +1us counter pattern. For reference, here are records from VXK26_FUT_CFE.scid showing the documented synthetic +1us pattern at the SCID level for comparison. Fields are DateTime, Open, High, Low, Close, NumTrades, TotalVolume, BidVolume, AskVolume: 2025-09-08 18:47:02.254000, 0.0, 21.899999618530273, 21.799999237060547, 21.850000381469727, 1, 1, 0, 1 2025-09-08 18:47:02.254001, 0.0, 21.850000381469727, 21.799999237060547, 21.850000381469727, 1, 1, 0, 1 2025-09-09 13:55:24.431000, 0.0, 21.950000762939453, 21.850000381469727, 21.899999618530273, 1, 1, 1, 0 2025-09-09 13:55:24.431001, 0.0, 21.899999618530273, 21.850000381469727, 21.899999618530273, 1, 1, 0, 1 2025-09-09 13:55:24.431002, 0.0, 21.950000762939453, 21.899999618530273, 21.899999618530273, 1, 1, 1, 0 So my question is whether the depth file timestamps are exchange-side (preserved from PITCH through to SCDD) or applied somewhere in your ingest pipeline. |
| [2026-04-30 16:45:30] |
| seandunaway - Posts: 385 |
|
it sounds like you're working on cool stuff 😎 i agree, it's sierrachart's openness about such things and documenting it that really separates them from other options it's a good point this sort of thing is more logically deserving of support than HoW dO I cHaNgE tHe FoNt CoLoR? |
| [2026-05-06 13:32:56] |
| AndyB - Posts: 111 |
|
Bumping this. I am hoping to get some answers to my questions listed in my previous message. I am not asking for support, just clarity. Thank you |
| [2026-05-11 06:24:18] |
| Sierra_Chart Engineering - Posts: 23752 |
|
We apologize for the delay. We are just able to quickly get to these right now. This is all for now.: 3. Are the microsecond timestamps in SCDD files exchange-side or applied during your ingest? These definitely come from the exchange but we are not sure we are using the microseconds, probably just milliseconds but we have to check on that.4. Any current or planned L3 access for CFE, even just yes/no? Are you referring to market by order data? This is definitely processed with CFE and relayed through. There should not be any filtering of this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2026-05-11 06:26:31
|
| [2026-05-11 06:37:21] |
| Sierra_Chart Engineering - Posts: 23752 |
|
We did see that there is filtering for CFE for market by order data and that filtering has been removed. We did see that orders of a quantity of 1, were getting filtered and that has now been removed.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing |
| [2026-05-11 19:14:57] |
| AndyB - Posts: 111 |
|
Thanks for getting back, I appreciate the engagement. To clarify what I'm asking: I've already done detailed binary-level analysis of the SCID and SCDD files, so I know exactly what's in them. What I'm asking is what's present at the boundary between Cboe and your ingest, and what your licensing allows you to pass through, so I know whether to keep asking you about these things or look elsewhere. Q1: At the boundary between Cboe and your ingest, do you receive PITCH Order Executed (0x23) and Trade (0x2A/0x2B) messages with the Trade Condition byte (S=Spread, B=Block, E=ECRP, D=Derived, O=Opening)? Q2: At that same boundary, do you receive PITCH Transaction Begin (0xBC) and Transaction End (0xBD) boundary messages? Q3: Definitely get back to me on the millisecond/microsecond usage findings. Like I was saying earlier, the SCID records have a microsecond portion of the timestamp on par with what you'd expect to see per the documentation outlining the fact that it is merely a counter: 2025-09-08 18:47:02.254000, 0.0, 21.899999618530273, 21.799999237060547, 21.850000381469727, 1, 1, 0, 1 2025-09-08 18:47:02.254001, 0.0, 21.850000381469727, 21.799999237060547, 21.850000381469727, 1, 1, 0, 1 2025-09-09 13:55:24.431000, 0.0, 21.950000762939453, 21.850000381469727, 21.899999618530273, 1, 1, 1, 0 2025-09-09 13:55:24.431001, 0.0, 21.899999618530273, 21.850000381469727, 21.899999618530273, 1, 1, 0, 1 2025-09-09 13:55:24.431002, 0.0, 21.950000762939453, 21.899999618530273, 21.899999618530273, 1, 1, 1, 0 The depth records, however, do not seem to follow the "counter" logic, and seem to have their own actual microseconds: 2026-04-27 00:00:00.186575 5 1 21 20.799999237060547 233 2026-04-27 00:00:00.186620 4 1 3 20.700000762939453 15 2026-04-27 00:00:00.186630 4 1 4 20.700000762939453 41 2026-04-27 00:00:00.186667 4 1 5 20.700000762939453 59 2026-04-27 00:00:00.186692 4 1 6 20.700000762939453 60 2026-04-27 00:00:00.186696 5 1 24 20.75 202 2026-04-27 00:00:00.264284 4 1 5 20.700000762939453 59 2026-04-27 00:00:00.274228 5 1 14 24.299999237060547 347 2026-04-27 00:00:00.321752 4 1 24 20.649999618530273 293 2026-04-27 00:00:00.353627 5 1 5 25.200000762939453 53 2026-04-27 00:00:00.409679 4 1 6 20.700000762939453 78 2026-04-27 00:00:00.409732 4 1 7 20.700000762939453 79 2026-04-27 00:00:00.442353 5 1 23 20.75 197 2026-04-27 00:00:00.475546 5 1 3 26.299999237060547 11 2026-04-27 00:00:00.484968 4 1 6 20.700000762939453 60 2026-04-27 00:00:00.610145 5 1 15 24.25 378 As you can see, the microseconds are very much included, as opposed to CME's depth records that are granular only to the millisecond (except for in specific snapshot cases). So the question for Q3 is whether those microseconds are coming through from the Cboe feed at your ingest boundary, or whether something in your pipeline is the source of them. For full context, I'm trying to reconcile trades against depth updates. With CME, that's tractable because SCID files carry the first/last sub-trade sentinels, depth records are millisecond-granular, and trades are too (with microseconds as a counter). That gives me enough to match programmatically. CFE doesn't carry the same signals, which is why I'm trying to pin down what is and isn't available at your end. Thank you. |
| [2026-05-13 19:40:22] |
| Sierra_Chart Engineering - Posts: 23752 |
|
The timestamps with microsecond resolution for market depth data from CFE is from the exchange.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2026-05-13 19:42:07
|
To post a message in this thread, you need to log in with your Sierra Chart account: