Support Board
Date/Time: Tue, 28 Apr 2026 04:28:28 +0000
Post From: CFE feed: PITCH data preservation in SCID/SCDD
| [2026-04-27 23:13:36] |
| AndyB - Posts: 108 |
|
I've been doing detailed binary-level analysis of CFE depth and intraday data files (VXK26_FUT_CFE specifically) and have a few questions about what Sierra Chart is currently preserving versus stripping from the source PITCH feed, and whether some of the missing information could be carried in the existing record structures. Upfront context: I understand Sierra Chart operates under licensing constraints with the exchanges and with Barchart/CQG, and that the recent CME DTC access changes show how those constraints can shift. So for each item below, even an answer of "Cboe/Barchart/CQG doesn't license us to redistribute that" is useful, since it tells me whether to look elsewhere rather than wait. I'm not asking for new data Sierra Chart isn't licensed to provide. I'm asking whether information Sierra Chart already receives is being unnecessarily dropped before reaching the SCID/SCDD files. Trade Condition codes in SCID. The CFE Multicast PITCH spec defines a one-byte Trade Condition field on Order Executed (0x23) and Trade (0x2A/0x2B) messages with values S = Spread trade, B = Block trade, E = ECRP trade, D = Derived, O = Opening trade. None of this is preserved in s_IntradayRecord. For VX I've empirically confirmed zero 0.005-aligned prints in a 1.15M-record SCID (i.e., zero apparent Block or ECRP trades), which suggests these condition-tagged trades are being filtered out entirely rather than just stripped of their tags. The existing SCID format already uses the Open field as a sentinel carrier: Open=0.0 for SINGLE_TRADE_WITH_BID_ASK, plus the FIRST_SUB and LAST_SUB constants provided for CME products. Adding 4 or 5 more sentinel float values for the PITCH trade conditions would be a backward-compatible extension following the exact pattern that already exists. No format version change required, no field reinterpretation, just additional sentinel constants that existing readers safely ignore. Question: is the upstream filtering happening at the Sierra Chart layer or at Barchart/CQG, and if the former, is preserving the condition tags in the Open field as additional sentinels something you'd consider? Transaction Begin/End in SCDD. PITCH wraps causally related events in Transaction Begin (0xBC) and Transaction End (0xBD) messages. For example, one aggressive order matching multiple resting orders gets bracketed as one transaction block. The SCDD format has FLAG_END_OF_BATCH but no transaction-block markers (I have a very intimate understanding of the peculiarities of your files, FYI, so no, I was not expecting these to be present), so this causal grouping appears to be lost during the L3 to L2 aggregation. The s_MarketDepthFileRecord struct already has a 4-byte Reserved field documented as "Not used, but exists due to byte padding." A transaction batch ID written into that field, with the same ID for all records within one PITCH transaction block and zero outside, would reconstruct the grouping perfectly without changing the record size or format version. Existing readers ignore the Reserved field, so this is fully backward-compatible. Alternatively, defining new Command enum values (COMMAND_TRANSACTION_BEGIN = 8, COMMAND_TRANSACTION_END = 9) would also work and adds at most two extra records per transaction. Question: is the transaction grouping information available anywhere in the Sierra Chart pipeline, or is it stripped during the L3 to L2 aggregation? If it's available, would either of the above approaches be feasible? SCDD timestamp source. **This is the most interesting to me**. I'm seeing genuine microsecond-precision timestamps in CFE depth files (deltas of 3us, 33us, 215us, etc., not the synthetic +1us counter pattern used in SCID). Documentation states depth is millisecond-only with no microsecond adjustment. This is a clarification request, not a feature request: are the microseconds I'm observing exchange-side timestamps preserved through the Barchart/CQG path, or vendor-ingest timestamps applied somewhere downstream? This affects whether they can be used for causal ordering relative to trade events. L3 order-level access. I recognize this is a much bigger ask than items 1 through 3 and likely runs into both engineering scope and licensing questions. Is there any current or planned mechanism to expose the L3 PITCH order stream (Add/Modify/Delete/Order Executed by Order Id) for CFE? Even an acknowledgment that this isn't available and isn't planned helps me decide whether to supplement Sierra Chart with a direct CFE feed. I appreciate that CFE may not be a priority exchange given its volume relative to CME, and I understand the support burden of the Barchart/CQG symbol mapping situation that's been discussed in past threads. Items 1 and 2 in particular look like small additions that fit cleanly into existing structures, so I wanted to surface them in case they're easier than they might initially appear. To summarize the questions: 1. Is the filtering of Trade Condition codes happening at Sierra Chart or upstream at Barchart/CQG, and if at Sierra Chart, would preserving them as additional Open-field sentinels be feasible? 2. Is PITCH transaction grouping available in the Sierra Chart pipeline, and if so, would either the Reserved-field batch ID or new Command enum values be feasible? 3. Are the microsecond timestamps I'm observing in CFE depth files exchange-side or vendor-ingest? 4. Is there any current or planned L3 order-level access for CFE? Thanks. |