Support Board
Date/Time: Sun, 19 Jul 2026 03:52:45 +0000
Post From: Binance - Market Depth Data (Level 2)
| [2026-06-04 19:32:55] |
| User537545 - Posts: 4 |
|
Thank you for the clarification, that makes complete sense. Since I am not a programmer I have built a DTC Bridge to the Binance API using Claude Opus 4.7. I have used your entire DTC documentation as well as the Binance API documentation. Live data and historical data is now streaming to a separate Sierra Chart installation, but I have some open questions: 1. Historical tick data download takes ~5-15 minutes for 24 hours of high-volume futures data (e.g. BTCUSDT-PERP, ~2-3M trades/day). The pipeline is: Binance public archive (.zip) → in-process decompression → JSON DTC messages via TCP loopback → SC client. I use uvloop and orjson on the bridge side; observed throughput is roughly 3000-5000 records/sec sustained. Is there a recommended page size, message batching strategy, or server-side pacing pattern that maximises SC's ingest rate? Is the bottleneck typically the JSON parsing in SC, the .scid disk write (flushed every 5s by default), or socket throughput? 2. How much "HISTORICAL_PRICE_DATA_RECORD_RESPONSE" messages can the SC-client reliably process per second using JSON-encoding? Is there a recommended page size or server-side pacing strategy to prevent SC from being overwhelmed? 3. During long-running historical data streams (e.g. several minutes of streaming HISTORICAL_PRICE_DATA_RECORD_RESPONSE messages for millions of tick records), does the SC client still enforce the heartbeat timeout (typically 2.5× HeartbeatIntervalInSeconds)? Should the server interleave HEARTBEAT messages between historical records, or is the steady flow of records sufficient to be treated as a "keep-alive"? If interleaved heartbeats are recommended, what's the optimal frequency that won't cause re-ordering issues with the historical stream?" 4. When the server streams HISTORICAL_PRICE_DATA_RECORD_RESPONSE messages, does SC perform any validation before writing to .scid? Specifically: (a) are records with StartDateTime strictly less than the last written record's timestamp silently dropped, kept, or do they overwrite? (b) Are duplicate timestamps (e.g. two records with identical StartDateTime) merged, kept side-by-side, or rejected? 5. Sierra Chart is using NTFS compression but since I am running the application on an iMac with M3 chip using CrossOver is there an alternative compression method that I can use to compress the .scid files? Currently I am receiving the errors '...Windows error code 50: request not supported'. Is this expected because CrossOver/Wine's NTFS layer on top of APFS does not support FSCTL_SET_COMPRESSION? Is there a built-in alternative within SC itself (e.g. zlib/zstd compression of records inside the .scid file, independent of the host filesystem)? 6. Is it supported by the Sierra Chart DTC client to set HISTORICAL_PRICE_DATA_REQUEST.UseZLibCompression=1 with JSON encoding instead of Binary encoding and can a measurable throughput gain be expected on a loopback connection (127.0.0.1), or does the additional CPU overhead for Deflate/Inflate tend to slow down the overall throughput? Are there any official benchmarks or recommendations regarding the data distance or bandwidth at which compression becomes worthwhile? If JSON+zlib is not officially supported, is there a tracking issue or planned improvement? We chose JSON encoding for the bridge to keep the implementation portable and debuggable; switching to Binary encoding would add significant complexity. Knowing whether compression is a viable optimisation path with JSON or not would help me prioritise." 7. If the DTC bridge crashes in the middle of a historical data stream, what happens during reconnection? Does SC resend requests starting from the last written .scid record (with StartDateTime = last_time + 1 µs), or is the entire range requested again? How many incomplete or duplicated records can the server deliver without corrupting the .scid file? 8. Sierra Chart sometimes sends Symbol="BTCUSD_PERP"+Exchange=“BINANCE” in the DTC-MARKET_DATA_REQUEST, and sometimes Symbol=""+Exchange=“BTCUSD-BINANCE”, and sometimes a combination of the two. Are there official documents describing SC's canonical symbol naming for crypto perpetuals (USD-M vs COIN-M), so that DTC server implementers can match SC's expectation deterministically? Or is the server expected to accept any naming variant the user types in their chart definitions, and resolve to the correct underlying instrument via exchangeInfo / symbol metadata? 9. If the "Intraday Data Storage Time Unit" is set to 1 tick and the user opens a daily chart for the same symbol, does SC aggregate the data on the fly from the .scid file, or does it retrieve the daily bars separately? If it retrieves them separately, does it then read both files (.scid for updates, .dly for daily data), or is one given priority over the other? 10. HISTORICAL_PRICE_DATA_TICK_RECORD_RESPONSE (804) does not contain fields for the bid price and ask price at the time of the trade — only AtBidOrAsk (direction), Price, and Volume. To allow server implementers to obtain bid/ask data per trade in the .scid format (for studies that use s_IntradayRecord.High as Ask and s_IntradayRecord.Low as Bid), the only option is the workaround described in the section “Providing Bid Volume and Ask Volume” using HISTORICAL_PRICE_DATA_RECORD_RESPONSE (803) with NumTrades=1. Is the limitation of TICK_RECORD_RESPONSE intentional? What exactly does SC write to s_IntradayRecord.High and s_IntradayRecord.Low when the server uses TICK_RECORD_RESPONSE (which does not provide these values)? My observation was that the fields are apparently populated with the live BBO current at the time of processing, which leads to records with Close < Low or Close > High in historical downloads. I have now switched to RECORD_RESPONSE with NumTrades=1 as the workaround, and the .scid records are clean (Close always within [Low, High], 0% bad records over 30 days). For confirmation: is this RECORD_RESPONSE-with-NumTrades=1 approach the long-term-supported convention for per-trade tick data with bid/ask context, or is it considered a transitional pattern until TICK_RECORD_RESPONSE gets BidPrice/AskPrice fields? Your documentation regarding the DTC protocol message structures is excellent. However, for server implementers there's no end-to-end reference: the 'DTC Protocol Test Client' is available for verifying messages, but no reference SERVER implementation exists that demonstrates best practices end-to-end (logon handshake → encoding negotiation → market data subscription lifecycle → historical streaming with backpressure → graceful shutdown). Do you have a reference DTC server implementation (in any language) I could study? Or could you point to an existing 3rd-party server known to implement the protocol correctly that I could use as a reference? I really appreciate your support. Date Time Of Last Edit: 2026-06-04 19:39:45
|