Home >> Support Board

Support Board

Date/Time: Mon, 29 Apr 2024 07:15:59 +0000

How to Post a Message for Support

[User Discussion] - Sierra users, how good is your data feed?

View Count: 2747

[2020-01-17 16:15:28] #1 [Go To First Post]
JM-JO - Posts: 38
I share a study (source code and DLL are attached) which timestamps the ticks arriving from data feed. The study writes all the ticks into a .csv file. The .csv file can be read by a spreadsheet (Excel for instance) after conversion. The timestamps have millisecond precision. The study necessitates the bar period to be set : - for indexes : 1 second. - for tradables : 1 trade per bar. The study refresh is based on the Chart Update Interval of Sierra Chart. In order to get optimal measurements I recommend to set Chart Update Interval to 10 ms (Chart Settings >> Advanced Settings), and use a chartbook that can handle 10 ms updates (that means ideally charts with no other study). And for even better results, minimize the chart windows and Sierra. A few notes: - timezone difference between the computer and market data is handled automatically by the study. - to get optimal timestamps I recommend to synchronize Windows before lauching the study or at least check its sync on www.time.is . - the study does not write into the .csv file during a recalculation or during data download (data would be wrong). - daily reset for intradayindex is 18:00 (it can be changed in line 125 of the code). Maybe you are wondering what is the point of timestamping ticks at millisecond precision since Sierra does not yet offer milliseconds. Actually by observing the first ticks of every second, and the last ticks of every seconds you will get an estimation of the data feed lag. In my next post I will show a few examples of how to read the data acquired by the study.


TimeStampDataFeed_v1.0_64.dll - Attached On 2020-01-17 16:14:07 UTC - Size: 890.5 KB - 562 views TimeStampDataFeed_v1.0.cpp - Attached On 2020-01-17 16:14:34 UTC - Size: 9.07 KB - 336 views

[2020-01-17 16:53:10] #2 [Go To First Post]
JM-JO - Posts: 38
First example : data from NQ100 index. Image 1 (the local computer is in France) : the data arrive with an average delay of 230ms after the full second (000ms), with a lot of deviation. Not shown here, the data sometime arrive with 2 seconds lag when the market is volatile. Image 2 (the local computer is in Chicago (VPS)) : the data arrive with an average delay of 160ms after full second with much less deviation than in image 1.


Image 1 - NQ100 index - PC in France.PNG / V - Attached On 2020-01-17 16:51:58 UTC - Size: 101.42 KB - 406 views Image 2 - NQ100 index - VPS in Chicago.PNG / V - Attached On 2020-01-17 16:53:02 UTC - Size: 99.15 KB - 347 views

[2020-01-17 17:21:56] #3 [Go To First Post]
JM-JO - Posts: 38
Second example : data from NQ100 future sent to a VPS in Chicago. Image 3 shows that the ticks at 10:56:55 arrive with a delay under 29ms. The other thing that this study showed is that this VPS gets a maximum of 13 packets of ticks per second, whatever the volatility, and that the delay was only rarely over 500ms (this can be guesstimated from the timestamps of the last ticks of the second). Those feeds are from Barchart/Sierra. Let me know how good your data feed is. And don't hesitate to bother me if you need help with the study. Date Time Of Last Edit: 2020-01-17 17:25:23


Image 3 - NQ100 future - VPS in Chicago.PNG / V - Attached On 2020-01-17 17:18:59 UTC - Size: 102.43 KB - 313 views

[2020-01-17 18:58:21] #4 [Go To First Post]
T44 - Posts: 363
The market is slow now but here is some data for ES on CME and CL on NYMEX. Interested to see my Eurex latency but hardly anything trading there at the moment. Here is Dax attached. I'm in Western Europe. How do you interpret these results? As I understand, the feed is timestamped only to the second, so we'd have to take the minimum latency from the delta you calculate? e.g. if your true latency is 50 millisecond and a trade happens at 500ms past the second you'll receive it at 550 and your "delta" will say 550? My minimum latency on both CME products was 98. Yet my minimum latency to Chicago at https://www.netmeter.co.uk/ping-test/ (Chicago (United States) - VULTR.net) is reported at 110ms. Date Time Of Last Edit: 2020-01-17 19:15:45


Timestamps - ESH20_FUT_CME - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 18:58:06 UTC - Size: 65.89 KB - 284 views Timestamps - CLH20_FUT_NYMEX - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 18:58:12 UTC - Size: 9.42 KB - 286 views Timestamps - FDAXH20_FUT_Eurex - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 18:58:18 UTC - Size: 528 B - 257 views

[2020-01-17 19:08:39] #5 [Go To First Post]
T44 - Posts: 363
I suppose it would be interesting to run this at the close on e.g. ES & NQ simultaneously and see if more congested trading conditions cause the latency profile you describe. Not at all concerned about my order routing speed as above 100 micros it doesn't make a difference, however if my feed gets 2-3 seconds behind that very well could make a difference, occasionally. Once in two months of using the SC / Barchart feed I've seen what seemed like multiple seconds of latency not long after the US open, the DOMs were lagging badly, too. Is the timestamp of order book updates included anywhere in the feed? I know its exposed in ACSIL. I'd like to know how many depth updates are lagging. Again, not having sub second timestamping may be an issue here. Otherwise, very happy with this feed. For manual trading, as long as either data or order routing latency always is below 1 second I don't think there is any impact on value.

[2020-01-17 19:11:46] #6 [Go To First Post]
JM-JO - Posts: 38
Hi T44, thanks for your feedback. Alas I cannot see the 3 files you have uploaded, you need to set them "Public". My minimum latency on both CME products was 98. Yet my minimum latency to Chicago at https://www.netmeter.co.uk/ping-test/ (Chicago (United States) - VULTR.net) is reported at 110ms. Those numbers seem pretty fine to me. The ping (110ms) is a round trip, so you can expect a 110/2=55ms latency at best, which seems consistent with 98ms. As I understand, the feed is timestamped only to the second, so we'd have to take the minimum latency from the delta you calculate? e.g. if your true latency is 50 millisecond and a trade happens at 500ms past the second you'll receive it at 550 and your "delta" will say 550? Yes and yes. The study I share will not tell the latency precisely, but by observing hundreds of seconds, you will get the feeling of what is happening : 1° an estimate of your latency, 2° an estimate of the stability of the latency when the market turns volatile. The interesting ticks are the first one in the second, and the last one in the second. When you get 1000 ticks per second (SP500 future with a bit of volatility for instance) and the delta says "400" then you know you have a delay of 400ms. Same here if the last tick of the second has a delta of 1400, the latency is about 400ms. When volatility is low (say 3 ticks per second), the first tick only tells the max value of the latency, and the last tick only tells the min value of the latency. Date Time Of Last Edit: 2020-01-17 19:18:06

[2020-01-17 19:16:59] #7 [Go To First Post]
T44 - Posts: 363
These are public now, thanks. I'm running this on CL@NYMEX into the close and will report back in 20 mins. Thanks for sharing this study, it is interesting. Date Time Of Last Edit: 2020-01-17 19:18:01

[2020-01-17 19:34:41] #8 [Go To First Post]
T44 - Posts: 363
CL close. Min 71ms, max 1182 so at least 182ms of latency.


Timestamps - CLH20_FUT_NYMEX - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 19:33:00 UTC - Size: 365.19 KB - 281 views

[2020-01-17 20:24:02] #9 [Go To First Post]
JM-JO - Posts: 38
T44, I have just finished analyzing your timestamps for ES. Indeed your lowest Delta is 98ms, which means that at this moment the real latency was below 98ms. Your highest Delta is 1117 (117ms beyond the next second) which means that at this moment the real latency was over 117ms.

[2020-01-17 20:53:03] #10 [Go To First Post]
JM-JO - Posts: 38
I suppose it would be interesting to run this at the close on e.g. ES & NQ simultaneously and see if more congested trading conditions cause the latency profile you describe. My PC in Europe has acquired days of data for NQ100 future and sometimes the delay gets ugly (> 2s). I have also recently started acquiring data with a VPS located in Chicago. Fortunately the delay remains more acceptable when there is volatility, but there is yet an issue. I asked Support about this here: Sierra Chart Real-Time Data Feed is not fast enough. How to reduce lag ? Interestingly your data for ES show a similar packet pattern: the packets arrive with an 80ms heartbeat. It would be interesting if you could provide the first seconds of market open to check if you get more than 13 packets per second (which is the ultimate maximum I ever got), and what the delay looks like. Date Time Of Last Edit: 2020-01-17 20:54:28

[2020-01-17 20:55:37] #11 [Go To First Post]
T44 - Posts: 363
I'll provide the close in a few minutes, ES and NQ. Monday is the US market holiday. I run an AWS instance linked up to the second data connection to use with laptop+4G in case of a power cut. If it is any use, I could run the monitor from that. What I'd like to do is modify your study to give an alert if the delta becomes more than say 1 second, for more than a few trades. A visual alert for a potential lagging data feed would be useful. However without millisecond precision in the exchange data stamps I wonder if it would work well enough. I'm connected to Eurex (and soon ICE/LIFFE) and would be especially interested in any data latency if the quotes are coming through a Barchart server in North America. Date Time Of Last Edit: 2020-01-17 20:59:16

[2020-01-17 21:08:51] #12 [Go To First Post]
T44 - Posts: 363
Minimum 250ms, maximum 1400ms. Price and volume columns removed as the data belongs to CME. I'd appreciate if someone with a CQG feed could run a similar test. I wish I'd done it before moving. Especially for Eurex.


Timestamps - ESH20_FUT_CME - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 21:08:42 UTC - Size: 252.41 KB - 275 views Timestamps - NQH20_FUT_CME - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 21:08:47 UTC - Size: 105.96 KB - 253 views

[2020-01-17 21:43:30] #13 [Go To First Post]
JM-JO - Posts: 38
I run an AWS instance linked up to the second data connection to use with laptop+4G in case of a power cut. If it is any use, I could run the monitor from that. Thanks, but no need for me, you have already been very helpful! What I'd like to do is modify your study to give an alert if the delta becomes more than say 1 second, for more than a few trades. I have already coded this for my own robots. I'll send you code via PM (request for PM sent). I'd appreciate if someone with a CQG feed could run a similar test. I wish I'd done it before moving. Especially for Eurex. I'd appreciate if someone with Rithmic could participate as well :) Rithmic should be the the best I guess. I have analyzed your latest data. The hearbeat of your packets may be slightly better than mine (maybe 60ms vs 80ms). Have you subscribed to Denali or are you with the classic Sierra feed? I am with the classic. Date Time Of Last Edit: 2020-01-17 21:48:03

[2020-01-21 19:53:55] #14 [Go To First Post]
Chad - Posts: 231
Hi elji, may I see a copy of your alert code for time deltas? Just sent you request for PM.

[2020-01-21 22:15:03] #15 [Go To First Post]
JM-JO - Posts: 38
I have received several requests for Direct Message Permission today. I am willing to share source code for studies handling data lag and other incidents. All I ask in return is a contribution to this thread. Posting a .csv file with one hour of 1-tick future data is considered a contribution.

[2020-01-21 22:31:32] #16 [Go To First Post]
Chad - Posts: 231
Here is ~100 minutes of tick data for the 30-year US Treasury futures, March 2020 expiry (ZBH20) - times shown are GMT -06:00 (Central Std Time). Date Time Of Last Edit: 2020-01-22 00:19:05

[2020-01-21 22:38:21] #17 [Go To First Post]
JM-JO - Posts: 38
Thanks Chad, please set the file to "public".

[2020-01-21 22:41:48] #18 [Go To First Post]
Chad - Posts: 231
Whoops...should be fixed now.

[2020-01-21 22:52:27] #19 [Go To First Post]
JM-JO - Posts: 38
I can see the attached file now. Sorry you misunderstood my call about data acquisition. What I asked for is to contribute with one hour of 1-tick future data acquired with the study "TimeStampDataFeed" of post #1. Date Time Of Last Edit: 2020-01-21 22:54:21

[2020-01-21 23:12:30] #20 [Go To First Post]
Chad - Posts: 231
I wondered if that was case (acquired with study), but I took your message as it was :) I am letting it run for the next hour, will post the .csv here when completed.

[2020-01-22 00:18:18] #21 [Go To First Post]
Chad - Posts: 231
Here's data for the past hour for NYMEX HH Natural Gas, Feb '20 expiry. Also, WTI Crude Oil, Mar '20 expiry. Again, timestamps in GMT -06:00 (US Central). Was collected while on a modest wifi connection, hence the relatively-high latency.


Timestamps - CLH20 - TU 1 tick - 2020-01-20.csv - Attached On 2020-01-22 00:15:39 UTC - Size: 26.97 KB - 266 views Timestamps - NGG20 - TU 1 tick - 2020-01-20.csv - Attached On 2020-01-22 00:16:14 UTC - Size: 45.66 KB - 237 views

[2020-01-22 13:37:16] #22 [Go To First Post]
JM-JO - Posts: 38
Your data are totally unusable: - your PC clock is ahead by 10min. - your SC refresh is way beyond the 10ms I recommended. Date Time Of Last Edit: 2020-01-22 13:37:36

[2020-01-22 14:59:34] #23 [Go To First Post]
Chad - Posts: 231
Sorry; I'll run again on my Chicago VPS and ensure the 10ms refresh is enabled with PC clock synched.

[2020-01-22 15:31:34] #24 [Go To First Post]
nosast - Posts: 290
Very interesting thread and I will contribute shortly on data from Europa to Chicago with Denali feed. Problem I see is the Chart update interval. Since I went from W7 to W10 the update times have doubled or tripled and one core is nearly at 100% usage when setting update interval low on a market depth chart. Market Depth causes my CPU to struggle and it is an up to date machine with an i7 9700K. Afaik Sierra is not using the low setting of 10ms when the update in reality takes more than 10ms. It will get delayed automatically. But if I let the test run on just a blank candle chart without any studys 10 ms drawing time should be realistic. Anyone else notice this issue with MDH and Windows 10?

[2020-01-22 15:44:48] #25 [Go To First Post]
JM-JO - Posts: 38
Anyone else notice this issue with MDH and Windows 10? SC chart drawing time with Win 10 is much longer than with Win 7 the update times have doubled or tripled I observe exactly the same increase as you : between x2 and x3. Afaik Sierra is not using the low setting of 10ms when the update in reality takes more than 10ms. It will get delayed automatically. I suggest to close all chartbooks and open a new chartbook dedicated for data acquisition, and only open one or two charts, with no study added. The lower the refresh of SC and the better the sync of your PC, the better the results you get. When you minimize SC, the CPU load should be close to 0%. Date Time Of Last Edit: 2020-01-22 19:30:29

To post a message in this thread, you need to log in with your Sierra Chart account: