Login Page - Create Account

Support Board


Date/Time: Wed, 19 Feb 2020 20:58:16 +0000



[User Discussion] - Sierra users, how good is your data feed?

[2020-01-17 16:15:28]
elji00 - Posts: 28 | Ending Date: 2020-05-28
I share a study (source code and DLL are attached) which timestamps the ticks arriving from data feed.
The study writes all the ticks into a .csv file.
The .csv file can be read by a spreadsheet (Excel for instance) after conversion.
The timestamps have millisecond precision.

The study necessitates the bar period to be set :
- for indexes : 1 second.
- for tradables : 1 trade per bar.

The study refresh is based on the Chart Update Interval of Sierra Chart. In order to get optimal measurements I recommend to set Chart Update Interval to 10 ms (Chart Settings >> Advanced Settings), and use a chartbook that can handle 10 ms updates (that means ideally charts with no other study). And for even better results, minimize the chart windows and Sierra.



A few notes:
- timezone difference between the computer and market data is handled automatically by the study.
- to get optimal timestamps I recommend to synchronize Windows before lauching the study or at least check its sync on www.time.is .
- the study does not write into the .csv file during a recalculation or during data download (data would be wrong).
- daily reset for intradayindex is 18:00 (it can be changed in line 125 of the code).



Maybe you are wondering what is the point of timestamping ticks at millisecond precision since Sierra does not yet offer milliseconds. Actually by observing the first ticks of every second, and the last ticks of every seconds you will get an estimation of the data feed lag.


In my next post I will show a few examples of how to read the data acquired by the study.
attachmentTimeStampDataFeed_v1.0_64.dll - Attached On 2020-01-17 16:14:07 UTC - Size: 890.5 KB - 27 views
attachmentTimeStampDataFeed_v1.0.cpp - Attached On 2020-01-17 16:14:34 UTC - Size: 9.07 KB - 24 views
[2020-01-17 16:53:10]
elji00 - Posts: 28 | Ending Date: 2020-05-28
First example : data from NQ100 index.

Image 1 (the local computer is in France) : the data arrive with an average delay of 230ms after the full second (000ms), with a lot of deviation.
Not shown here, the data sometime arrive with 2 seconds lag when the market is volatile.

Image 2 (the local computer is in Chicago (VPS)) : the data arrive with an average delay of 160ms after full second with much less deviation than in image 1.
imageImage 1 - NQ100 index - PC in France.PNG / V - Attached On 2020-01-17 16:51:58 UTC - Size: 101.42 KB - 66 views
imageImage 2 - NQ100 index - VPS in Chicago.PNG / V - Attached On 2020-01-17 16:53:02 UTC - Size: 99.15 KB - 51 views
[2020-01-17 17:21:56]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Second example : data from NQ100 future sent to a VPS in Chicago.

Image 3 shows that the ticks at 10:56:55 arrive with a delay under 29ms.


The other thing that this study showed is that this VPS gets a maximum of 13 packets of ticks per second, whatever the volatility, and that the delay was only rarely over 500ms (this can be guesstimated from the timestamps of the last ticks of the second).


Those feeds are from Barchart/Sierra.


Let me know how good your data feed is.
And don't hesitate to bother me if you need help with the study.
Date Time Of Last Edit: 2020-01-17 17:25:23
imageImage 3 - NQ100 future - VPS in Chicago.PNG / V - Attached On 2020-01-17 17:18:59 UTC - Size: 102.43 KB - 46 views
[2020-01-17 18:58:21]
T44 - Posts: 336 | Ending Date: 2020-08-01
The market is slow now but here is some data for ES on CME and CL on NYMEX.

Interested to see my Eurex latency but hardly anything trading there at the moment. Here is Dax attached.

I'm in Western Europe.

How do you interpret these results? As I understand, the feed is timestamped only to the second, so we'd have to take the minimum latency from the delta you calculate? e.g. if your true latency is 50 millisecond and a trade happens at 500ms past the second you'll receive it at 550 and your "delta" will say 550?

My minimum latency on both CME products was 98. Yet my minimum latency to Chicago at https://www.netmeter.co.uk/ping-test/ (Chicago (United States) - VULTR.net) is reported at 110ms.
Date Time Of Last Edit: 2020-01-17 19:15:45
attachmentTimestamps - ESH20_FUT_CME - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 18:58:06 UTC - Size: 65.89 KB - 17 views
attachmentTimestamps - CLH20_FUT_NYMEX - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 18:58:12 UTC - Size: 9.42 KB - 13 views
attachmentTimestamps - FDAXH20_FUT_Eurex - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 18:58:18 UTC - Size: 528 B - 16 views
[2020-01-17 19:08:39]
T44 - Posts: 336 | Ending Date: 2020-08-01
I suppose it would be interesting to run this at the close on e.g. ES & NQ simultaneously and see if more congested trading conditions cause the latency profile you describe.

Not at all concerned about my order routing speed as above 100 micros it doesn't make a difference, however if my feed gets 2-3 seconds behind that very well could make a difference, occasionally.

Once in two months of using the SC / Barchart feed I've seen what seemed like multiple seconds of latency not long after the US open, the DOMs were lagging badly, too.

Is the timestamp of order book updates included anywhere in the feed? I know its exposed in ACSIL. I'd like to know how many depth updates are lagging. Again, not having sub second timestamping may be an issue here.

Otherwise, very happy with this feed. For manual trading, as long as either data or order routing latency always is below 1 second I don't think there is any impact on value.
[2020-01-17 19:11:46]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Hi T44, thanks for your feedback. Alas I cannot see the 3 files you have uploaded, you need to set them "Public".


My minimum latency on both CME products was 98. Yet my minimum latency to Chicago at https://www.netmeter.co.uk/ping-test/ (Chicago (United States) - VULTR.net) is reported at 110ms.
Those numbers seem pretty fine to me. The ping (110ms) is a round trip, so you can expect a 110/2=55ms latency at best, which seems consistent with 98ms.


As I understand, the feed is timestamped only to the second, so we'd have to take the minimum latency from the delta you calculate? e.g. if your true latency is 50 millisecond and a trade happens at 500ms past the second you'll receive it at 550 and your "delta" will say 550?
Yes and yes.

The study I share will not tell the latency precisely, but by observing hundreds of seconds, you will get the feeling of what is happening : 1° an estimate of your latency, 2° an estimate of the stability of the latency when the market turns volatile.

The interesting ticks are the first one in the second, and the last one in the second.
When you get 1000 ticks per second (SP500 future with a bit of volatility for instance) and the delta says "400" then you know you have a delay of 400ms. Same here if the last tick of the second has a delta of 1400, the latency is about 400ms.


When volatility is low (say 3 ticks per second), the first tick only tells the max value of the latency, and the last tick only tells the min value of the latency.
Date Time Of Last Edit: 2020-01-17 19:18:06
[2020-01-17 19:16:59]
T44 - Posts: 336 | Ending Date: 2020-08-01
These are public now, thanks.

I'm running this on CL@NYMEX into the close and will report back in 20 mins.

Thanks for sharing this study, it is interesting.
Date Time Of Last Edit: 2020-01-17 19:18:01
[2020-01-17 19:34:41]
T44 - Posts: 336 | Ending Date: 2020-08-01
CL close. Min 71ms, max 1182 so at least 182ms of latency.
attachmentTimestamps - CLH20_FUT_NYMEX - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 19:33:00 UTC - Size: 365.19 KB - 12 views
[2020-01-17 20:24:02]
elji00 - Posts: 28 | Ending Date: 2020-05-28
T44, I have just finished analyzing your timestamps for ES.
Indeed your lowest Delta is 98ms, which means that at this moment the real latency was below 98ms.
Your highest Delta is 1117 (117ms beyond the next second) which means that at this moment the real latency was over 117ms.
[2020-01-17 20:53:03]
elji00 - Posts: 28 | Ending Date: 2020-05-28
I suppose it would be interesting to run this at the close on e.g. ES & NQ simultaneously and see if more congested trading conditions cause the latency profile you describe.

My PC in Europe has acquired days of data for NQ100 future and sometimes the delay gets ugly (> 2s).

I have also recently started acquiring data with a VPS located in Chicago. Fortunately the delay remains more acceptable when there is volatility, but there is yet an issue.
I asked Support about this here: https://www.sierrachart.com/SupportBoard.php?ThreadID=48210

Interestingly your data for ES show a similar packet pattern: the packets arrive with an 80ms heartbeat.
It would be interesting if you could provide the first seconds of market open to check if you get more than 13 packets per second (which is the ultimate maximum I ever got), and what the delay looks like.
Date Time Of Last Edit: 2020-01-17 20:54:28
[2020-01-17 20:55:37]
T44 - Posts: 336 | Ending Date: 2020-08-01
I'll provide the close in a few minutes, ES and NQ. Monday is the US market holiday.

I run an AWS instance linked up to the second data connection to use with laptop+4G in case of a power cut. If it is any use, I could run the monitor from that.

What I'd like to do is modify your study to give an alert if the delta becomes more than say 1 second, for more than a few trades. A visual alert for a potential lagging data feed would be useful. However without millisecond precision in the exchange data stamps I wonder if it would work well enough.

I'm connected to Eurex (and soon ICE/LIFFE) and would be especially interested in any data latency if the quotes are coming through a Barchart server in North America.
Date Time Of Last Edit: 2020-01-17 20:59:16
[2020-01-17 21:08:51]
T44 - Posts: 336 | Ending Date: 2020-08-01
Minimum 250ms, maximum 1400ms.
Price and volume columns removed as the data belongs to CME.

I'd appreciate if someone with a CQG feed could run a similar test. I wish I'd done it before moving. Especially for Eurex.
attachmentTimestamps - ESH20_FUT_CME - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 21:08:42 UTC - Size: 252.41 KB - 15 views
attachmentTimestamps - NQH20_FUT_CME - TU 1 tick - 2020-01-17.csv - Attached On 2020-01-17 21:08:47 UTC - Size: 105.96 KB - 13 views
[2020-01-17 21:43:30]
elji00 - Posts: 28 | Ending Date: 2020-05-28
I run an AWS instance linked up to the second data connection to use with laptop+4G in case of a power cut. If it is any use, I could run the monitor from that.
Thanks, but no need for me, you have already been very helpful!


What I'd like to do is modify your study to give an alert if the delta becomes more than say 1 second, for more than a few trades.
I have already coded this for my own robots. I'll send you code via PM (request for PM sent).


I'd appreciate if someone with a CQG feed could run a similar test. I wish I'd done it before moving. Especially for Eurex.
I'd appreciate if someone with Rithmic could participate as well :) Rithmic should be the the best I guess.



I have analyzed your latest data. The hearbeat of your packets may be slightly better than mine (maybe 60ms vs 80ms).

Have you subscribed to Denali or are you with the classic Sierra feed? I am with the classic.
Date Time Of Last Edit: 2020-01-17 21:48:03
[2020-01-21 19:53:55]
Chad - Posts: 165 | Ending Date: 2020-08-10
Hi elji, may I see a copy of your alert code for time deltas? Just sent you request for PM.
[2020-01-21 22:15:03]
elji00 - Posts: 28 | Ending Date: 2020-05-28
I have received several requests for Direct Message Permission today.

I am willing to share source code for studies handling data lag and other incidents.

All I ask in return is a contribution to this thread.

Posting a .csv file with one hour of 1-tick future data is considered a contribution.
[2020-01-21 22:31:32]
Chad - Posts: 165 | Ending Date: 2020-08-10
Here is ~100 minutes of tick data for the 30-year US Treasury futures, March 2020 expiry (ZBH20) - times shown are GMT -06:00 (Central Std Time).
Date Time Of Last Edit: 2020-01-22 00:19:05
[2020-01-21 22:38:21]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Thanks Chad, please set the file to "public".
[2020-01-21 22:41:48]
Chad - Posts: 165 | Ending Date: 2020-08-10
Whoops...should be fixed now.
[2020-01-21 22:52:27]
elji00 - Posts: 28 | Ending Date: 2020-05-28
I can see the attached file now.

Sorry you misunderstood my call about data acquisition.

What I asked for is to contribute with one hour of 1-tick future data acquired with the study "TimeStampDataFeed" of post #1.
Date Time Of Last Edit: 2020-01-21 22:54:21
[2020-01-21 23:12:30]
Chad - Posts: 165 | Ending Date: 2020-08-10
I wondered if that was case (acquired with study), but I took your message as it was :)

I am letting it run for the next hour, will post the .csv here when completed.
[2020-01-22 00:18:18]
Chad - Posts: 165 | Ending Date: 2020-08-10
Here's data for the past hour for NYMEX HH Natural Gas, Feb '20 expiry. Also, WTI Crude Oil, Mar '20 expiry. Again, timestamps in GMT -06:00 (US Central). Was collected while on a modest wifi connection, hence the relatively-high latency.
attachmentTimestamps - CLH20 - TU 1 tick - 2020-01-20.csv - Attached On 2020-01-22 00:15:39 UTC - Size: 26.97 KB - 13 views
attachmentTimestamps - NGG20 - TU 1 tick - 2020-01-20.csv - Attached On 2020-01-22 00:16:14 UTC - Size: 45.66 KB - 9 views
[2020-01-22 13:37:16]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Your data are totally unusable:
- your PC clock is ahead by 10min.
- your SC refresh is way beyond the 10ms I recommended.
Date Time Of Last Edit: 2020-01-22 13:37:36
[2020-01-22 14:59:34]
Chad - Posts: 165 | Ending Date: 2020-08-10
Sorry; I'll run again on my Chicago VPS and ensure the 10ms refresh is enabled with PC clock synched.
[2020-01-22 15:31:34]
nosast - Posts: 77 | Ending Date: 2020-02-29
Very interesting thread and I will contribute shortly on data from Europa to Chicago with Denali feed.

Problem I see is the Chart update interval. Since I went from W7 to W10 the update times have doubled or tripled and one core is nearly at 100% usage when setting update interval low on a market depth chart. Market Depth causes my CPU to struggle and it is an up to date machine with an i7 9700K.

Afaik Sierra is not using the low setting of 10ms when the update in reality takes more than 10ms. It will get delayed automatically.

But if I let the test run on just a blank candle chart without any studys 10 ms drawing time should be realistic.

Anyone else notice this issue with MDH and Windows 10?
[2020-01-22 15:44:48]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Anyone else notice this issue with MDH and Windows 10?
https://www.sierrachart.com/SupportBoard.php?ThreadID=47949

the update times have doubled or tripled
I observe exactly the same increase as you : between x2 and x3.

Afaik Sierra is not using the low setting of 10ms when the update in reality takes more than 10ms. It will get delayed automatically.
I suggest to close all chartbooks and open a new chartbook dedicated for data acquisition, and only open one or two charts, with no study added. The lower the refresh of SC and the better the sync of your PC, the better the results you get.
When you minimize SC, the CPU load should be close to 0%.
Date Time Of Last Edit: 2020-01-22 19:30:29
[2020-01-22 17:42:50]
Chad - Posts: 165 | Ending Date: 2020-08-10
Hi Elji, here is a new one for CLH20 - I made sure to complete all initial configurations that you listed. Sorry again for the incorrect versions, hopefully this one is sufficient.
attachmentTimestamps - CLH20 - TU 1 tick - 2020-01-22.csv - Attached On 2020-01-22 17:42:32 UTC - Size: 1.9 MB - 12 views
[2020-01-22 19:35:35]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Chad, same as before.
Your data are totally unusable:
- your PC clock is ahead by 10min.
- your SC refresh is way beyond the 10ms I recommended.
[2020-01-22 19:47:51]
Chad - Posts: 165 | Ending Date: 2020-08-10
I don't understand why; this latest file was made on a Windows Server 2012 VPS in Chicago, and I went to Internet Time settings from the taskbar then 'Update now' to sync with time.windows.com.

I set the SC refresh at 10ms from Global Settings>>General, and made sure the Chart settings were using the global refresh.
[2020-01-23 06:43:54]
nosast - Posts: 77 | Ending Date: 2020-02-29
Attached is my file logging NQ from Germany with Denali during globex asian session (low vol).

109 ms to 1300 ms

OS is Windows 10 and my typical ping to Chicago is around 110 ms.
Date Time Of Last Edit: 2020-01-23 13:01:14
attachmentTimestamps - NQH20_FUT_CME - TU 1 sec - 2020-01-22.csv - Attached On 2020-01-23 06:43:51 UTC - Size: 570.89 KB - 6 views
Attachment Deleted.
[2020-01-23 11:20:00]
elji00 - Posts: 28 | Ending Date: 2020-05-28
Thanks nosast, please set the file to "public".

To post a message in this thread, you need to login with your Sierra Chart account:

Login

Login Page - Create Account