Login Page - Create Account

Support Board


Date/Time: Wed, 24 Apr 2024 09:29:01 +0000



[Locked] - Denali Data Feed Compression Available

View Count: 6732

[2020-09-25 04:04:54]
ertrader - Posts: 644
For the southern coast Houston area, what do you recommend for TT routing/Futures data? Current primary is set to Server 1 with backup Server 1 as Server 4 and backup server 2 as server 4. These are defaults I have not changed.
Date Time Of Last Edit: 2020-09-25 04:15:51
[2020-09-25 05:01:16]
Sierra Chart Engineering - Posts: 104368
Use these servers in the following order:
Server 2
Server 1
Server 4

You can also swap the first two servers. These servers are only for trading.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2020-09-29 13:38:07]
ejtrader - Posts: 688
SC Team - with the reduced bandwidth usage with data compression - Would it be possible to include the Native Exchange timestamp for the CME data?

Based on your post below - bandwidth usage was a concern for not providing the native CME exchange timestamp.

Version 2151 Available: Foundation For Millisecond/Microsecond Timestamping | Post: 231087

Thanks
Date Time Of Last Edit: 2020-09-29 13:38:22
[2020-09-29 19:40:02]
Sierra Chart Engineering - Posts: 104368
We can do that because the DTC message that is being used does support microsecond time stamping. But we do want a unique timestamp per trade but there is not much issue if there is not some of the time. We will reconsider in a few weeks.

We are also going to be testing higher compression ratios. We should be able to achieve about 3 to 4 ongoing.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2020-09-29 19:40:43
[2020-10-01 01:13:21]
Sierra Chart Engineering - Posts: 104368
We want to update this thread in regards to compression. The data feed compression has proven to work exceptionally well. At some point compression will always be enabled and cannot be disabled. There is no reason to disable compression and all users using the Denali data feed should update to the current version of Sierra Chart.

Compression does reduce latency simply because there there is less information to transmit and therefore takes less time to be received and uses less bandwidth both on the server and client-side.

We are going to expand compression for all market data connections also use it for data transfer between our own systems.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2020-10-01 01:14:35
[2020-10-01 09:25:35]
User884550 - Posts: 149
does this compression feature also somehow relate to CQG data users?
I have also noticed jumpy chart update although settings are 50ms, all started in Sept
[2020-10-01 12:19:29]
Sierra Chart Engineering - Posts: 104368
CQG does not use data feed compression.

We recommend trying the Denali Exchange Data Feed to see if that works better for you:
Denali Exchange Data Feed

For a demonstration of this data feed, use the Delayed Exchange Data Feed:
Delayed Exchange Data Feed
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2020-10-06 11:07:14]
User754985 - Posts: 109
https://blosc.org/
[2020-10-06 20:36:59]
Sierra Chart Engineering - Posts: 104368
We have discovered a stability issue in our real-time server process, that has been triggered by compression. This has caused an abnormal shutdown of the process, which would cause lost connections during the trading day or evening. At that time you just simply would be reconnected within two seconds to another server.

We have isolated the problem, and resolved it. We are releasing the updated software in the next 30 minutes.

The issue was not caused by compression itself. Compression is completely stable and fast. It relates to the shutdown process of a network connection, and the use of compression, being that is handled on other threads, has altered the behavior of the shutdown process in some cases.

And before the users with connectivity problems, blame there connectivity issues on this, understand this has been an infrequent event, that at most occurred twice today on some servers, and the last observed time was a few days ago. And there is no connection between this, and market volatility. Some of the instances of this issue, occurred in the middle of the night.


----

Update 2020-10-08: Confirming that the issue has been resolved with no shutdowns of the real-time server processes after several days.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2020-10-08 19:51:50
[2020-10-08 10:45:23]
User657945 - Posts: 226
Hi I'm on Denali since 1 week and I noticed that the DOM on NQ is delayed compared to a local provider that for your info is not getting the data directly from CME but throught a partner (for wht I know should be JP) and it's not a question of milliseconds but is crearly visible by eye so is around 1 or 2 seconds. My chart is a 1 sec chart but it's the same on a huge volume chart (30k vol).I have installed 2177. Any suggestion? Is the compression already in the 2177?
thx
Date Time Of Last Edit: 2020-10-08 10:46:35
[2020-10-08 12:08:39]
User732341 - Posts: 4
hi - with the new compression on latest release, does it mean we can overcome the 10 millisecond minimum chart update at some point? I'd like to get the market data as hey come in my acsil code.
thanks
[2020-10-08 13:56:55]
User311344 - Posts: 136
Denali has 20ms minimal buffer threshold so theres no use to set chart update interval to less then 20 :-(

Interesting how it would work on direct DTC server connection. Probably same way.
Date Time Of Last Edit: 2020-10-08 13:58:27
[2020-10-08 15:20:43]
User657945 - Posts: 226
@user216714 I would be more than happy to have that ms but I can count 1-2 sometimes 3 before to get the dom updated and my latency on the network from itsaly to chicago is 115 to 120ms, it's not good (I can't get more also if I try to pay no physical optic line in my area) but it's not bad and in any case it does not justify the delay.
[2020-10-08 19:01:49]
Sierra Chart Engineering - Posts: 104368
In regards to post #37 do you notice that delay consistently or at certain times?

If you reconnect to the data feed does that make a difference?:
File Menu: Procedure to Reconnect to the Data and Trade Servers

Select Global Settings >> Data/Trade Service Settings >> SC Server Settings.

Make sure Use Real-time Data Compression is enabled.

Set Remote Buffer Delay Send Time in Milliseconds somewhere between 100 to 200 and see if that helps. The higher the number the more efficient the data transfer. It results in a higher compression ratio and fewer packets and therefore fewer return acknowledgment packets and less chance for packet retransmission which results in delays if the connection is not good.

Reconnect to the data feed after changes.

More information:
Prices / Data Falling Behind: 4.12 - Sierra Chart Exchange Data Feeds
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2020-10-08 20:04:08
[2020-10-08 22:37:22]
User657945 - Posts: 226
In regards to post #37 do you notice that delay consistently or at certain times? Yes it is always in delay of course the delay increase when the volume and volatility increase

Make sure Use Real-time Data Compression is enabled. It was enabled
I set to 100 ms and then reconnect it's a little beat better but I'm testting it right now 00.35 with very small volumes of contracts and Eurex flow on my chart at zero volumes since it's still closed, I will make a check tomorrow and let you know.
[2020-10-09 02:35:55]
Sierra Chart Engineering - Posts: 104368
This may be unrelated to network performance and connectivity, and instead related to performance issue on the Sierra Chart side. Refer to:
Overview of Trading: Sierra Chart Configuration for Most Low Response Time Trading

That would be more possible, the issue in your case. There is no need to post anything further here because there is nothing further we can advise, other than the information given.

The Denali data feed is a high-performance low latency data feed and if you have a good Internet connection you should not be experiencing lag, exceeding 100 to 200 ms or so.

If you are noticing a significant problem you have to start looking at your connection including whether there is any throttling on the ports being used, somewhere in the network path.

This table here for test results on a Mikrotik router:
https://mikrotik.com/product/RB2011UiAS-2HnD-IN#testresults

is a good demonstration of where larger packet sizes means more efficiency. Notice the third line (Routing  none (fast path)), with a packet size of 1518 bytes. The transfer speed is 1,481.6 megabits per second. But when the packet size goes down to 64 bytes it becomes 116.2 megabits per second. This is a huge difference.

So if you try to go too far to reduce latency, you create the opposite effect, and create an overload condition and therefore dramatically increasing your latency. Ultra low latency is not going to work for long-haul connections. It has to be a local area network connection.

You will also want to have a fast computer, and a fast network switch and router.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2020-10-09 02:39:14
[2020-10-09 03:25:13]
Sierra Chart Engineering - Posts: 104368
Not only do users need to consider the performance, of their computer, but you must also consider the performance of your network router. You need to have a high-performance router as well and understand its performance.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2020-10-20 17:59:12]
Sierra Chart Engineering - Posts: 104368
Regarding this released in version 2183:
Added support for higher compression level for real-time data compression. This is controlled through Global Settings >> Sierra Chart Server Settings. The setting is "High Compression"

We do not recommend using it. It appears to be causing a datastream corruption. Only use Standard Compression.


We will be looking into the problem.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2020-10-20 18:00:54

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account