Support Board
Date/Time: Wed, 10 Jun 2026 00:51:08 +0000
[Sticky] [Locked] - 2026-6-9: Teton Order Routing Connectivity Issue and CME Market Data Interruption 12:38 PM
View Count: 59
| [2026-06-09 19:32:12] |
| Sierra_Chart Engineering - Posts: 23970 |
|
Today around 12:38 US Eastern time, there was a problem with connectivity where some or all users would have lost a connection to the Teton order routing server and would have been connected through a backup connection. This logging below shows this. Line 12990: 2026-06-09 12:38:52.603 | Teton CME Routing | Connecting to the server futurestrading14.sierracharts.com. Port 11093
Line 13013: 2026-06-09 12:39:15.671 | Teton CME Routing | Connecting to the server futurestrading14.sierracharts.com. Port 11093 Line 13037: 2026-06-09 12:39:38.700 | Teton CME Routing | Connecting to the server futurestrading15.sierracharts.com. Port 11093 Line 13062: 2026-06-09 12:39:41.746 | Teton CME Routing | Connecting to the server futurestrading16.sierracharts.com. Port 11092 We want to explain this log above. What it shows is that there are two attempts to reconnect, to Teton order routing over our Zayo ISP, and then the third connection, is to our backup process for Teton order routing and the connection would have been denied there (This is a different server with a Lumen ISP connection). That is normal and expected. And finally the fourth connection, is going through one of our servers with a Lumen connection and it succeeds. That server with the Lumen connection then tunnels the connection over to the server, that is actually running the Teton order routing process over the internal LAN from our service provider, in the Aurora DC. This also affected market data being distributed from that same server. This is one of our main distribution servers. We have two others which were not affected. So not all users would have been affected. Only about half of the users receiving CME data. Our best analysis of the problem is that this looks like a problem with the ISP Zayo. That is the most likely explanation. Our other servers use Lumen and did not have any issue. None whatsoever. And yet load would have been the same. The server affected definitely has enormous processing capacity and the server was definitely not under a heavy load. All of the CME iLink connections for order routing were maintained. Within the server there was no excessive delays. Although we did see a delay of at times of several hundred milliseconds and during the incident of one second with market data processing. There was packet loss suffered on the server, but we think we know why that occurred and we actually developed a solution to this, over two months ago but we realize we did not yet released that. We are going to be releasing that change this evening at 5 PM. The server is equipped with a 10 Gb Internet connection. This is more than enough bandwidth and users are not connecting directly to it. So it is very unlikely this was a problem on the server side itself. After the fact it is really impossible for us to know what exactly went wrong other than that there was a connectivity issue, and the ISP is the primary component of that. All evidence indicates a problem with the ISP. If it is an ISP problem there is simply nothing we can do when issues like this occur. Simply there are going to be interruptions and then failovers. Most importantly, we are going to be doing some updates to address the packet loss issue. If this was not an issue with Zayo, why is it that the servers doing the same amount of data processing, and having the same amount of outbound network bandwidth usage, did not have any problems, and they are using a Lumen connection. Absolutely no problem at all. Not a single packet lost on the other two servers. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2026-06-09 20:20:55
|
| [2026-06-09 22:25:39] |
| Sierra_Chart Engineering - Posts: 23970 |
|
One issue that this problem causes, most likely at the OS level we would have to believe but not 100% certain, that is the entire cause, that there was significant CME multicast feed packet loss at the time. We have developed a Linux program for processing UDP multicast feeds, but we have not yet deployed that. This packet loss did not affect any historical data. And it would only have affected 50% of the users. And would have been corrected, once the secondary servers were connected to another server. This is something that would've helped with this issue because it will then isolate the multicast processing on its own Linux system and avoid any detrimental effects from external network connectivity causing to the multicast processing. It will have been completely unaffected. We are going to get that done as soon as possible. We have also released the changes on the affected server, this evening, to resolve the packet loss issue, to the extent it was being caused by restarting the snapshot feeds. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, use the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2026-06-09 22:27:05
|
To post a message in this thread, you need to log in with your Sierra Chart account: