Home >> Support Board

Support Board

Date/Time: Thu, 02 May 2024 22:22:23 +0000

How to Post a Message for Support

Short Interruption with TT Routing 2020-05-27 / TT Account Balance/Credit Data

View Count: 1280

[2020-05-28 02:17:17] #1 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
Around 11 AM US Eastern time today, 2020-05-27, we had a 2.5 minute interruption with the Sierra Chart/ TT order routing service due to an apparent issue on the TT server. 2 of 6 servers were affected. Details are below: Our tentative conclusion at this point, is that this was an issue with the TT FIX server. However, that still is yet to be determined. Most importantly, you always have the web-based trading panel available that you can use and can easily switch to any server: https://www.sierrachart.com/index.php?page=doc/SierraChartOrderRoutingServiceWithData.php#WebBasedTradingPanel There are several things we want to look into related to this and we have some logs from users. We want to make sure that Sierra Chart is automatically switching to the next server if it is unable to submit a new order, modify or cancel an order, due to the remote server not having a connection to the exchange. The order routing service does support this. And that there is no problem with passing through the server order ID immediately for a new order (Extremely unlikely there is a problem with this). So we are doing a thorough review of everything related to this incident. This is the log of the event from our Aurora order routing server: The data feed was lost. There was no activity for at least 2.0 minutes. \| 2020-05-27 11:04:41.203 Connection to the external service has been lost. \| 2020-05-27 11:04:41.203 TT FIX Trading: DisconnectFIX called. \| 2020-05-27 11:04:41.219 TT FIX Trading: Sending a Logout message. \| 2020-05-27 11:04:41.219 TT FIX Trading \| Connecting to 192.81.51.68. Port 11702 \| 2020-05-27 11:04:42.241 TT FIX Trading socket (81) \| Creating socket. Using TLS 1.2. \| 2020-05-27 11:04:42.241 TT FIX Trading socket (81) \| Connecting to IP: 192.81.51.68. \| 2020-05-27 11:04:42.241 TT FIX Trading: Connected to FIX server. \| 2020-05-27 11:04:42.243 TT FIX Trading \| Sending the Logon message. LocalCompID=LIVE_SIERRA_2 \| 2020-05-27 11:04:42.444 TT FIX Trading: Connected. \| 2020-05-27 11:04:43.864 This is the communication and information from our infrastructure provider regarding the crossconnect we have with TT: I just looked at the interfaces and both interfaces are clean with no drops. So this tells me that this was a software issue rather then a network/hardware issue. You should really just write directly to the CME and bypass TT for order entry! Here is the configuration of the ports: show configuration interfaces ge-2/1/4 description "XConnect to TT - Cable ID: 129113."; speed 1g; link-mode full-duplex; ether-options { no-loopback; auto-negotiation; no-flow-control; } unit 0 { family ethernet-switching { vlan { members service-121-tt-primary; show configuration interfaces ge-14/1/4 description "XConnect to TT - Cable ID: 129115."; speed 1g; link-mode full-duplex; ether-options { no-loopback; auto-negotiation; no-flow-control; } unit 0 { family ethernet-switching { vlan { members service-122-tt-backup; Here are the port stats: show interfaces ge-2/1/4 Physical interface: ge-2/1/4, Enabled, Physical link is Up Interface index: 944, SNMP ifIndex: 1541 Description: XConnect to TT - Cable ID: 129113. Link-level type: Ethernet, MTU: 1514, MRU: 0, LAN-PHY mode, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Enabled, Remote fault: Online, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Current address: 10:0e:7e:c5:f9:f3, Hardware address: 10:0e:7e:c5:f9:f3 Last flapped : 2020-04-16 16:45:37 CDT (5w6d 03:44 ago) Input rate : 656 bps (1 pps) Output rate : 696 bps (0 pps) Active alarms : None Active defects : None Interface transmit statistics: Disabled Logical interface ge-2/1/4.0 (Index 1861) (SNMP ifIndex 3370) Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge Input packets : 8375751 Output packets: 7203253 Protocol eth-switch, MTU: 1514 show interfaces ge-14/1/4 Physical interface: ge-14/1/4, Enabled, Physical link is Up Interface index: 934, SNMP ifIndex: 1666 Description: XConnect to TT - Cable ID: 129115. Link-level type: Ethernet, MTU: 1514, MRU: 0, LAN-PHY mode, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Enabled, Remote fault: Online, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x4000 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Current address: 28:8a:1c:62:b1:f3, Hardware address: 28:8a:1c:62:b1:f3 Last flapped : 2020-04-05 13:14:09 CDT (7w3d 07:16 ago) Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) Active alarms : None Active defects : None Interface transmit statistics: Disabled Logical interface ge-14/1/4.0 (Index 1860) (SNMP ifIndex 3369) Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge Input packets : 3252278 Output packets: 3428760 Protocol eth-switch, MTU: 1514 Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-05-28 02:19:54

[2020-05-28 02:31:19] #2 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
Also regarding the slow updates with TT Account balance/credit utilization data, we have implemented faster updates on Server 1 and Server 4. We have not yet deployed this to Server 2. Since most users are on server 2, we want to make certain everything is fine for at least a week with a smaller number of users although this has been well tested. But the ultimate solution to this, is going to be when we finally release towards the end of the year, hopefully, our own direct routing , Position tracking and account balance tracking independent of TT. In this case all account and margin balance data will be instantly updating. You can see how this works just by using our simulated trading services: Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing

[2020-05-28 02:46:52] #3 [Go To First Post]
ondafringe - Posts: 248
Appreciate the explanation. No doubt you folks will get it figured out. As to my three trades that were affected, it appeared to me that the problem persisted for at least half an hour. And I was wondering why the backup server didn't kick in automatically, so happy to hear you are looking into that, as well. Thanks, George

[2020-05-28 02:49:54] #4 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
We think in regards to post #3 the explanation is that those orders were never received by TT and they just could not be canceled because they were never acknowledged. And with TT we do not time orders out due to past problems with that. So we just have to clear them at the end of the day. Do you still see orders which you cannot cancel that are in a pending state? Although with the automatic reconnect to the TT server, and the refreshing of the order list, those orders should have been cleared. So we will look into your case more closely. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-05-28 02:51:55

[2020-05-28 08:48:25] #5 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
Ok this is what we have identified: Beyond the 2.5 minute interruption with connectivity to TT and we are waiting for response from TT on that, the remaining issue, was related to server-side bracket orders transmitted at that time. Since these are now required, we have identified a couple of minor issues related to these orders: 1. In the case of the Sierra Chart implementation of the DTC protocol both by the client and the server, when the client sends a server-side bracket order to the server and the server expressly rejects it, the children were getting rejected but the parent order was not. The parent order was just simply ignored. None of these orders were transmitted to the exchange. The side effect of this, is that a user would see the parent order left over from a rejected bracket order but this order was never sent to the exchange and was never actually working. It could have been easily cleared with this command: Trade Menu: Refresh Trade Data From Service (Trade menu) This is why we recommended just to reconnect to the data feed which effectively does the same thing. However, that order should have timed-out and in testing now we see that it is timing out. However, if the user kept trying to cancel it, it might not have. In any case there was never any harm done. This was never a working order. 2. The other problem was that Sierra Chart was not reconnecting to a backup server, if the server being connected to currently, rejects a new order, or an order modification or cancellation because it has no connection to the exchange server. This does work properly but was not working properly in the case of a rejected bracket order. This is now resolved. So this reconnect will happen. 3. Unrelated: There was never any inaccurate position reporting. That always remains accurate. If your broker was telling you something different about your positions, it was not accurate. The Sierra Chart order routing service, has consistently been reporting accurate positions with no known issue. ---- The items described above are not specific to the Sierra Chart order routing service, but apply in general, to order processing with the DTC protocol which would include LMAX, and also when using a sub instance of Sierra Chart. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-05-28 19:13:04

[2020-05-28 09:19:08] #6 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
The other thing is we dropped the timeout for loss of activity on the TT FIX connection from 120 seconds down to 70 seconds. This is not yet released. However, this will get released to Server 4 as well as all changes described above during this early morning. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-05-28 09:19:23

[2020-05-28 11:59:37] #7 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
The version with the updates described in post #5, is 2113 and this is the latest prerelease. We did live testing with all of the changes on Server 4, with MES. No problems found. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing

[2020-05-28 12:35:12] #8 [Go To First Post]
ondafringe - Posts: 248
Since that issue, I have not placed another order. I'll let you know if I have anymore problems, and I'll keep the disconnect/reconnect to data feed in mind, as well. Thanks, George

[2020-05-29 05:02:18] #9 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
we do have an answer from TT about the interruption: We have checked this and found it that this was caused by a crash of the order routing node the session pointed to. The connection was then failed over to another server and services restored within about 1-2 minutes. We have escalated this to check further on the crash. Please let us know if there are any further requests. It came shortly after we contacted TT, but we did not see it because we did not get an email notice from their ticketing system about this. Now there are a few things we want to say here. It is not good, that they are experiencing a "crash". Our own order routing technology we are developing to the CME which simply utilizes Sierra Chart itself, is going to be rocksolid and super stable and fully redundant. We have not experienced any "crashes" with our CME market data feed processing, or even the Barchart open feed processing, or the TT order routing relating to all of the software and hardware technology on our side. This is all super solid and most importantly simplified technology. That is the key: simplification, stability, and redundancy. And our order routing has ultra low latency at below 50 µs. You can see above, that our infrastructure provider recommends that we bypass TT (We will continue to support TT though just like we continued to offer Interactive Brokers technical integration). We are working on that. But this is a step-by-step process in order to develop a proper solution that clearing firms can utilize and trust and rely on. Everything we need to do to accomplish this, will come this year. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-05-29 05:06:58

[2020-05-29 13:02:58] #10 [Go To First Post]
patrader - Posts: 92
At the present moment for trading the CME markets, what would server would Sierra recommend setting as the primary server? And what servers for Backup 1 and Backup 2?

[2020-05-29 18:47:37] #11 [Go To First Post]
Sierra Chart Engineering - Posts: 104368
Usually this combination: Server 2, Server 1 and Server 4. You could use Server 1, Server 2, and Server 4. Now that there is the crossconnect we have, to the TT server, any of these servers will be fine to use. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, change to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2020-05-30 02:45:48

To post a message in this thread, you need to log in with your Sierra Chart account: