Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Thu, 25 Apr 2024 21:36:05 +0000
[Locked] - TT Order Routing issue 2019-08-02
View Count: 1575
| [2019-08-03 11:58:04] |
| |
|
On 2019-08-02 from 11:37 to 14:11 US Eastern time, the following issues were occurring with the Sierra Chart TT based order routing service: After about a minute of entry, open orders were put into a state which indicated they were no longer working even though they were actually still working. Any pending child orders either on the client side in your local Sierra Chart or on the server side, which were not yet sent to TT/exchange, would have been canceled. If they were sent to TT and therefore to the exchange, they would still be working but the OCO functionality would not have been working. Order fills for orders which were in an inactive state in some cases were not getting logged. At all times, orders were routed very efficiently through to TT. Positions were properly reported. Market orders were not affected. The situation was analyzed, a code change was made, and therefore it became corrected. We could have reestablished the connection to TT FIX from our Sierra Chart server programs, and that would have shorten the duration of this problem but it took us time to understand it and as a general rule we do not interrupt the connection during live trading. But that would have been the best thing for us to do to get the problem cleared in a shorter amount of time. The reason for the problem is as follows: TT reported some orders as being in a pending state (Like pending cancel) and those orders were not coming out of the pending state. Sierra Chart will query the status of those orders once a minute because they appear like orders which are nonresponsive. The result of this query was interpreted by our execution report handling, as being part of a full mass order status request. Part of this issue is because there is no way with TT FIX to tag different order status requests with unique identifiers. Since there was only a single order coming back, the other working orders were put into an inactive state because they were not reported and believed to be not working. Update: In regards to the above, after further code review the execution report related to getting the status of a single order was very much unexpected by us based on the TT documentation relevant to this. We did not believe it would result in a single order coming back looking like part of a mass order status request resulting in other working orders being marked as inactive. We have now requested TT to add the unique identifier field to a order status request. Whether they add it or not, it does not matter at this point because the problem has been resolved regardless. TT reported this to us on Sunday (2019-08-04) regarding the incident on Friday (2019-08-02): We experienced an issue with one of our order connectors that resulted in disruptions in order routing to the CME. Once we identified the issue, we failed over to our back up order connector where users were able to place and cancel orders.
This was an additional problem and actually what was triggering the problem on the Sierra Chart side.We are investigating how this order connector got into this state, and once we have more information, we will update you. Now what continued to happen is that TT continued to show certain orders from some of you, as being in a pending state but that status was not correct. The text from TT on those orders in the execution report indicated "Pending order action exceeded timeout interval". This then caused another order status request, and the cycle repeated where open orders were marked as inactive because only one order came back. TT continued to show some orders as being in a pending state when they were no longer pending or had completely finished. We have seen a similar situation where an order status is not being correctly reported and the order being indicated as pending, when that is not the case. Reestablishing the session, and doing a full mass order status request corrected the problem. TT has acknowledged this, and they said it has been sent to their development team. This was before Friday's incident. The previous issue did not cause any problem like we had on Friday. It was a scenario which we did not foresee. We have once again reported this problem to TT. It is something they need to resolve, but aside from when they resolve it, it will not cause the problems described at the beginning of this post any longer. We do not want to fault TT here aside from the order routing issue they had to the CME, but if TT was correctly reporting the order status through, the problems described at the beginning of this post, would have been greatly limited or never occurred. We want to stress here that these problems will *never* happen again. It has been solved and solved in a very durable way. It will never happen again because of several durable changes which have been made, and of the fact that we no longer will remove any order which is believed to not be working and instead will leave that for the user to decide. If you see an order that you believe should not be working and you cancel it, if it really was not working, it will just then be put into an "error" state. This change has already been deployed. You can observe this, by going to one of your canceled or completed orders in the Trade >> Trade Orders and Positions >> Orders tab and then selecting it and then pressing Cancel and confirming. You will notice the status goes to "Error" and if you look at the Order Action Source on it, you will see text that says "order not in book". A scenario where you could see an open order which is no longer working is if the status of that order was completed but was not communicated from TT FIX to our server and therefore it just was a lingering order in an open state. This however would be a rare occurrence. If you then cancel that order it will then just simply be marked with an "Error" state. And once again when we say this problem will never happen again, it will not and we say that with total confidence due to the changes made. The issue has been durably solved by us at this time and we have complete confidence in the service. Overall when it comes to connectivity and other functionality we have a lot more confidence in this service than we do with the others Rithmic, CQG or CTS. The situation that has arisen is not because the service is relatively new. This type of order routing service is not anything new for us. This is something that we have done for at least four years now with other services. And support for TT is not new either. It is a solid and stable service. There was a set of circumstances on the TT side and also a handling issue on our side with order status response processing, which in combination created the issue. It is now resolved. We want this to be a service where you can trust and rely on it for your orders to be routed with very low latency and for it to be trouble-free. The issues which occurred did not meet this expectation and we have remedied that. We want this to be a service that provides a reliable gateway to the exchanges for you, which does not require emergency intervention. It already has a very redundant design. As a result of this incident, we are issuing a courtesy credit of 4 months of service package 5 as a credit to your Sierra Chart account, if you were trading on Friday, August 2, 2019 and using this service. We had 16 users trading on August 2, 2019 using the TT order routing service. We think only a few of those users were affected. RESTORING MISSING FILLS: The information below applies only to when using the Sierra Chart order routing service. During the time of this incident, the logging of some fills was not occurring. Market orders should not have been affected. We have now restored those, but they do not contain the Position quantity. But nevertheless they have been restored. To download the missing fills within Sierra Chart, follow the below instructions: Go to Trade >> Trade Activity Log. Select the Trade Activity tab with the menu command at the top. Select Fills in the top left list box. Set the Trade Account from the list box of accounts at the top to your trading account. In the list box of symbols select All Symbols (Non-Sim) In the list box of Notes select All Notes. Set the Date Range to Display >> From to 2019-08-02 11:30 (this is US Eastern time). Adjust as necessary for your time zone. Select all of the fills from the top to the very end by selecting the very first fill at the top, and then holding the shift key on your keyboard and then selecting the very last fill at the end of the list. Select Edit >>Delete Trade Activity Entry. Confirm deletion. There is always a backup made of the affected files in the Backups folder. Then follow the instructions to reconnect to the data feed: File Menu: Procedure to Reconnect to the Data and Trade Servers ---- One of our users said they saw duplicate fills in the Trade Activity Log. If you do, please start a separate support board post about that and we will look into it. We may just have to clean those up manually. Not sure why that would have happened but we will understand it once we can look at it. Update: We understand the reason for this issue and it is resolved. ADDITIONAL NOTES: One thing we want to point out, is that if you submitted order, from Sierra Chart and it disappears no matter what trading service you are using, you should not assume that the order is somehow canceled or not working. You need to contact your broker and stop trading. Here is the documented procedure to follow: Overview of Trading: Procedure to Handle Unexpected Trading Issue This is not a Sierra Chart specific procedure. It is just a common sense procedure. Never assume that an order you submitted, is not actually working if you no longer see it if you did not expressly cancel it, or if the price action did not cause it to fill. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2019-08-26 03:11:52
|
To post a message in this thread, you need to log in with your Sierra Chart account: