Login Page - Create Account

Support Board


Date/Time: Sun, 19 May 2024 15:10:21 +0000



[User Discussion] - Problem with market depth desapearing

View Count: 3958

[2014-04-03 15:15:23]
User30668 - Posts: 108
SC Engineering,

I have been experiencing a problem with market depth, when using IB data feed and live trading account.


The problem is that SC is very frequently and momentarily missing some of the levels of the market depth. They then come back and disappear again, flashing back and forth. It happens in both Trade Dom and Market Depth window.



At first I obviously thought that the problem was coming from IB's data feed but after some tests that doesn't seem to be the origin of the problem.



Thus, I am going to try and explain here as thoroughly as possible the conclusion of my testing:


- The instrument I notice the problem more is Euro Futures;


- The problem is more noticeable when there are some trade flow (not need much);


- It just happens with IB's live account and doesn't happen with IB’s paper account – this is strange for me as the data feed is supposedly the same;


- And, as a final test, with all connected at the same time to IB's data feed, I tested SC Trade Dom and SC Market Depth window, side by side with TWS Dom (Book Trader) and Ninja Trader Dom – Both Tws Book Trader and NT Dom are showing the market depth normally while SC is showing the described problem.



This is an important issue for me because I need to use IB’s T&S data in my charts and need as well to have the Trade Dom in the same instance to be able to see the orders in those charts.



Would you please take a look at this issue?

Thanks.


Date Time Of Last Edit: 2014-04-03 15:17:04
imageMarket Depth Problem 1.PNG / V - Attached On 2014-04-03 15:14:31 UTC - Size: 22.42 KB - 594 views
imageMarket Depth Problem 2.PNG / V - Attached On 2014-04-03 15:14:43 UTC - Size: 22.25 KB - 519 views
imageMarket Depth Problem 3.PNG / V - Attached On 2014-04-03 15:15:06 UTC - Size: 26.98 KB - 590 views
imageMarket Depth Problem 4.PNG / V - Attached On 2014-04-03 15:15:19 UTC - Size: 25.6 KB - 542 views
[2014-04-03 17:16:01]
Sierra Chart Engineering - Posts: 104368
We see your Sierra Chart account has activated the Sierra Chart Real-time and Historical Exchange Data Feed. Are you using this data feed?
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2014-04-03 19:14:44]
User30668 - Posts: 108
Yes, I am using both SC data feed and IB data feed.


Thus, I have SC data feed for the charting but because of a custom study that I have and use for my strategies, I need to have this SC instance using IB's Time&Sales data (non chart data).


As I need to have the Trade Dom in this instance too in order to see my orders in the charts, it is unfortunately using IB's market depth data.


I wish I could use SC market depth in that same instance just for the Trade Dom but it is not possible.


What I am doing now is running another SC instance just for the Trade Dom (with market depth from SC data feed) but this way I can't see my orders in the charts of the main instance. And it is important to see those orders in the charts...


I hope I was clear enough in my explanation.


Would you please see what you can do about that market depth problem?

Thanks.
Date Time Of Last Edit: 2014-04-03 19:15:47
[2014-04-03 20:57:57]
Sierra Chart Engineering - Posts: 104368
In the copy of Sierra Chart you are using the Sierra Chart Real-time Data Feed, do you have any charts open which use data from another instance of Sierra Chart? If so, close those charts and reconnect to the data feed and see if that resolves the problem.

We think this has something to do with the data sharing and market depth data being received from two sources.

Otherwise, or should not be a problem like this with the Sierra Chart data feed market depth.

And the handling of the IB market depth, is as reliable as we can make it. We do not know what more could be done with that. Trying to workaround an IB market depth problem is definitely something we will not spend any time with. The underlying problem is going to be with IB if you are having a problem with the IB depth, despite what you may see in another program. There may be something we are doing to work around some other IB market depth problem which is introducing this other issue.

This is why we offer our own data feed. It is far more productive for us to help you rely solely on the Sierra Chart market depth data and that is where we can help you with this.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-04-06 01:03:23
[2014-04-04 16:39:26]
User30668 - Posts: 108
In the copy of Sierra Chart you are using the Sierra Chart Real-time Data Feed, do you have any charts open which use data from another instance of Sierra Chart? If so, close those charts and reconnect to the data feed and see if that resolve the problem.

We think this has something to do with the data sharing and market depth data being received from two sources.

In order to isolate the problem I opened a separate Trade Dom in an instance solely with IB data feed and it shows the exact same problem, so the problem is not being caused because of the data sharing.

I completely understand that working with IB is not an easy task; nevertheless this is an issue that should be addressed as it prevents users from trading an important instrument (Euro Futures) if they don't have SC data feed or another external data feed.

I have spent a few hours today trying to figure out a chartbooks setup in order to be able to get SC market depth in my Trade Dom and keep IB's market depth in some charts where I need it. Unfortunately I couldn't find a way to set it up because the option "Use Non-Chart Data From Remote Instances" is directed for all the charts and Trade Doms in the chartbook.

Sorry to insist with this issue but if it is too difficult to fix this market depth problem, would you please consider creating an option for the Trade Dom to allow the user to chose which market depth to use in it, not depending on the option "Use Non-Chart Data From Remote Instances" being checked or not?

Please don't get this in the wrong way - I have idea how difficult that would be.

Thanks.





[2014-04-06 01:05:28]
Sierra Chart Engineering - Posts: 104368

I completely understand that working with IB is not an easy task; nevertheless this is an issue that should be addressed as it prevents users from trading an important instrument (Euro Futures) if they don't have SC data feed or another external data feed.
We had a look at how the IB market depth data is processed and we think the issue has to do with special handling to work around another IB problem. In that code there is an inexact comparison of floating-point values. This will be solved in the next release and we think there is a good chance this will take care of the problem.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2014-04-07 16:54:14]
User30668 - Posts: 108
1117 Release Date: 2014-04-07

Solved an IB market depth problem which occurs when prices have a large number of decimal places.

I am very sorry to bring bad news but this market depth problem is still occurring.

I have installed version 1117 today and despite today's very slow price action the issue showed frequently. It is however much easier to see the problem when the market flow is faster.

Please try to fix it when possible, ok?

Thanks.
imageMarket Depth Problem 5.PNG / V - Attached On 2014-04-07 16:54:13 UTC - Size: 8.9 KB - 490 views
[2014-04-07 17:40:46]
Sierra Chart Engineering - Posts: 104368
We are not able to test this because we do not get any market depth data on the test account we use.

The problem most likely is occurring because of the clearing of Market Depth prices which are out of order to solve a problem we have seen with that in the past with Interactive Brokers. We have removed this automatic clearing of out of order depth levels.

So this should resolve the problem, but if there is any problem with out of order market depth prices, it is not our issue and we will spend no further time on this.

This will be out in the next release.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2014-04-10 19:42:31]
User30668 - Posts: 108
SC Engineering,

Just a quick update.

I upgraded today to pre-release version 1118.

SC is now showing IB market depth normally (as good as IB's data feed can be...). The issue is fixed.

Thank you.
Date Time Of Last Edit: 2014-04-10 19:43:13
[2014-04-15 21:04:05]
Sierra Chart Engineering - Posts: 104368
As we suspected, this is the problem that has now been created by the change that we did to solve your issue:
Messy TradeDOM levels

So your issue is fixed and now there is a new problem.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2014-04-16 11:07:51]
User30668 - Posts: 108
Post #6:

We had a look at how the IB market depth data is processed and we think the issue has to do with special handling to work around another IB problem.

Post #8:

The problem most likely is occurring because of the clearing of Market Depth prices which are out of order to solve a problem we have seen with that in the past with Interactive Brokers.

According to what you have written before, the problem I have reported in this thread was being caused by a "work around" you had implemented in order to fix another problem.

Now you are saying that correcting the problem I reported is causing the original problem to show again...

I am very sorry, I am confused now - are you suggesting that this is my fault? Should I be unable to trade, keep quite and stay with a problem that was being caused by an implemented "turn around" to fix another problem? If that's the case you are not being fair.

Again, I am very sorry that I really needed to report the problem I was having but I had no other option. I can just remark my appreciation to you again for taking off that "turn around" that was causing the problem and resolving it.

I understand that your work might get frustrating sometimes due to third parties problems that in the end affect Sierra Chart software. Believe me, I try to just report any problems or issues when I really need it because I am aware that you are overwhelmed with work.

I am trustful that you will find the best way to resolve that original problem without the "collateral effects" that caused that secondary problem which I had to report and you fixed.

Kind regards.
[2014-04-16 16:29:49]
Sierra Chart Engineering - Posts: 104368

According to what you have written before, the problem I have reported in this thread was being caused by a "work around" you had implemented in order to fix another problem.


Now you are saying that correcting the problem I reported is causing the original problem to show again...
Yes.

are you suggesting that this is my fault?
No. We are only pointing out the difficulty with working with the Interactive Brokers system. We have worked on market depth for IB numerous times over the years solving different issues. And there are still issues. It is hard to work around them all. It is not possible to have perfect processing because the depth data from IB is flawed.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-08-13 22:14:07
[2014-08-14 22:44:47]
Sierra Chart Engineering - Posts: 104368


SC is now showing IB market depth normally (as good as IB's data feed can be...). The issue is fixed.
This "fix" is going to be reverted back to the way it was previously. The effects of it are even worse than the problem it is solving.

For more information refer to this thread:
DOM with IB

There is another possibility and we do not have the ability to test this, but we can process the IB Market depth Insert, Update and Delete messages using prices rather than positions. We suspect this is not going to work but we can implement it and you can let us know if it does work.


Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-08-14 22:45:49
[2014-08-15 10:41:56]
User30668 - Posts: 108
This "fix" is going to be reverted back to the way it was previously.
Well, after reading this message it only occurs to me to say that I am very sorry and sad with your decision as it will bring the problem back to me (and believe me, it is a serious problem as it will prevent me to use SC Dom to trade).

Apparently just because of someone that use some bad manners talking to you, you choose to please him and so disregard the other users problems.


The effects of it are even worse than the problem it is solving.
Regarding this comment, I must remind you that MY REPORTED ISSUE WAS A CONSEQUENCE of you trying to resolve another previous problem and not the other way around, as you mentioned yourself in post #6, being your words the following:
We had a look at how the IB market depth data is processed and we think the issue has to do with special handling to work around another IB problem.



Please don't get me wrong, I am not here to dispute or argue with you. I want you to know that I completely understand that you have a lot of work in your hands and this particular problem is burdening you even more.


However, please don't simply disregard the problem as it is of utmost importance for me and for other users in a similar situation. Please try to find a solution for it.




There is another possibility and we do not have the ability to test this, but we can process the IB Market depth Insert, Update and Delete messages using prices rather than positions. We suspect this is not going to work but we can implement it and you can let us know if it does work.

Please let me know when you implement these changes. I will give you my feed back asap.


As it is of your knowledge, I have an account with IB. Therefore, in order to help, I will be available to do any testing that you need, regarding this issue or any other.


I sincerely hope that you will somehow be able to find a good solution for the problem.

Thank you.

[2014-08-15 11:32:47]
Sierra Chart Engineering - Posts: 104368
Thinking about this some more, really there is an easy solution to this that we did not previously think about.

Basically the change that we made is restored the zeroing out of price levels in the market depth which are out of order. So we can make this an option. The user can set the option whichever way solves the problem.

This will be out in the next release in a couple of days.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2014-08-19 07:05:06]
Sierra Chart Engineering - Posts: 104368
In version 1175 and higher there is the following IB specific option you will want to set to False:

Global Settings >> Data/Trade Service Settings >> Data and Other Settings >> Clear Out Of Order Market Depth Data.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2014-08-19 07:06:15
[2014-08-19 15:08:54]
User30668 - Posts: 108
Sierra Chart Engineering,

Just a quick follow up.


As per your instructions, I have installed today Pre-Release 1175 and set the option "Clear Out Of Order Market Depth Data" to False.


As expected the Market Depth is showing normally, as it was previously. I hope it works out well for the other problematic situations too.


I believe that this implementation will save you and IB / SC users from a lot of trouble in the future.


Thank you for your effort and for creating this option.
Date Time Of Last Edit: 2014-08-19 15:09:54
[2015-03-30 20:12:27]
i960 - Posts: 360
This is a Sierra Chart problem and it needs to be fixed.

Clear Out Of Order Market Depth Data: TRUE (the default) will *not* work with high volume contracts like NQ, ES, etc. It'll just barely work with CL as it never rises above 300 contracts typically.
Clear Out Of Order Market Depth Data: FALSE (the "workarond") will *not* work in any stable fashion without totals jumping around to columns they shouldn't even occupy.

Obvious bug here on the Sierra side. I have recorded *both* situations happening, and alongside that 3 separate vendor's DOMs doing the absolute right thing and Sierra Chart's DOM display totally crapping out the entire time. A Jigsaw (through Ninjatrader), Buttontrader, and Booktrader DOM being straight up solid and SC being straight up not solid absolutely does not mean this is an IB problem. It's how SC is handling the IB data combined with some kind of obviously latent bug on the SC side.

Once again: 3 separate non-SC DOMs working completely fine. I'd totally use the workaround if it even worked reliably - but it doesn't work; it's a totally different bug when using the False flag.

Documented video evidence of both scenarios:

Clear Out of Order Market Depth Data: TRUE: https://www.youtube.com/watch?v=VU00Thmmzms
Clear Out of Order Market Depth Data: FALSE: https://www.youtube.com/watch?v=ACr8gZ2EZr0

Now tell me, how in the above situation anyone is supposed to work with that? Trade low-volume contracts only? Don't trade ES or NQ? Look, if the situation is happening as the volume of a contract increases it's absolutely screaming race condition or other similar bug on the SC side.
[2015-03-30 21:13:30]
Sierra Chart Engineering - Posts: 104368
The responsibility to solve this problem is on the Interactive Brokers side. There is no doubt about that at all.

There is definitely nothing wrong with the processing of the depth according to how Interactive Brokers is providing it. This we are 110% certain of. And you do not have to believe us. This is fine.

The only other possibility is rather than working with Level indexes for Insert, Update and Delete operations, we work just with the price values but not sure those are provided for delete operations and we are just not sure how that would work altogether. It really is hard to say without actually putting it out in the field and seeing what the user feedback is since we do not know the order of the operations and the logic used by TWS.

In any case, why is it that Interactive Brokers is not properly providing the depth data through to begin with.


Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2015-03-30 21:23:11
[2015-03-30 21:26:34]
i960 - Posts: 360
What are you talking about? There most definitely IS doubt. There are 3 different vendor DOMs working perfectly well while SC's DOM is not working reliably at all with the exact same symbols/datafeed. On top of that, I cannot even use the workaround because that doesn't work properly either.

Please explain to me how this is an IB problem when everyone else's DOM is working exactly as it should be. You cannot have me get involved in a "one vendor points the finger at the other" game here. Your users are reporting known and documented issues whilst providing direct evidence of other software not having the same issues at the same time.

Seriously, are you trying to tell me that even in the workaround case (false) that totals drawing on top of other DOM levels is an IB issue? That's straight up ridiculous. UI/presentation layer has nothing to do with the datafeed layer (and we're still only talking about the workaround here!).

How much more proof do you actually need here? Accept that you have a bug here and drop your prejudices with IB in this specific case so we can actually use the DOM reliably.
[2015-03-30 22:20:49]
Sierra Chart Engineering - Posts: 104368
Post #19 has been updated. You cannot have any definite knowledge of the problem without looking at the Sierra Chart code which is actually quite simple.

Please explain to me how this is an IB problem when everyone else's DOM is working exactly as it should be.
This is a very vague and indirect, and impossible to directly defend against. What counts is the data being transmitted by TWS, and the logical processing of the data according to the TWS documentation.

Seriously, are you trying to tell me that even in the workaround case (false) that totals drawing on top of other DOM levels is an IB issue?
Yes of course. And we will say it again, the problem is on the Interactive Brokers side. It is caused by out of order data because of faulty transmission of market depth data from TWS.

If there is a problem/bug on our side, do you see it below. This is how the market depth with Interactive Brokers is handled when not clearing out of order prices:

UpdateDOMByCommand(BuySellEnum Side, int Level, int Operation, float Price, t_Volume Size, bool ClearOutOfOrderPrices)
{
  if (Level < 0 || Level >= MAX_NUM_DOM_LEVELS)
  {
    _ASSERT(0);
    return;
  }
  
  s_SCDomStructure* DomToUpdate;
  
  if (Side == BSE_BUY)
    DomToUpdate = m_BasicData.BidDOM;
  else if (Side == BSE_SELL)
    DomToUpdate = m_BasicData.AskDOM;
  else
    return;
  
  switch (Operation)
  {
  case INSERT_DOM:
    {
      s_SCDomStructure TempDomStructure = DomToUpdate[Level];
      for (int UpdateLevel = Level; UpdateLevel < MAX_NUM_DOM_LEVELS - 1;
        UpdateLevel++)
      {
        if (TempDomStructure.Price == 0)
          break; // no more DOM Levels
        
        Swap(DomToUpdate[UpdateLevel + 1], TempDomStructure);
      }
      
      DomToUpdate[Level].Price = Price;
      DomToUpdate[Level].Volume = Size;
    }
    break;
    
  case UPDATE_DOM:
    {
      DomToUpdate[Level].Price = Price;
      DomToUpdate[Level].Volume = Size;
    }
    break;
    
  case DELETE_DOM:
    {
      // Clear the single level
      memset(&DomToUpdate[Level], 0, sizeof(s_SCDomStructure));
      
      for (int UpdateLevel = Level; UpdateLevel < MAX_NUM_DOM_LEVELS - 1; ++UpdateLevel)
      {
        DomToUpdate[UpdateLevel] = DomToUpdate[UpdateLevel+1];
        //
        //if (DomToUpdate[UpdateLevel + 1].Price == 0)
        //  break; // no more DOM Levels
      }
      
      //if (DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Price != 0)
      {
        DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Price = 0;
        DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Volume = 0;
      }
    }
    break;
    }
}

Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2015-03-30 22:22:58
[2015-03-31 02:16:29]
i960 - Posts: 360
Yes of course. And we will say it again, the problem is on the Interactive Brokers side. It is caused by out of order data because of faulty transmission of market depth data from TWS.

Absolutely not. Anyone writing serious event driven code knows order is *never* guaranteed. It's a given by definition of standard concepts of asynchronous programming. Your code is not resilient against inserts, updates, and deletes arriving out of order while the API makes no guarantee of such a thing (nor should it). There's absolutely zero reason to assume that all inserts happen first or all updates happen first. In correct code they can arrive in any order and the final result should be the same always - just the same as if I add 2+3-7+4 the result is the exact same as -7+4+3+2. Order is not guaranteed with event driven architectures!

This code has potential issues, you should be ready to accept that.

Let's use a hypothetical value for inserting at a level of 2 with size of 114.


case INSERT_DOM:
{
s_SCDomStructure TempDomStructure = DomToUpdate[Level];

The above is a copy of DomToUpdate[2], which is fine, because it's potentially going to be used for a Swap().


for (int UpdateLevel = Level; UpdateLevel < MAX_NUM_DOM_LEVELS - 1;
UpdateLevel++)
{
if (TempDomStructure.Price == 0)
break; // no more DOM Levels

Swap(DomToUpdate[UpdateLevel + 1], TempDomStructure);
}

DomToUpdate[Level].Price = Price;
DomToUpdate[Level].Volume = Size;
}

break;

The above "0 as a sentinel for nothing left" conditional is completely suspect. It assumes that everything always arrives in a specific order of completely non-zero updates for every single level beforehand and relies on a sentinel value that isn't even guaranteed not to be sent by the datafeed, and worse, terminates the processing of the remaining levels being moved if inserts arrive out of order before a full update of all levels has occurred (it's entirely possible partial updates could occur followed by inserts) or an explicit value of 0 has already been updated from the feed. It's particularly prone as I bet the structure is being initiated with zeros beforehand. The following will completely break:

0: 123
1: 89
2: 77
3: 0
4: 55

Request to insert at level 2, with value of 114 will result in this afterward:

0: 123
1: 89
2: 114
3: 77
4: 55

Level 3 got shifted with the old value of level 2 correctly, but the old value of 0 for level 3 just got lost and now level 4 is not what the feed thinks. This is important because the feed has it's own idea of levels that the client structure must follow 1:1 at all times. A correct insert would look like:

0: 123
1: 89
2: 114
3: 77
4: 0
5: 55

It doesn't matter if 4 has a value of 0, as it doesn't necessarily mean "terminate further shifting of levels past the thing we're inserting." It's not a true sentinel value because it's not guaranteed the feed cannot send it nor is it guaranteed all levels are populated with non-zero values before inserts even start arriving. The correct thing to do here to avoid always processing all values up to MAX_NUM_DOM_LEVELS and also allow 0 as a valid existing price at any level is simply not rely on this false sentinel but keep a simple high watermark int in one of the parent structures. Said int points to the highest level processed either by an insert, update, or delete. An insert for level 0, means last_level is incremented. An update for level 7 means last_level = MAX(last_level, MIN(MAX_NUM_DOM_LEVELS, Level)). A delete for level 6 means last_level is decremented. It's a latching variable that just bounds the amount of processing the for loop has to do but more importantly gets the 0 value out of the picture. Obviously set it with a min() macro such that it's bounded by MAX_NUM_DOM_LEVELS, e.g. last_level = MIN(Level, MAX_NUM_DOM_LEVELS) so that values on the wire are safe. Then change the for loop to use UpdateLevel < last_level rather than UpdateLevel < MAX_NUM_DOM_LEVELS - 1. You're going to have to find a place in either m_BasicData to place this last_level for both Bid and Ask DOMs, otherwise the DOM structure itself is going to have to be something other than a simple array. Alternatively, one could just comment out the "if (TempDomStructure.Price == 0)" check and process every single index up to MAX_NUM_DOM_LEVELS. That will achieve the same thing as not breaking on legitimate levels that already have 0, not require any structure changes, but will require more iterative processing. Since the amount of inserts and deletes is probably far less than updates, this may be an acceptable tradeoff as it's only a single line change to make the problem go away.

The following will also totally break:

0: 123
1: 89
2: 0
3: 77
4: 55

Request to insert at level 2, with value of 114 will result in this afterward:

0: 123
1: 89
2: 114
3: 77
4: 55

On the surface, that may look right, but that's now potentially wrong because the feed's idea of 3 was 77 *before* the insert was sent. It's existing view probably looks like this:

0: 123
1: 89
2: 114
3: 0
4: 77
5: 55

Once again, 0 is not a safe sentinel value. Also, making an equality comparison to 0 using a float will probably work safely here, but in general floating point comparison is a known problem area. Aside from that, it's just not safe here to use 0 for anything other than 0 as a value, and not a special control value.


Given these existing values, the code will work right with an insert at level 2, with value of 114:

0: 123
1: 89
2: 77
3: 55

Resulting in:

0: 123
1: 89
2: 114
3: 77
4: 55

Which is the assumed normal case.



case UPDATE_DOM:
{
DomToUpdate[Level].Price = Price;
DomToUpdate[Level].Volume = Size;
}

break;

The above code looks fine although remember, it's entirely possible (although it might be weird) for the feed to send a value of 0.0 for price and the feed is going to expect it's idea of the levels being worked with is followed by your client using them. If a bunch of initial updates happened with a wide spread or possibly a level without any bids/asks (this *does* happen), you're going to end up with a level that's already 0 if the structure was initialized to 0's beforehand or explicitly updated by the feed with a price of 0.0. Remember, this is then going to break the insert processing because it thinks 0 (or 0.0) means "end of the array" when that isn't necessarily the case, and as a result it won't shift the remaining levels to the right positions that the feed thinks they should be at.



case DELETE_DOM:
{
// Clear the single level
memset(&DomToUpdate[Level], 0, sizeof(s_SCDomStructure));

The above isn't 100% portable, although it'll work probably everywhere. It's also safer to use "sizeof DomToUpdate[Level]" incase it's type changes. However, it's not even necessary because as long as Level is < MAX_NUM_DOM_LEVELS - 1 it's going to get stomped by the DomToUpdate[UpdateLevel] = DomToUpdate[UpdateLevel+1] below.

Additionally, when Level is == MAX_NUM_DOM_LEVELS - 1, it's going to get unconditionally stomped by the block which is following the for loop anyway.


for (int UpdateLevel = Level; UpdateLevel < MAX_NUM_DOM_LEVELS - 1; ++UpdateLevel)
{
DomToUpdate[UpdateLevel] = DomToUpdate[UpdateLevel+1];
//
//if (DomToUpdate[UpdateLevel + 1].Price == 0)
// break; // no more DOM Levels
}

//if (DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Price != 0)
{
DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Price = 0;
DomToUpdate[MAX_NUM_DOM_LEVELS - 1].Volume = 0;
}
}
break;

Again, 0 is not a safe value to use to mark the end of the levels. Only a separate variable holding the last unused index is safe. It's perfectly sane to zero out all of the Price and Volume members here for reasons of a placeholder value (as it's not like any other value makes sense, and I'm sure downstream code knows to display an empty DOM level if the value is 0.0), but it's not safe to do it just to manipulate the insert code's idea of what the end of the array is.

Also, this code has to process every level up to MAX_NUM_DOM_LEVELS on any delete. Using a last_level variable as I described previously would mean it wouldn't have to do all of that on every single delete. It just deletes up to that point because there's no way any levels beyond that have ever been touched.

Here's another example DOM that's busted due to these ordering issues with how the code is handling it....
Date Time Of Last Edit: 2015-03-31 03:08:14
imagesc_busted_dom_cl.png / V - Attached On 2015-03-31 02:13:36 UTC - Size: 90.39 KB - 413 views
[2015-03-31 03:00:23]
Sierra Chart Engineering - Posts: 104368
We understand what you are saying but this still indicates a fault on the IB side if they are sending through an insert or an update with a value of zero. The code is designed to be efficient. And we know that for a market that is trading around zero like a spread contract will be problem but this is not the issue here with the depth problems reported.


Also, this code has to process every level up to MAX_NUM_DOM_LEVELS on any delete. Using a last_level variable as I described previously would mean it wouldn't have to do all of that on every single delete. It just deletes up to that point because there's no way any levels beyond that have ever been touched.
Yes, we know and we were breaking earlier but removed that to ensure all the levels are shifted down towards index 0 because of the problems with the Interactive Brokers depth.

If a bunch of initial updates happened with a wide spread or possibly a level without any bids/asks (this *does* happen), you're going to end up with a level that's already 0 if the structure was initialized to 0's beforehand or explicitly updated by the feed with a price of 0.0.
If we understand this correctly, we do not agree with this. Even if there are skipped prices, that does not mean that there are blank levels, the depth level will not exist between the skipped prices.

So we still see this as a problem on the IB side and there still is not any determination otherwise. For Interactive Brokers to be updating a level with a price of zero, assuming they do that, makes no sense unless it is a spread contract. For them to be doing an insert with a price of zero makes no sense, assuming they do that, unless it is a spread contract. Neither of these has any logic whatsoever unless it were a spread contract. And it shows an incorrect data stream being transmitted , assuming this is even happening.

In any case, we changed the following code which will be out in the next release:

if (TempDomStructure.Price == 0 && TempDomStructure.Volume == 0)
break; // no more DOM Levels
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2015-03-31 03:37:36
[2015-03-31 03:38:05]
Sierra Chart Engineering - Posts: 104368
This sentence above has been updated:
Yes, we know and we were breaking earlier but removed that to ensure all the levels are shifted down towards index 0 because of the problems with the Interactive Brokers depth.

Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2015-03-31 04:16:17]
Sierra Chart Engineering - Posts: 104368
And we know that TWS sending an update or insert message with a price and size of 0 will not be solved by this code:

if (TempDomStructure.Price == 0 && TempDomStructure.Volume == 0)
break; // no more DOM Levels

So it has to be established that they are actually doing that to begin with as being the source of the problem.

The way we see this, is there is too much tolerance to substandard programming from an external service and because of the uncooperative nature of the external service, the problem gets dumped on us as if it is our problem. If we can do something to solve it, we will. So we will see about implementing what you have said. But it is not very likely that is the only problem or possibly even the problem at all.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2015-03-31 04:41:16

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account