Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Wed, 24 Apr 2024 06:07:19 +0000
DTC Decode error on one message only using same decode function. Google protocol buffers.
View Count: 2589
| [2018-04-15 23:32:36] |
| enemyspy - Posts: 304 |
|
Hi, using google protocol buffers on My DTC server. I am getting this error when decoding the DTC message SECURITY_DEFINITION_FOR_SYMBOL_REQUEST sent from Sierra Chart version 1727 remote client: google.protobuf.message.DecodeError: Error parsing message This snippet from my log includes the encoded message and the decoded header. received b'\x16\x00\xfa\x01\x08\x01\x12\x08LTC--BTC\x1a\x04OKEX\x18\x00e\x00\x08\x01\x10\x01\x1a\x08LTC--BTC"\x04OKEX' Message Enum: SECURITY_DEFINITION_FOR_SYMBOL_REQUEST decoded header: (22, 506) serlialized message: b'\x08\x01\x12\x08LTC--BTC\x1a\x04OKEX\x18\x00e\x00\x08\x01\x10\x01\x1a\x08LTC--BTC"\x04OKEX' Using the same uniform decode method: LOGON,HEARTBEAT. MARKET_DATA/DEPTH requests alld decode properly. So wondering why the google protocol decoding would only fail on the one message above from Sierra Chart. Does the message above look to be encoded properly? Full log with stack trace of exception if necessary: starting up on xxx.xxx.xxx.xxx accepting connections connection from ('xx.xx.xx.xx', 65427) Active Threads: 3 accepting connections received b'\x10\x00\x06\x00\x08\x00\x00\x00\x04\x00\x00\x00DTC\x00' sending response response sent back to client received b'7\x00\x01\x00\x08\x08\x12\x08enemyspy\x1a\tlemonbomb"\x00(\x000\x008\x14@\x02J\x00R\x00Z\x0cSierra Chart' Message Enum: LOGON_REQUEST sending response response sent back to client received b'\x0f\x00\x03\x00\x08\x00\x11\x00\x00\x00\x00\x00\x00\x00\x00' Message Enum: HEARTBEAT sending response response sent back to client received b'\x0f\x00\x03\x00\x08\x00\x11\x00\x00\x00\x00\x00\x00\x00\x00' Message Enum: HEARTBEAT sending response response sent back to client received b'\x0f\x00\x03\x00\x08\x00\x11\x00\x00\x00\x00\x00\x00\x00\x00' Message Enum: HEARTBEAT sending response response sent back to client received b'\x16\x00\xfa\x01\x08\x01\x12\x08LTC--BTC\x1a\x04OKEX\x18\x00e\x00\x08\x01\x10\x01\x1a\x08LTC--BTC"\x04OKEX' Message Enum: SECURITY_DEFINITION_FOR_SYMBOL_REQUEST decoded header: (22, 506) serlialized message: b'\x08\x01\x12\x08LTC--BTC\x1a\x04OKEX\x18\x00e\x00\x08\x01\x10\x01\x1a\x08LTC--BTC"\x04OKEX' Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/var/www/FlaskApp/FlaskApp/dtc/dtcServer.py", line 99, in listenToClient res = self.onMessage(data,ID) File "/var/www/FlaskApp/FlaskApp/dtc/dtcServer.py", line 145, in onMessage return dtc.SecurityDefinitionResponse(msg,self.symbset) File "/var/www/FlaskApp/FlaskApp/dtc/dtcProtocol.py", line 116, in SecurityDefinitionResponse request.ParseFromString(msg) google.protobuf.message.DecodeError: Error parsing message |
| [2018-04-16 04:25:48] |
| |
|
Make sure you are using the latest Google protocol buffer file: https://dtcprotocol.org/DTC_Files/DTCProtocol.proto We see no problem with the encoding and decoding of this message when it is used between installations of Sierra Chart using Google protocol buffer encoding. Also the details you provided, do not mean anything to us. The low-level encoding is meaningless to us. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2018-04-17 04:53:56] |
| enemyspy - Posts: 304 |
|
Ok, I re-compiled the .proto from the link above and result is same all messages decoding except security definition for symbol. if I decode using just a regular byte decoder this is what the message yields in plain text: LTCBTCOKEXLTCBTC"OKEX
This is what the sc message log says about the security def request: DTC Service | Starting real-time market data updates for: LTCBTC.OKEX. ID: 1 | 2018-04-17 04:34:37 DTC Service | Requesting security definition data for: LTCBTC.OKEX. ID: 1 | 2018-04-17 04:34:37 So looking at the string message it appears that the ID: 1 is missing. All I do to decode is bytes to byte array The header get bytearray[:4] and the message get bytearray[4:]. The header decodes to size 22 & enum 506. The enum is right but can't confirm if the size is right. I could take that decoded string at the top and reconstruct a response but not without the ID. I could maintain an ID count to get around it though. Can you confirm if the IDS simply get incremented on every new symbol? |
| [2018-04-17 20:41:32] |
| |
|
The Request ID field definitely should be in the security definition for symbol request message. We do not see how there could be a problem with this. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2018-04-17 22:45:33] |
| enemyspy - Posts: 304 |
|
yes I am scratching my head at it. I am going to try re installing the google protocol packages as the c++ cython implementation in python instead of the default when I get home and see if that works. But I find it strange that the Request ID does not even show up when decoded to a string. If that does not work though I am going to have to employ a hacky work around which is why I wanted to know if it simply increments a count to generate ids.
|
| [2018-04-17 22:47:01] |
| enemyspy - Posts: 304 |
|
I wonder if it is some kind of windows -> linux encoding kind of issue.
|
| [2018-04-17 23:59:56] |
| |
|
Use the symbol ABCD and then provide us the entire series of raw bytes in hex byte by byte for the 506 message. We will compare it to what we are encoding. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2018-04-18 00:00:13
|
| [2018-04-19 05:39:46] |
| enemyspy - Posts: 304 |
|
Ok just note that there was an original of this post but I figured nobody read it yet. So I deleted and re-did it with better information on further testing. In short only 4 char symbols work see details below: So ABCD.GDAX or any 4 char symbol works: Hex: \x12\x00\xfa\x01\x08\x01\x12\x04ABCD\x1a\x04GDAX\x14\x00e\x00\x08\x01\x10\x01\x1a\x04ABCD"\x04GDAX However on the first time the python socket server accepts a connection google protobuf posts the following warning on decode below: /var/www/FlaskApp/FlaskApp/dtc/dtcProtocol.py:122: RuntimeWarning: Unexpected end-group tag: Not all data was converted
This still seems to decode and has all the info necessary to send a response. If you close the connection client side and reopen it without closing the server it does not continue to trigger the warning. I find this interesting but that is not a sierra chart problem as the hex is identical.request.ParseFromString(msg) The problem is as soon as you enter more than 4 chars for the symbol it causes a full exception and crashes the server with the original error I posted: google.protobuf.message.DecodeError: Error parsing message
The symbol for this:RDTTB.OKEX hex: \x13\x00\xfa\x01\x08\x03\x12\x05RDTTB\x1a\x04OKEX\x15\x00e\x00\x08\x01\x10\x03\x1a\x05RDTTB"\x04OKEX ABCD with no exchange also causes a full google protobuf exception: hex: \x19\x00\xfa\x01\x08\x01\x12\x04ABCD\x1a\x0bequities.us\x1b\x00e\x00\x08\x01\x10\x01\x1a\x04ABCD"\x0bequities.us Date Time Of Last Edit: 2018-04-19 07:01:16
|
| [2018-04-22 19:00:53] |
| enemyspy - Posts: 304 |
|
This was the solution: Data and Trading Communications (DTC) Protocol Test Client: Populating Symbol List in Sierra Chart |
| [2018-04-23 19:21:50] |
| |
|
We still need some time to do some testing. None of this really makes sense from our perspective. May be it would help to use the Google protocol buffer version 3 compiler and library. That is what we are using. It is still backward-compatible with Google protocol buffer version 2 definition files. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2018-04-25 03:29:10] |
| enemyspy - Posts: 304 |
|
I used the v3 library which installs by default using pip install for python 3.5 on linux ubuntu, and using the v2 .proto as you suggested. It does not make sense to me either. I don't want you take a bunch of time trying to sort this out when it is mostly likely going to turn out to be on my side. I will triple check to make sure that both the compiler and library are both version 3. Sending the unsolicited security defs directly after the logon response and indicating that the security defs are not supported in the login response seems to be a suitable work around. However it would be a nice feature if Update Global Symbol Settings button in sierra chart global symbol settings could optionally call the DTC SERVICE instead of Sierra Chart servers. This make it so that a batch sercurity def update can be sent only when it is specifically requested, instead of everytime the client connects. That is not a huge deal though. If I find any fixes I will post them here, in case someone in the future runs into this with python. Date Time Of Last Edit: 2018-04-25 03:50:14
|
| [2018-05-21 05:15:40] |
| enemyspy - Posts: 304 |
|
I encountered the same problem again and I realized what the issue was the whole time and fixed. It was pretty dumb and completely my fault. All I can say is the google protocol buffers work great, and it is awesome that this whole thing is possible.
|
| [2018-05-21 08:53:45] |
| |
|
Okay that is good. That is one less thing for us to look at. We still have not gotten to this yet, in part because you said you had a solution. However it would be a nice feature if Update Global Symbol Settings button in sierra chart global symbol settings could optionally call the DTC SERVICE instead of Sierra Chart servers. We will see about this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: