Support Board
Support Board
Search Board
Control Panel
View My Posts / Threads
Direct MessagesDate/Time: Mon, 29 Apr 2024 14:06:50 +0000
Sierra needs to handle IB pacing violations more gracefully
View Count: 3164
| [2015-09-30 21:01:12] |
| i960 - Posts: 360 |
|
The way SC functions now when hitting a pacing violation is to drop the data for that which it got a violation for, shift the time window to the next day, and hit a pacing violation again - basically continuing until either the violations stop (which won't happen until SC stops trying to download) or the contract runs out. This is a problem because if a contract is long enough (say a quarterly like ES, SPI, N225, HSI, HHI, etc) then one may end up in a state where they can never effectively download the missing data. Delete and redownload is not a solution because you cannot say "redownload from the middle of the contract." Aside from that, it just makes no sense for SC to keep trying to download things that it's just going to get a pacing violation on. How it should be resolved: at the first pacing violation, SC should not move the time window to the next day - it should pause and retry to download the data it just got a violation for. If it gets another pacing violation, it should pace itself by doubling the pause time, and then attempting to download again. When it then successfully downloads the the current time window it should revert the pause interval back to normal and continue to the next day. Additionally, it can optionally give up if it never succeeds and the pause time has been doubled to some max n value, but that's an implementation detail. I have had a bear of a time getting intraday data for contracts with long duration (3 months) where I'm trying to get a year worth of intraday data. It takes multiple attempts to redownload the data and one has to keep deleting and redownloading but being very careful to only delete the contract month back which actually had the problem. It's quite tedious and if SC's pacing handling would simply retry/pause longer after the first pacing violation nobody would have to do this because it would do the right thing. Date Time Of Last Edit: 2015-09-30 21:02:14
|
| [2015-10-01 01:09:59] |
| |
|
We have reviewed this and we will replace the existing "Pacing Violation" avoidance method with the following: When a Pacing Violation is encountered, we will wait 10 seconds and not advance to the next request. If another pacing violation has occurred, we will double this time. When a historical data request goes through without a Pacing Violation, we will reset the delay time to 10 seconds next time one is encountered. When waiting for historical data requests because of a Pacing violation, we will add a message to the Message Log every 10 seconds indicating a wait is occurring and update the historical data activity variable to avoid a timeout. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-10-01 01:36:27
|
| [2015-10-01 03:49:24] |
| i960 - Posts: 360 |
|
Thanks, that sounds great SC. I'd imagine it's not too difficult and might reduce your support load from people having less problems with it as well.
|
| [2015-10-01 09:06:44] |
| i960 - Posts: 360 |
|
BTW, just looked up from IB's docs, might want to increase the backoff to 20 seconds: The following conditions can cause a pacing violation:
Making identical historical data requests within 15 seconds; Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds. |
| [2015-10-01 09:33:11] |
| |
|
OK we will do that. Will try to get this out by Friday evening.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2015-10-14 05:34:37] |
| |
|
We have implemented the changes for better Interactive Brokers "pacing violation" handling in version 1310. Please try it and let us know if you have any problems or it does not solve the issue. Otherwise, no need to comment. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2015-10-14 05:34:59
|
| [2015-10-20 06:53:56] |
| i960 - Posts: 360 |
|
"Improved handling when encountering a Interactive Brokers "Pacing Violation" historical data error. The historical data request will be retried on a delay. There should no longer be problems with missing data." Thanks for doing this - it will help out ALOT. I'll let you know of any issues I encounter. |
| [2015-10-20 16:41:24] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
Thanks for this, much needed.
|
| [2016-03-13 11:57:16] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
Yes, the backfill from IB is much better now, with the new pacing violation handling. However, it seems to be a problem when opening a chartbook with many charts in it. I created a chartbook containing 24 small stock daily charts (HISTORICAL CHARTS), to have a quick visualization of the stocks I'm following (see attached screenshot chartbook.png). When I open this chartbook, SC tries to update all the 24 charts, but only the first one gets updated, and the remaining 23 display a DOWNLOAD CANCELED error. See attached SC log file (SC LOG.txt). I see this error in the log file, regarding the #2 chart in the chartbook: --- HD Request # 3 | Downloading Historical Daily chart data for UCG-STK-SMART/BVME-EUR. Starting date: 2016-03-10. Service: ib | 2016-03-13 12:51:15 HD Request # 3 | Requesting 13 days of historical Daily data for UCG-STK-SMART/BVME-EUR with end date 2016-03-23 | 2016-03-13 12:51:15 Message from IB: Error reading request:Message type 26. Unable to parse data. java.lang.NumberFormatException: For input string: "STK". IB Error Code: 320. Request ID: 26. | 2016-03-13 12:51:15 --- So, to test if this is a symbol problem, I select EDIT => DELETE ALL DATA AND DOWNLOAD on the UCG-STK-SMART/BVME-EUR chart, and it gets correctly updated (see last lines of the log file). |
 chartbook.png / V - Attached On 2016-03-13 11:32:51 UTC - Size: 241.41 KB - 405 views  SC LOG.txt - Attached On 2016-03-13 11:56:53 UTC - Size: 30.68 KB - 351 views |
| [2016-03-14 15:02:31] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
Two updates to the problem: 1) the problem is not related to the number of charts in a chartbook: it persists even if the chartbook contains only 2 charts. Only the first chart is updated. 2) the problem seems related to italian stocks (exchange SMART/BVME): when using american stocks the problem no longer exists. |
| [2016-03-14 16:59:28] |
| |
|
This is the problem: HD Request # 3 | Requesting 13 days of historical Daily data for UCG-STK-SMART/BVME-EUR with end date 2016-03-23 | 2016-03-13 12:51:15
Message from IB: Error reading request:Message type 26. Unable to parse data. java.lang.NumberFormatException: For input string: "STK". IB Error Code: 320. Request ID: 26. | 2016-03-13 12:51:15 Contact Interactive Brokers about that error message because the symbol is being properly transmitted to Interactive Brokers as shown here: Interactive Brokers | Starting real-time market data updates for: UCG-STK-SMART/BVME-EUR. ID: 3 | 2016-03-13 12:51:15
Interactive Brokers | Subscribing to Symbol: UCG, SecurityType: STK, Expiration: , Exchange: SMART/BVME, Currency: EUR, Multiplier: | 2016-03-13 12:51:15 Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2016-03-14 16:59:48
|
| [2016-03-15 21:55:48] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
I contacted IB and they said this: In you historical data request, I see that you specified formatDate = 2. Please note that formatDate = 2 is only valid for intraday bars. For bar size of 1 Day or larger you would need to use formatDate = 1 instead. Please adjust your program accordingly and see if you still get the same error message.
Is this information useful? |
| [2016-03-15 22:53:01] |
| |
|
We will check this.
Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2016-06-01 08:03:55] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
up.
|
| [2016-06-01 09:48:53] |
| |
|
This was looked into and verified again just now: In you historical data request, I see that you specified formatDate = 2. Please note that formatDate = 2 is only valid for intraday bars. For bar size of 1 Day or larger you would need to use formatDate = 1 instead. Please adjust your program accordingly and see if you still get the same error message.
It has been set correctly and is now. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2016-06-01 09:49:33
|
| [2016-07-05 09:53:17] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
Ok... but the error is still happening: => see attached log file "log1.txt". What it's strange is that the error happens only if there are more than one historical chart in a chart book. The first historical chart gets correctly updated: HD Request # 2 | Downloading Historical Daily chart data for IBUS30-CFD-SMART-USD-BAAVG. Starting date: 2016-07-01. Service: ib | 2016-07-05 11:35:24
HD Request # 2 | Requesting 14 days of historical Daily data for IBUS30-CFD-SMART-USD with end date 2016-07-15 | 2016-07-05 11:35:24 HD Request # 2 | Processing historical data message from TWS | 2016-07-05 11:35:26 HD Request # 2 | Receiving historical Daily data for IBUS30-CFD-SMART-USD-BAAVG starting at 2016-07-01 | 2016-07-05 11:35:26 HD Request # 2 | Writing historical Daily data to the file IBUS30-CFD-SMART-USD-BAAVG.dly | 2016-07-05 11:35:26 HD Request # 2 | Received 6 records and wrote 2 records from 2016-06-27 00:00:00 to 2016-07-05 00:00:00. | 2016-07-05 11:35:26 HD Request # 2 | Received 6 records from 2016-07-01 00:00:00 to 2016-07-05 00:00:00 (5.0 days) and wrote 2 records for IBUS30-CFD-SMART-USD-BAAVG | 2016-07-05 11:35:26 HD Request # 2 | Daily download COMPLETE for IBUS30-CFD-SMART-USD-BAAVG. Completion time: 1s. Unique request ID: 1 | 2016-07-05 11:35:26 Removed historical data download ID 1 | 2016-07-05 11:35:26 Then the second chart generates the error: HD Request # 3 | Downloading Historical Daily chart data for IBDE30-CFD-SMART-EUR-BAAVG. Starting date: 2016-07-04. Service: ib | 2016-07-05 11:35:26
HD Request # 3 | Requesting 11 days of historical Daily data for IBDE30-CFD-SMART-EUR with end date 2016-07-15 | 2016-07-05 11:35:26 Message from IB: Error reading request:Message type 10. Unable to parse data. java.lang.NumberFormatException: For input string: "CFD". IB Error Code: 320. Request ID: 10. | 2016-07-05 11:35:26 Socket (4) | Closed. No error. | 2016-07-05 11:35:26 and the updating stops, without any retry. If there is only one historical chart in the chartbook, no error is generated => see attached log file "log2.txt". This makes me think that SC make a correct call to IB to retrieve historical data for the first chart and then, for whatever reason, the call for the second chart is incorrect... it might be just a wrong handling of a variabile into a cycle... just guessing, sorry if I do... only try to help. Date Time Of Last Edit: 2016-07-05 09:54:37
|
| log1.txt - Attached On 2016-07-05 09:47:01 UTC - Size: 16.44 KB - 757 views log2.txt - Attached On 2016-07-05 09:47:04 UTC - Size: 7.13 KB - 319 views |
| [2016-07-05 10:09:43] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
Here's another case: - only 1 historical chart - I do "delete all data and download" => update is fine - I repeat "delete all data and download" on the same chart => error: Interactive Brokers | Using primary service for historical data for IBUS30-CFD-SMART-USD-BAAVG | 2016-07-05 12:06:52
IBUS30-CFD-SMART-USD-BAAVG Daily #1 | Reloading chart. | 2016-07-05 12:06:52 Interactive Brokers | Using primary service for historical data for IBUS30-CFD-SMART-USD | 2016-07-05 12:07:02 HD Request # 6 | Downloading Historical Daily chart data for IBUS30-CFD-SMART-USD-BAAVG. Starting date: 1966-07-05. Service: ib | 2016-07-05 12:07:02 HD Request # 6 | Requesting 365 days of historical Daily data for IBUS30-CFD-SMART-USD with end date 2016-07-05 | 2016-07-05 12:07:02 Message from IB: Error validating request:-'yd' : cause - Historical data requested duration is invalid. IB Error Code: 321. Request ID: 4. | 2016-07-05 12:07:02 Socket (15) | Closed. No error. | 2016-07-05 12:07:02 |
| log3.txt - Attached On 2016-07-05 10:09:38 UTC - Size: 7.46 KB - 352 views |
| [2016-07-06 04:19:46] |
| |
|
It could not be this: it might be just a wrong handling of a variabile into a cycle... just guessing, sorry if I do... only try to help.
What it's strange is that the error happens only if there are more than one historical chart in a chart book. The first historical chart gets correctly updated: We do not know how to help with the problem. The problem must be on the Interactive Brokers side. We ran several tests and cannot reproduce it: Interactive Brokers | Using primary service for historical data for IBUS30-CFD-SMART-USD | 2016-07-05 15:14:40
HD Request # 6 | Downloading Historical Daily chart data for IBUS30-CFD-SMART-USD-BAAVG. Starting date: 1966-07-06. Service: ib | 2016-07-05 15:14:40 HD Request # 6 | Requesting 365 days of historical Daily data for IBUS30-CFD-SMART-USD with end date 2016-07-05 | 2016-07-05 15:14:40 HD Request # 6 | Message from IB: No security definition has been found for the request. IB Error Code: 200. Request ID: 6. | 2016-07-05 15:14:40 HD Request # 6 | Error downloading historical Daily data for IBUS30-CFD-SMART-USD-BAAVG. The symbol is unknown to the server. | 2016-07-05 15:14:40 HD Request # 6 | Received 0 records from 00:00:00 to 00:00:00 (0.0 seconds) and wrote 0 records for IBUS30-CFD-SMART-USD-BAAVG | 2016-07-05 15:14:40 HD Request # 6 | Daily download COMPLETE for IBUS30-CFD-SMART-USD-BAAVG. Completion time: 1s. Unique request ID: 5 | 2016-07-05 15:14:40 Removed historical data download ID 5 | 2016-07-05 15:14:40 Message from IB: No historical data query found for ticker id:6. IB Error Code: 366. Request ID: 6. | 2016-07-05 15:14:40 Interactive Brokers | Using primary service for historical data for IBUS30-CFD-SMART-USD-BAAVG | 2016-07-05 15:14:41 IBUS30-CFD-SMART-USD-BAAVG Daily #1 | Reloading chart. | 2016-07-05 15:14:41 It is not surprising that we cannot reproduce it. But nevertheless, we know that the issue is something that Interactive Brokers needs to help you with. If there is a problem with one of the fields, they need to tell us specifically which one and why. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
| [2016-07-06 07:51:00] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
I'll contact IB and see what happens.
Date Time Of Last Edit: 2016-07-06 07:53:00
|
| [2016-07-06 12:44:08] |
| Jagui - Posts: 19 Invalid SC Account Name. |
|
I contacted IB and received a prompt reply. I sent IB the TWS log file that corresponds to the SC message log errors shown in the previous posts (this TWS log is attached). MY MESSAGE TO IB: 'm trying to get historical updates, using the Sierra Chart software, for the symbol IBUS30 (the error happens also with other symbols).
The first historical request after connecting to TWS goes fine, but the second fails. See attached TWS API log file. For what I understand, there is a first historical request that goes fine, and the second fails: 10:04:25:008 <- 1-10-1--IBUS30- CFD-----SMART-- USD---0--0- 10:04:25:008 <- 9-7-1--IBUS30-C FD-----SMART-US D---1--- 10:04:25:024 <- 20-5-3--IBUS30- CFD-----SMART-- USD---0-2016071 6 23:59:59 GMT-1 day-11 D-0-MIDPOINT-1- 10:04:25:477 -> 10-8-1-IBUS30-C FD--0--SMART-US D-IBUS30-IBUS30 -IBUS30-1117678 79-0.01--ACTIVE TIM,ADJUST,ALER T,ALLOC,AVGCOST ,BASKET,COND,CO NDORDER,DAY,DEA CT,DEACTDIS,DEA CTEOD,GAT,GTC,G TD,GTT,HID,LIT, LMT,MIT,MKT,MTL ,NGCOMB,NONALGO ,OCA,OPG,SCALE, SCALERST,SNAPMI D,SNAPMKT,SNAPR EL,STP,STPLMT,T RAIL,TRAILLIT,T RAILLMT,TRAILMI T,USEMID,USESTK MD,WHATIF-SMART -1-111746713-US 30------GB-2016 0706:0700-2000;20160 707:0700-2000-20160 706:0700-2000;20160 707:0700-2000---0- 10:04:25:477 -> 52-1-1- 10:04:25:523 -> 17-3-3-20160706 01:59:59-20160717 01:59:59-4-20160630-1 7682.82-17915.7 35-17645.435-17 913.73--1--1.00 -false--1-20160 701-17871.49-17 984.895-17843.0 15-17928.94--1- -1.00-false--1- 20160705-17885. 97-17899.965-17 771.23-17822.13 --1--1.00-false --1-20160706-17 772.72-17855.24 5-17763.22-1779 5.80--1--1.00-f alse--1- 10:04:33:526 <- 20-5-4--IBUS30- CFD-----SMART-- USD---0-49-1-20 -5-5-- 10:04:33:526 -> 4-2-4-321-Error validating request:-'yd' : cause - Historical data requested duration is invalid.- --------------- --------------- ---------- I contacted Sierra Chart, and they said it could be a TWS problem. Ca you please help? Best Regards. FILE ATTACHED : api.9368395.Wed .log INTERACTIVE BROKERS ANSWER: Dear Mr Giuranna,
Thank you for creating a web ticket and providing the log file. After careful investigation I have noticed, that unfortunately your second historical data request is passing some strange values instead of the needed parameters. Here is your first request, which looks fine: 10:04:25:024 <- 20-5-3--IBUS30- CFD-----SMART-- USD---0-2016071 6 23:59:59 GMT-1 day-11 D-0-MIDPOINT-1- It resulted in a successful reply: 10:04:25:477 -> 10-8-1-IBUS30-C FD--0--SMART-US D-IBUS30-IBUS30 -IBUS30-1117678 79-0.01--ACTIVE TIM,ADJUST,ALER T,ALLOC,AVGCOST ,BASKET,COND,CO NDORDER,DAY,DEA CT,DEACTDIS,DEA CTEOD,GAT,GTC,G TD,GTT,HID,LIT, LMT,MIT,MKT,MTL ,NGCOMB,NONALGO ,OCA,OPG,SCALE, SCALERST,SNAPMI D,SNAPMKT,SNAPR EL,STP,STPLMT,T RAIL,TRAILLIT,T RAILLMT,TRAILMI T,USEMID,USESTK MD,WHATIF-SMART -1-111746713-US 30------GB-2016 0706:0700-2000;20160 707:0700-2000-20160 706:0700-2000;20160 707:0700-2000---0- Now here is your second request: 10:04:33:526 <- 20-5-4--IBUS30- CFD-----SMART-- USD---0-49-1-20 -5-5-- You can see the difference. It does not have valid durations string or the end time. So this resulted in the error message you have received. That being said the issue here is most probably not on the IB TWS/API side, since we are not receiving correct requests from the third party software you are using. Please let Sierra Charts know about it, so that they could further troubleshoot on their side. Best regards, Pavel S. Interactive Brokers API Support, Technical Assistance Center |
Private File |
| [2016-07-06 17:18:24] |
| |
|
We will think about this but there is really zero percent chance that this is actually true: Now here is your second request:
This could not be what Sierra Chart is actually sending. And none of our other Interactive Brokers users are experiencing this.
10:04:33:526 <- 20-5-4--IBUS30- CFD-----SMART-- USD---0-49-1-20 -5-5-- You can see the difference. It does not have valid durations string or the end time. So this resulted in the error message you have received. Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing Date Time Of Last Edit: 2016-07-06 17:26:30
|
| [2016-07-06 18:14:30] |
| |
|
We checked and we verified that this cannot possibly be true: Now here is your second request:
10:04:33:526 <- 20-5-4--IBUS30- CFD-----SMART-- USD---0-49-1-20 -5-5-- You can see the difference. It does not have valid durations string or the end time. So this resulted in the error message you have received. Not at all. So this answer from Interactive Brokers is not correct as far as it being a problem with the client application request data being incorrect. The data is being made incorrect somewhere within TWS most likely. Furthermore, we verified that even this request which appears correct on their side: 10:04:25:024 <- 20-5-3--IBUS30- CFD-----SMART-- USD---0-2016071 6 23:59:59 GMT-1 day-11 D-0-MIDPOINT-1- Sierra Chart Support - Engineering Level Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy: https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service: Sierra Chart Teton Futures Order Routing |
To post a message in this thread, you need to log in with your Sierra Chart account: