Login Page - Create Account

Home >> Support Board

Support Board

Date/Time: Thu, 25 Apr 2024 21:51:22 +0000

How to Post a Message for Support

[User Discussion] - Python for Sierra Chart

View Count: 34158

[2019-06-27 15:33:37]
[Go To First Post]
User183724 - Posts: 183
Kiwi

Quick question...do you have a SCID to TEXT version of your Text2SCid.py that you would be willing to share? I'd like to read the SCID file into a program as a text or CSV file.

Im new to Python (used a lot of other languages)... have been using SC for a while...I kind of understand what youre doing here and would be interesting in trying to develop something.

Thank You
[2019-06-27 22:39:17]
[Go To First Post]
Kiwi - Posts: 374
This will probably work. It converts an scid file (named after the imports) to a tab separated variable file of the same name (but .scid becomes .tsv) & path.

Let me know if you have problems (including any error messages). It's updated for Python 3 & tested with 3.7.4.
Date Time Of Last Edit: 2019-06-27 22:39:57
attachmentSCID_to_TSV.py - Attached On 2019-06-27 22:38:54 UTC - Size: 3.55 KB - 757 views
[2019-06-28 00:23:47]
[Go To First Post]
User183724 - Posts: 183
Kiwi- Thanks for the file. I'll let you know how it goes
[2019-06-28 16:28:36]
[Go To First Post]
User183724 - Posts: 183
Has anyone worked with the SCSS spreadsheet files? Do you have py files that read/write to these files that you would be willing to share with me? Thanks in advance
[2019-07-01 20:57:55]
[Go To First Post]
Sierra Chart Engineering - Posts: 104368
The Spreadsheet file format is too complex to be working with in this way.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
Date Time Of Last Edit: 2019-07-01 20:58:08
[2019-07-02 00:49:00]
[Go To First Post]
User183724 - Posts: 183
Thanks for the reply. I'm really just looking for options on how to read real time data into Python. Spread sheet SCSS files were just something I was looking at
[2019-09-16 15:29:03]
[Go To First Post]
Mack - Posts: 83
What is really missing is a Python framework for SierraChart. Something that allows to code, backtest and review strategies.

@Kiwi: in your opinion, would it be better to develop a standalone backtest system? or something that is tightly integrated with SierraChart? In theory, you could just use SC to get data, and execute signals. (but do the backtesting in your own backtester)
Date Time Of Last Edit: 2019-09-16 15:31:58
[2021-06-18 18:10:52]
[Go To First Post]
biconoid - Posts: 5
Trying to read Sierra Chart SCID file using Python and export as CSV/TSV

Using following approach:

With reference to s_IntradayRecord Structure Member Descriptions at
Intraday Data File Format

- The Intraday data record structure, s_IntradayRecord, is 40 bytes in size.
- The Intraday data file header, s_IntradayHeader, is 56 bytes in size.
- The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer.
It represents the number of microseconds since the SCDateTime epoch of December 30, 1899.

s_IntradayRecord()// Constructor
struct s_IntradayRecord
{
static const float SINGLE_TRADE_WITH_BID_ASK;
static const float FIRST_SUB_TRADE_OF_UNBUNDLED_TRADE;
static const float LAST_SUB_TRADE_OF_UNBUNDLED_TRADE;

s_IntradayRecord()// Constructor PYTHON PART
Sierra Chart | { Type | https://docs.python.org/3/library/struct.html
----------------------- |---------------------------------- | -----------------------------------------------
SCDateTimeMS DateTime; | DateTime; d |
float Open; | Open = 0.0; f | Format C Python type Standard
float High; | High = 0.0; f | Type Type Size(bytes)
float Low; | Low = 0.0; f | ------------------------------------------
float Close; | Close = 0.0; f | f float float 4
u_int32 NumTrades; | NumTrades = 0; I | I unsigned int integer 4
u_int32 TotalVolume; | TotalVolume = 0; I | d double float 8
u_int32 BidVolume; | BidVolume = 0; I |
u_int32 AskVolume; | AskVolume = 0; I |
| }
};



Procedure:
Data unpacking as:
size_of_header = 0x36 (hex) # 56 bytes # Discarded
size_of_record = 0x28 (hex) # 40 bytes (<d 4f 4I) # Refer to above table
unpacked_data = struct.unpack('<d 4f 4I', data) # Further exported and saved to files like below

# Following TSV (CSV) file is obtainted after getting the data.scid (check attached)

$ cat data.tsv
Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume
1.8752101406881474e-308 2623.0 0.0 2623.0 2623.0 1 1 1 0
1.8754475234208735e-308 2717.0 0.0 2717.0 2717.0 1 1 1 0
1.875547350372747e-308 2750.0 0.0 2750.0 2750.0 1 1 1 0
1.8757877222026304e-308 2742.5 2757.75 2742.5 2742.5 1 1 1 0
1.876050741013588e-308 2830.0 2830.0 2805.75 2830.0 5 0 0 5
1.876054622887367e-308 2845.0 2845.0 2814.75 2845.0 3 0 0 3
1.876054699961608e-308 2845.0 2845.0 2814.75 2845.0 2 0 0 2
1.8761342153807153e-308 2900.0 2900.0 2870.75 2900.0 3 0 0 3
1.876134249471245e-308 2900.0 2900.0 2870.75 2900.0 2 0 0 2
1.876686289274035e-308 2890.0 2918.25 2890.0 2890.0 1 1 1 0

# Looks similar in data_frame
$ python3
Python 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.read_csv('data.tsv', sep='\t')
Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume
0 1.875210e-308 2623.0 0.00 2623.00 2623.0 1 1 1 0
1 1.875448e-308 2717.0 0.00 2717.00 2717.0 1 1 1 0
2 1.875547e-308 2750.0 0.00 2750.00 2750.0 1 1 1 0
3 1.875788e-308 2742.5 2757.75 2742.50 2742.5 1 1 1 0
4 1.876051e-308 2830.0 2830.00 2805.75 2830.0 5 0 0 5
5 1.876055e-308 2845.0 2845.00 2814.75 2845.0 3 0 0 3
6 1.876055e-308 2845.0 2845.00 2814.75 2845.0 2 0 0 2
7 1.876134e-308 2900.0 2900.00 2870.75 2900.0 3 0 0 3
8 1.876134e-308 2900.0 2900.00 2870.75 2900.0 2 0 0 2
9 1.876686e-308 2890.0 2918.25 2890.00 2890.0 1 1 1 0
>>>
Note: Above dataframe output is almost same EXCEPT DATETIME as if i export SCID to a text file and import into a dataframe.

Manual conversion of SCID file to TXT via SierraChart
#ESM21-CME-Sample.txt
Date, Time, Open, High, Low, Last, Volume, NumberOfTrades, BidVolume, AskVolume
2020/4/8, 00:32:59, 2623.00, 0.00, 2623.00, 2623.00, 1, 1, 1, 0
2020/4/13, 14:00:47, 2717.00, 0.00, 2717.00, 2717.00, 1, 1, 1, 0
2020/4/15, 22:08:19, 2750.00, 0.00, 2750.00, 2750.00, 1, 1, 1, 0
2020/4/21, 13:16:57, 2742.50, 2757.75, 2742.50, 2742.50, 1, 1, 1, 0
2020/4/27, 17:09:33, 2830.00, 2830.00, 2805.75, 2830.00, 5, 1, 0, 5
2020/4/27, 19:20:30, 2845.00, 2845.00, 2814.75, 2845.00, 3, 1, 0, 3
2020/4/27, 19:23:06, 2845.00, 2845.00, 2814.75, 2845.00, 2, 1, 0, 2
2020/4/29, 16:05:27, 2900.00, 2900.00, 2870.75, 2900.00, 3, 1, 0, 3

----------------------------------------------------------------------------------------------------------------
Issue:
----------------------------------------------------------------------------------------------------------------
Unable to extract/convert the datetime part to a proper value. Anyone has any pointers. Tried via
- numpy
- datetime module

Sample Python code
-----------------

$ python3
Python 3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime
>>> datetime.fromtimestamp(1.8752101406881474e-308)
datetime.datetime(1969, 12, 31, 19, 0)
>>>
FAIL: That's an error


Sample Python code
-----------------
import numpy as np
from time import time

"""
From: Intraday Data File Format

The DateTime member variable is a SCDateTimeMS variable.
This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899.
"""

# Single parameters
ep = np.datetime64('1899-12-30') # epoch date
tz = np.timedelta64(5, 'h')/np.timedelta64(1, 'D') # time-zone adjustment UTC +5
dv = 1.8752101406881474e-308 # d[0] value from struct.unpack('<d 4f 4I') refer to TSV file or DataFrame above
dt = dv + tz # adjusted time

# Try to get full date
ms = (np.timedelta64(int(round((dt-int(dt))*86400)),'ms')) # Delta in millisecond for current day
my_time = ep + np.timedelta64(int(dt)) + ms # Epoch Time + Delta + millisecond spent

print(my_time)

#Output
-------------------------
1899-12-30T00:00:18.000
FAIL: That's an error

# Attachment
# data.scid - is a subset of 'ESM21-CME.scid' 2.5Gb file
i've only extracted
456 bytes for simplifying data extraction
(56 + 400 bytes)
56 bytes = header
400 bytes = 40 x 10 intraday records
attachmentdata.scid - Attached On 2021-06-18 18:09:39 UTC - Size: 457 B - 396 views
[2021-06-18 19:51:38]
[Go To First Post]
User907968 - Posts: 802
Your unpack is not correct, you are using double for the date time variable, which it is not.
[2021-06-18 20:09:53]
[Go To First Post]
biconoid - Posts: 5
As per https://docs.python.org/3/library/struct.html there are only three options to unpack 64 bits (8 bytes), i.e. q, Q and d.
What would be correct structure to unpack?

Format C Python Standard
Type Type Size(bytes)
---------------------------------------------------
Q unsigned long long integer 8
q long long integer 8
d double float 8

Also Sierra documentation mentions
Versions prior to 2151, use a double precision floating point type variable to represent Date-Time values.

Just for testing purposes i've tested using q and Q.
$ cat files/data.tsv
Date Open High Low Close Volume NumberOfTrades BidVolume AskVolume
3795467579000000.0 2623.0 0.0 2623.0 2623.0 1 1 1 0
3795948047000000.0 2717.0 0.0 2717.0 2717.0 1 1 1 0
3796150099000000.0 2750.0 0.0 2750.0 2750.0 1 1 1 0
3796636617000000.0 2742.5 2757.75 2742.5 2742.5 1 1 1 0

When unpacking using Q or q.
>>> datetime.fromtimestamp(3795467579)
datetime.datetime(2090, 4, 9, 20, 32, 59)
>>>

and
>>> datetime.fromtimestamp(3795467579000000.0 )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: year 120275549 == out of range
>>>
Date Time Of Last Edit: 2021-06-18 20:26:19
[2021-06-18 22:20:48]
[Go To First Post]
Kiwi - Posts: 374
Here's a python file that converts an scid to csv with defaults for eur.usd in my data directory. Feel free to extract what you need or modify and reuse.

"""
Takes a SierraChart scid file (input argument 1) and converts
it to a comma separated file (comma)

Timezone conversion can follow the users local timezone, or a
specified integer (input l or an integer but if the default
filename is being used, '' must be specified for the filename)

Inputs: filename & tz adjust (else UTC).
if the filename is 'all' then all scids in directory are converted
"""
import csv
import re
import sys
import os
import struct
from datetime import date, datetime, time, timedelta


def deserialize(excelDateAndTime):
"""
Returns excel date time as Python datetime object
"""
date_tok = int(excelDateAndTime)
time_tok = round(86400*(excelDateAndTime - date_tok))
d = date(1899, 12, 30) + timedelta(date_tok)
t = time(*helper(time_tok, (24, 60, 60, 1000000)))
return datetime.combine(d, t)


def helper(factor, units):
factor /= 86399.99
result = list()
for unit in units:
value, factor = divmod(factor * unit, 1)
result.append(int(value))
result[3] = int(0)
return result


def datetimeTwo(excelDateAndTime, tzAdjust):
"""
Returns excel date time in format YYYYMMDD HHMMSS
"""
t = str(deserialize(excelDateAndTime + tzAdjust.seconds/86400.0))
return (re.sub(r'(:|-)', '', t)[:8], re.sub(r'(:|-)', '', t)[9:15])


def excelTimeAdjust(date_and_time, tzAdjust):
return date_and_time + tzAdjust.seconds/86400.0


def convertRecords(filename, fileoutput, tzvar, dtformat):
"""
Read in records from SierraChart .scid data filename
dtformat determines the datetime representation in column 1 (and 2)
"""
sizeHeader = 0x38
sizeRecord = 0x28
if tzvar == 99999:
tzAdjust = datetime.fromtimestamp(0) - datetime.utcfromtimestamp(0)
else:
tzAdjust = timedelta(hours=tzvar)

header = ('Date', 'Time', 'O', 'H', 'L', 'C', 'V', 'T')
ftsv = open(fileoutput, 'w')
fileout = csv.writer(ftsv, delimiter=',')
with open(filename, 'rb') as fscid:
fscid.read(sizeHeader) # discard header
fileout.writerow(header)
data = fscid.read(sizeRecord)
while data:
dataRow = struct.unpack('d4f4I', data)
if dtformat == 0:
adjustedTime = dataRow[0] + tzAdjust.seconds/86400.0
outrow = (str(adjustedTime), str(adjustedTime),
str(dataRow[1]), str(dataRow[2]), str(dataRow[3]),
str(dataRow[4]), str(dataRow[5]), str(dataRow[6]))
fileout.writerow(outrow)
elif dtformat == 1:
date, time = datetimeTwo(dataRow[0], tzAdjust)
outrow = (date, time,
str(dataRow[1]), str(dataRow[2]), str(dataRow[3]),
str(dataRow[4]), str(dataRow[5]), str(dataRow[6]))
fileout.writerow(outrow)
data = fscid.read(sizeRecord)
return


if __name__ == '__main__':
filename = '/home/john/zRamdisk/Data/EUR.USD-CASH-IDEALPRO.scid'
tzvar = 0
if len(sys.argv) > 1:
filename = sys.argv[1]
if len(sys.argv) > 2:
print(sys.argv[2], type(sys.argv[2]))
if sys.argv[2] == 'l':
tzvar = 99999
print(tzvar, type(tzvar))
else:
tzvar = float(sys.argv[2])
print(tzvar, type(tzvar))
# filename = 'all' ############## remove ##########################
if filename != "all":
convertRecords(filename, filename[:-4] + 'csv', tzvar, 1)
else:
files = os.listdir(path='.')
for filename in files:
if filename[-5:] == '.scid':
print("Converting this file", filename, "using Tz", tzvar)
convertRecords(filename, filename[:-4] + 'csv', tzvar, 1)
[2021-06-19 06:29:49]
[Go To First Post]
biconoid - Posts: 5
Hi Kiwi, Thank you so much for your code. It was well written and consie.
Upon running, i'm still seeing the same issue of
- datatime reported from 1899, SCDateTime epoch of December 30, 1899 in column[0] and
- timestamp as 000000 in column[1]

So i've a few questions going fwd.
- eur.usd data you're mentioning, it is spot FX market?
- Time based or tick based?
- If time based, what's the time series difference?
- Have you ran this with futures data?

In my case, i have only tried to run on Futures/FX obtained by Denali feed
- 1 tick ES (SC v2232)
- 10 tick GC (SC v2275)
- 10 seconds EURUSD) (SC v2275)

In all above cases i get wrong datetime value.
Output
Date,Time,O,H,L,C,V,T
18991230,000000,1.1748149394989014,1.1750800609588623,1.1748149394989014,1.17507004737854,16,16
18991230,000000,1.1750950813293457,1.1751049757003784,1.1750850677490234,1.1751049757003784,4,4
18991230,000000,1.1751099824905396,1.1752899885177612,1.1751099824905396,1.1752899885177612,16,16
18991230,000000,1.1752749681472778,1.1752749681472778,1.1752500534057617,1.1752500534057617,6,6
18991230,000000,1.1752550601959229,1.1752550601959229,1.1752550601959229,1.1752550601959229,1,1
18991230,000000,1.1752400398254395,1.1752500534057617,1.1752300262451172,1.1752500534057617,4,4
18991230,000000,1.1752400398254395,1.1752400398254395,1.175225019454956,1.175225019454956,3,3
NORMAL EURUSD.csv utf-8[dos] 0% ㏑:1/5635752
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Date,Time,O,H,L,C,V,T
18991230,000000,1548.800048828125,1548.800048828125,1548.800048828125,1548.800048828125,1,1
18991230,000000,1556.7000732421875,1556.7000732421875,1556.7000732421875,1556.7000732421875,1,1
18991230,000000,1552.0,1552.0,1552.0,1552.0,1,1
18991230,000000,1534.7000732421875,1534.7000732421875,1534.7000732421875,1534.7000732421875,1,1
18991230,000000,1595.0,1595.0,1595.0,1595.0,1,1
18991230,000000,1485.0,1485.0,1485.0,1485.0,1,2
18991230,000000,1503.0,1503.0,1503.0,1503.0,1,1
NORMAL GCQ21-COMEX.csv utf-8[dos] 0% ㏑:1/191884
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Date,Time,O,H,L,C,V,T
18991230,000000,0.0,0.0,2623.0,2623.0,1,1
18991230,000000,0.0,0.0,2717.0,2717.0,1,1
18991230,000000,0.0,0.0,2750.0,2750.0,1,1
18991230,000000,0.0,2757.75,2742.5,2742.5,1,1
18991230,000000,0.0,2830.0,2805.75,2830.0,1,5
18991230,000000,0.0,2845.0,2814.75,2845.0,1,3
18991230,000000,0.0,2845.0,2814.75,2845.0,1,2
NORMAL ESM21-CME.csv utf-8[dos] 0% ㏑:1/191884
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────


Is there a subset of SCID file you're using in your testing and maybe can share probably first 2-3 MB.
Date Time Of Last Edit: 2021-06-19 06:49:55
[2021-06-19 11:46:40]
[Go To First Post]
Kiwi - Posts: 374
Its old code and it was working early 2020 but I notice it doesn't work on current data so SC presumably changed how they stored times. I will have a look at it and figure it out but it might take a week.

I assume it relates to the 2151 change so it might be quite simple. I'll look tomorrow & if its easy I'll update, otherwise it could take a while.


DateTime
[Link] - [Top]
The DateTime member variable is a SCDateTimeMS variable.

This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899.

Versions prior to 2151, use a double precision floating point type variable to represent Date-Time values. For a complete explanation of the Date component of this value, refer to Date Value. The date value is the integer portion of the double. The fractional portion is the Time value which is represented as a fraction of one day where 1/86400000 is 1 ms. 86400000 is the number of milliseconds in a day.

This Date and Time value is and must be in the UTC time zone.

In Sierra Chart, if the Intraday Data Storage Time Unit in Global Settings >>Data/Trade Service Settings is set to a value greater than 1 Second like 1 Minute and the first trade within a new minute begins at a time after the beginning of the minute, for example 9:28:22, the time value part of the DateTime member in the record will be set to this time.

A new record will begin if there is any trading on or after 9:29:00 and not 9:29:22. The data is aligned on time boundaries which are multiples of the Intraday Data Storage Time Unit.

When writing tick by tick data to an Intraday data file, where each tick is a single record in the file, and multiple ticks/trades have the same timestamp, then it is acceptable for the Date-Time of the record to be the same timestamp as prior records. What Sierra Chart itself does in this case, is increment the microsecond portion of the Date-Time to create unique timestamps for each trade within a millisecond.

Sierra Chart provides the /ACS_Source/SCDateTime.h file with various functions for working with these Date-Time values.
Date Time Of Last Edit: 2021-06-19 11:51:54
[2021-06-19 12:48:50]
[Go To First Post]
biconoid - Posts: 5
Thank you Kiwi. No rush and as your time permits.

Yes it seems related to newer implemtation to microseconds and highly unlikey but possible gap in documentation.
It's good to know it's a genuine problem and a step in the right direction.
I'll also try to read more into the Sierra header files, worth exploring.
Date Time Of Last Edit: 2021-06-19 12:50:11
[2021-06-19 14:47:59]
[Go To First Post]
User907968 - Posts: 802
You wrote the information necessary to answer your question in the original message, but then used conflicting information when constructing your code.


Trying to read Sierra Chart SCID file using Python and export as CSV/TSV

Using following approach:

With reference to s_IntradayRecord Structure Member Descriptions at
Intraday Data File Format

- The Intraday data record structure, s_IntradayRecord, is 40 bytes in size.
- The Intraday data file header, s_IntradayHeader, is 56 bytes in size.
- The DateTime member variable is a SCDateTimeMS variable. This is a 64-bit integer.
It represents the number of microseconds since the SCDateTime epoch of December 30, 1899.

"SCDateTimeMS variable. This is a 64-bit integer" which means you need to use 'q' as opposed to 'd' to unpack ('q4f4I' not 'd4f4I') - q being correct for 64-bit integer (long long)

If you want the data in excel format, one method is to simply divide the datetime variable returned by the number of microseconds per day.

For example, using the first value from your data.scid file, 3795467579000000 -> 43929.022905 -> 08/04/2020 00:32:59
[2021-06-19 20:15:37]
[Go To First Post]
biconoid - Posts: 5
Hi User907968, Thank you very much for your helpful and precise pointers.

As a result i've tried to approach the problem below. Now quite close the output.
- Date Part is working fine for all values provided from SCID.
- Time part still is off by a little. Could you please check where could be the issue?

"""
The DateTime member variable is a SCDateTimeMS variable.
This is a 64-bit integer. It represents the number of microseconds since the SCDateTime epoch of December 30, 1899."""

import numpy as np

def calculate_time_from(scid_date_time, zone=0):
# Constant values
sec_per_min, min_per_hour, hours_per_day = (60, 60, 24)
sec_per_hour = sec_per_min * min_per_hour # 3600
sec_per_day = sec_per_hour * hours_per_day # 86400
u_sec_per_day = sec_per_day * 1000_000 # 86400_000_000

excel_epoch_time = np.datetime64('1899-12-30') # excel epoch date
scid_date_time = int(scid_date_time) # d[0] value from struct.unpack('<q 4f 4I')
time_zone = int(zone * min_per_hour) # UTC value, e.g +5,

# Components to get full time
time_elapsed = scid_date_time / u_sec_per_day # total time elapsed since Dec 30, 1899
date_part, time_part = str(time_elapsed).split('.') # split integer(date) and fraction(time) parts

# Calculate delta in numpy_values
delta_days = np.timedelta64(date_part, 'D') # timedelta in days for date_part
delta_us = np.timedelta64(time_part, 'us') # timedelta in micro seconds for time_part
delta_timezone = np.timedelta64(time_zone, 'm') # timedelta in minutes for time_zone

my_time = excel_epoch_time + (delta_days + delta_us + delta_timezone) # final excel datetime value
date_val, time_val = str(my_time).split('T') # individual date and time values

return date_val, time_val
------------------------------

# print(calculate_time_from(3795467579000000.0, -0.5)) # Output: ('2020-04-08', '00:08:10.509259')
# print(calculate_time_from(3795467579000000.0, 0)) # Output: ('2020-04-08', '00:38:10.509259')
# print(calculate_time_from(3795467579000000.0, 0.5)) # Output: ('2020-04-08', '01:08:10.509259')


# print(f'Time: {my_time}') # Time: 2020-04-08T00:38:10.509259
# print(f'scid_date: {scid_date_time}') # scid_date: 3795467579000000
# print(f'time_elapsed: {time_elapsed}') # time_elapsed: 43929.02290509259
# print(f'date_part: {date_part}') # date_part: 43929
# print(f'time_part: {time_part}') # time_part: 02290509259
# print(f'np_units: {np.timedelta64(date_part)}') # np_units: 43929 generic time units
# print(f'np_units: {np.timedelta64(time_part)}') # np_units: 2290509259 generic time units
[2021-06-19 23:20:51]
[Go To First Post]
Kiwi - Posts: 374
Here is an update with a new dtformat (2) for data stored after SC2151. It could be modified to handle both but I don't use it so I've kept the old code and added a new branch when dtformat is set to 2 (as it is here). Let me know if it works with your files. I didn't worry about the sub second timestamps so they will all be the same timestamp (but with the data in the correct sequence) in the output csv file.

Note that it won't work on older files unless you change the format in the covertRecords calls from 2 back to 1.

"""
Takes a SierraChart scid file (input argument 1) and converts
it to a comma separated file (comma)

Timezone conversion can follow the users local timezone, or a
specified integer (input l or an integer but if the default
filename is being used, '' must be specified for the filename)

Inputs: filename & tz adjust (else UTC).
if the filename is 'all' then all scids in directory are converted
"""
import csv
import re
import sys
import os
import struct
from datetime import date, datetime, time, timedelta


def deserialize(excelDateAndTime):
"""
Returns excel date time as Python datetime object
"""
date_tok = int(excelDateAndTime)
time_tok = round(86400*(excelDateAndTime - date_tok))
d = date(1899, 12, 30) + timedelta(date_tok)
t = time(*helper(time_tok, (24, 60, 60, 1000000)))
return datetime.combine(d, t)


def helper(factor, units):
factor /= 86399.99
result = list()
for unit in units:
value, factor = divmod(factor * unit, 1)
result.append(int(value))
result[3] = int(0)
return result


def datetimeTwo(excelDateAndTime, tzAdjust):
"""
Returns excel date time in format YYYYMMDD HHMMSS
"""
t = str(deserialize(excelDateAndTime + tzAdjust.seconds/86400.0))
return (re.sub(r'(:|-)', '', t)[:8], re.sub(r'(:|-)', '', t)[9:15])


def excelTimeAdjust(date_and_time, tzAdjust):
return date_and_time + tzAdjust.seconds/86400.0


def convertRecords(filename, fileoutput, tzvar, dtformat):
"""
Read in records from SierraChart .scid data filename
dtformat determines the datetime representation in column 1 (and 2)
"""
sizeHeader = 0x38
sizeRecord = 0x28
if tzvar == 99999:
tzAdjust = datetime.fromtimestamp(0) - datetime.utcfromtimestamp(0)
else:
tzAdjust = timedelta(hours=tzvar)

header = ('Date', 'Time', 'O', 'H', 'L', 'C', 'V', 'T')
ftsv = open(fileoutput, 'w')
fileout = csv.writer(ftsv, delimiter=',')
with open(filename, 'rb') as fscid:
fscid.read(sizeHeader) # discard header
fileout.writerow(header)
data = fscid.read(sizeRecord)
while data:
if dtformat == 2: # post SC2151 Microsecond timing
dataRow = struct.unpack('q4f4I', data)
date, time = datetimeTwo(dataRow[0]/86400000000, tzAdjust)
outrow = (date, time,
str(dataRow[1]), str(dataRow[2]), str(dataRow[3]),
str(dataRow[4]), str(dataRow[5]), str(dataRow[6]))
fileout.writerow(outrow)
else:
dataRow = struct.unpack('d4f4I', data)
if dtformat == 0:
adjustedTime = dataRow[0] + tzAdjust.seconds/86400.0
outrow = (str(adjustedTime), str(adjustedTime),
str(dataRow[1]), str(dataRow[2]), str(dataRow[3]),
str(dataRow[4]), str(dataRow[5]), str(dataRow[6]))
fileout.writerow(outrow)
elif dtformat == 1:
date, time = datetimeTwo(dataRow[0], tzAdjust)
outrow = (date, time,
str(dataRow[1]), str(dataRow[2]), str(dataRow[3]),
str(dataRow[4]), str(dataRow[5]), str(dataRow[6]))
fileout.writerow(outrow)
data = fscid.read(sizeRecord)
return


if __name__ == '__main__':
filename = '/home/john/zRamdisk/Data/EUR.USD-CASH-IDEALPRO.scid'
# filename = '/home/john/zRamdisk/Data/HSI-202106-HKFE.scid'
tzvar = 0
if len(sys.argv) > 1:
filename = sys.argv[1]
if len(sys.argv) > 2:
print(sys.argv[2], type(sys.argv[2]))
if sys.argv[2] == 'l':
tzvar = 99999
print(tzvar, type(tzvar))
else:
tzvar = float(sys.argv[2])
print(tzvar, type(tzvar))
# filename = 'all' ############## remove ##########################
if filename != "all":
convertRecords(filename, filename[:-4] + 'csv', tzvar, 2)
else:
files = os.listdir(path='.')
for filename in files:
if filename[-5:] == '.scid':
print("Converting this file", filename, "using Tz", tzvar)
convertRecords(filename, filename[:-4] + 'csv', tzvar, 2)
[2021-09-03 09:41:21]
[Go To First Post]
User155017 - Posts: 41
Hi,
is there a way to generate automatic trades from spreadsheets using the custom fields added to the files for buy and sell signals?
In general, how do you submit the order using the signal saved in the files generated in python?

Thanks
[2021-09-25 18:27:00]
[Go To First Post]
User155017 - Posts: 41
How do you send orders using python? You have written a c++ code to read the signal from file and submit from sierra?
Do you have any example?
[2021-09-25 21:33:45]
[Go To First Post]
Kiwi - Posts: 374
My order sending code has always been in C++ dlls to gain access to Sierra's rich interface.

Communication of intent from Python can be by way of shared text files to pipe the commands in and responses out. Your dll can use these as a pipeline to SC's ordering api.
[2021-11-23 19:11:53]
[Go To First Post]
User155017 - Posts: 41
Thanks. That is what I suspected. Is there a sample c++ code to look at which reads commands from files to send and modify orders?
Could it be made general enough to manage several order independently?
It would be nice to have a full api support to trade from a jupyter notebook, such as ibinsync for IB or web api of tradestation.
[2021-11-25 00:45:04]
[Go To First Post]
Kiwi - Posts: 374
My code is completely focussed on my personal approach to this so, sorry, but I won't risk sharing it. However the ACSIL instructions are fairly clear and as long as you test what you're doing should work pretty well.
[2022-01-15 09:26:26]
[Go To First Post]
User155017 - Posts: 41
Using txt files to communicate with an external program is not as fast using the DTC server I suppose, but in absence of a working open source DTC client which would allow to trade directly from Python, it looks like the best practical solution.
Using dtc would be much faster?

I have some code in Python using multi-threading, could I use multi-threading in the dll to use in SC in case i would re-write it in c++?
For now the file interface sounds the simplest solution, since I am not doing hft .. but I would like to know what could be the time gain in using a DTC server.
Ideally if the dll can support multi-processing/threading it would be faster and no external communication would be required, but re-writing and debugging the code again is too much work for now.
[2022-01-15 23:11:43]
[Go To First Post]
Kiwi - Posts: 374
I suspect the delay between SC calling the dll would be much greater than the time required to access a short file but file access time depends on what its stored on - mine are in my data directory which is on a linux ramdisk so that might not be much longer than the dtc access but if it was on a hdd that was spun down it could be quite slow.

Just doing a python read of a short file off my ramdisk shows about 10 microseconds to open, read, and close, however a different method shows 100 microseconds, which is still fast.

https://i.imgur.com/PPLKQ3s.png

Edit: a later test in C suggests around 45 microseconds so the first reading must be an error due to my testing method.
Date Time Of Last Edit: 2022-01-17 08:30:09
[2022-01-16 14:32:33]
[Go To First Post]
user2837 - Posts: 76
Thanks Kiwi, lots of great info here.

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account