Login Page - Create Account

Support Board


Date/Time: Fri, 03 Feb 2023 23:02:06 +0000



[User Discussion] - Python for Sierra Chart

View Count: 25108

[2014-07-10 03:45:46]
Kiwi - Posts: 372
OK.

I had read that and didn't see why HDF5 was chosen. Python is extremely happy reading CSVs (or even SCIDs with a little conversion).
[2014-07-10 08:09:53]
ganz - Posts: 1048
Kiwi

The CSV is good but we need to ask SC Team to support ISO 8601.

http://pandas.pydata.org/pandas-docs/version/0.13.1/io.html#io-read-csv-table

read_csv has a fast_path for parsing datetime strings in iso8601 format, e.g “2000-01-01T00:01:02+00:00” and similar variations. If you can arrange for your data to store datetimes in this format, load times will be significantly faster, ~20x has been observed.
This has been tested and proved. (~16x in my case)

And yes, I'm happy to use scid for charting any data I need.

Another "+" for HDF5 : the format has an internal "file system". It makes much more easy to manage thousands of symbols, it makes users feel comfortable with a db, I believe.
[2014-07-10 13:14:28]
ganz - Posts: 1048
for the record:

in order to use the script on a x64 system it needs to define the byte order for the struct function as explained there https://docs.python.org/3.0/library/struct.html for little-endian

in other words use '='or '<':

src = struct.unpack('=d4f4L', tick)

or

src = struct.unpack('<d4f4L', tick)


[2014-07-12 05:34:31]
ganz - Posts: 1048
Kiwi

Why use HDF5?

http://pytables.github.io/FAQ.html#why-hdf5


Why HDF5?

HDF5 is the underlying C library and file format that enables PyTables to efficiently deal with the data. It has been chosen for the following reasons:

Designed to efficiently manage very large datasets.
Lets you organize datasets hierarchically.
Very flexible and well tested in scientific environments.
Good maintenance and improvement rate.
Technical excellence (R&D 100 Award).
It’s Open Source software


[2014-08-14 22:05:43]
User13778 - Posts: 24
Excellent work Kiwi

I use Python to analyse NQ and ES and trade with Interactive Brokers.

Im looking to do similar with a python / Sierra Chart system on FXCM
[2014-10-30 14:09:26]
User85468 - Posts: 9
Hello,

how can I have my PnF chart (or reversal bar chart), say in 15 minute periods, print and/or remove Xs or Os depending on the direction, for every price move on the last bar? It works on the Tick, but the chart is really worthless in such a short time-frame.

Kind regards
[2014-11-22 23:30:23]
ganz - Posts: 1048
Why use HDF5?

http://youtu.be/hnhN2_TpY8g
[2015-04-01 14:27:50]
Shaharabin - Posts: 28
Hi Guys,

Interesting stuff.

My question to you all is; can i export real time "Time and Sales" data from sierra chart and use it with any other program? using any type of language.

My end goal is to stream that data onto the PC's serial port and build a physical gauge

Thank for any replay
Shahar.
[2015-04-01 18:46:10]
Sierra Chart Engineering - Posts: 104357
In regards to post #32 something that may be of help for this, is the relay server:
https://www.sierrachart.com/index.php?page=doc/doc_RelayServer.php

Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2015-04-01 20:03:51]
Shaharabin - Posts: 28
Holy-S***
This is great stuff, how did I not know about it....
I'll start working on it, which means, a whole lot of new questions coming your way :)

Thanks a lot!
[2015-04-01 22:22:56]
Sierra Chart Engineering - Posts: 104357
If you have further questions for Sierra Chart support, post them in a new thread because we are not monitoring this thread.


Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. Try to keep your questions brief and to the point. Be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

For the most reliable, advanced, and zero cost futures order routing, *change* to the Teton service:
Sierra Chart Teton Futures Order Routing
[2017-07-17 20:26:58]
rhovega - Posts: 279
Hi @ganz hoping you are still around this forum and get to see this!

I am new to Python; running your code from post 22 gives me error "IndexError: list index out of range", while running code "~/>python3 this_script.py chart.scid" (with the appropriate filenames) from the Python console gives me error "SyntaxError: invalid syntax". Can't figure it out.

I am aware this is a very old thread but maybe if still around you could give me some pointers to move along. Would be much appreciated.

Thank you.
[2017-07-22 09:10:57]
ganz - Posts: 1048
rhovega

hey ...

the only way in this case is to learn what each of line exactly does
study Python step by step in order to get it working

imo :)
[2019-03-05 19:33:57]
Chad - Posts: 221
@Kiwi, have you looked into adding a function within your Python script for reading market depth (.depth) files?
[2019-03-05 22:45:49]
Kiwi - Posts: 372
No CB, I trade purely on price vs time

My latest work on Python has been a program using ib_insync to connect to the twsapi directly but its for basic information presentation & order entry/management to complement TWS charting.


https://i.imgur.com/nRbivpe.png
[2019-05-10 01:25:52]
Kiwi - Posts: 372
So here's another little bit of code.

It will convert all the text files in a directory into scid files in another directory. If the constant KEEP_UPDATING was True it will then check the text files every REFRESH seconds and if they've changed it will update the scid files.

Set FILES_IN and FILES_OUT. In windows the format is "C:\\mydir\\mysubdir\\"

Also set Timezone if you wish to change the times in Sierra's scid files (Sierra saves them in the UTC time zone).

Save the code in a file named something like Text2Scid.py and run it using python Text2SCid.py.
Tested with Python 3.7.Have fun:

"""
Convert's Text files to SCID files for SierraChart
It will create a new scid file for every text file it finds
format is assumed to be D T O H L C V T with comma separated variables
It will continue to update the scid file every REFRESH seconds
generally refresh would be set to 1/2 the time period for the text file updates
"""

import os
import struct
from time import sleep
from datetime import datetime as dt, timedelta as tdelta
import subprocess

# these are linux format, windows might need to use double backslashes \\
FILES_IN = "/home/john/Python/Trading/textfiles/"
FILES_OUT = "/home/john/zRamdisk/SierraData/"
TIMEZONE = 0 # scid saved as UTC, use this if the text file is another time zone
KEEP_UPDATING = True # set to False to convert and stop; True for live updating
REFRESH = 0.25 # refresh every 1/4 second for 1/2 second updating text files
TEXT_FILE_SUFFIX = ".txt" # usually .txt or .csv

sizeHeader = 0x38
sizeRecord = 0x28
file_header = (b'SCID8\x00\x00\x00\x28\x00\x00\x00\x01\x00\x00\x00\x00\x00' +
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' +
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' +
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')


def excel_date(datetime, offset=TIMEZONE):
"""convert date&time with offset in hours to excel format for scid"""
delta = datetime - dt(1899, 12, 30) + tdelta(hours=offset)
return delta.days + delta.seconds / 86400


def python_date(exceldt, offset=TIMEZONE):
"""convert date&time with offset in hours to python format"""
excel_as_delta = tdelta(days=exceldt//1) + tdelta(seconds=exceldt%1*86400)
return dt(1899, 12, 30) + excel_as_delta + tdelta(hours=offset)

def write_struct(record, f):
if len(record) == 8 and record[0][0].isnumeric():
out = [excel_date(dt.strptime(f'{record[0]} {record[1]}', '%Y-%m-%d %H:%M:%S.%f'))]
out.extend([float(s) for s in record[2:]])
# note that SC packs dt, O, H, L, C, T, V not V, T so swap them
wt = struct.pack('d4f4I', out[0], out[1], out[2], out[3], out[4], int(out[6]), int(out[5]), 0, 0)
f.write(wt)


def create_base(f):
record = None
with open(FILES_OUT + f[: -3] +'scid', "wb") as scidfile:
scidfile.write(file_header)
with open(FILES_IN + f, "r") as txtfile:
for line in txtfile:
record = line.split(',')
write_struct(record, scidfile)
return f'{record[0]} {record[1]}' # will update when date+time changes


def update_if_changed(files):
for f in files:
with open(FILES_IN + f, "r") as txtfile:
record = txtfile.readlines()[-1].split(',')
last_dt = f'{record[0]} {record[1]}'
if files[f] == last_dt:
continue
files[f] = last_dt
with open(FILES_OUT + f[: -3] +'scid', "ab") as scidfile:
write_struct(record, scidfile)


if __name__ == '__main__':
files = sorted(k for k in os.listdir(path=FILES_IN) if k[-4:] == TEXT_FILE_SUFFIX)
files = {k: '' for k in files}
for f in files:
files[f] = create_base(f)
print('finished base')
if KEEP_UPDATING:
# now test each input file each REFRESH interval & update if datetime changed
while True:
sleep(REFRESH)
update_if_changed(files)


edit: changed so that it will run despite the extra space & linefeed added after each line. Probably better to use the attached file though.
Date Time Of Last Edit: 2019-05-11 12:42:07
attachmentText2Scid.py - Attached On 2019-05-11 12:41:30 UTC - Size: 3.31 KB - 460 views
[2019-05-10 06:02:42]
User21223 - Posts: 89
Hi
Can anyone check the above code. Because I'm getting an error. I do not know if it's my part.


C:\Users\mkruss>python Text2Scid.py
File "Text2Scid.py", line 33
file_header = b'SCID8\x00\x00\x00\x28\x00\x00\x00\x01\x00\x00\x00\x00\x00\x0
0' \

^
SyntaxError: unexpected character after line continuation character



Sample text file to be converted into a scid in the attachment
Date Time Of Last Edit: 2019-05-10 06:12:59
Attachment Deleted.
attachmentSymbol2.txt - Attached On 2019-05-10 06:11:49 UTC - Size: 18.34 KB - 423 views
[2019-05-10 06:40:08]
Kiwi - Posts: 372
Yes. It seems this board adds extra characters (an extra space and a linefeed) at the end of the line when you cut & paste from it. So here's a raw version without extra characters.
Date Time Of Last Edit: 2019-05-10 06:46:10
attachmentText2Scid.py - Attached On 2019-05-10 06:39:58 UTC - Size: 3.58 KB - 472 views
[2019-05-10 08:31:55]
User21223 - Posts: 89
Now. This error is shown

C:\Users\mkruss>python Text2Scid.py
finished base
Traceback (most recent call last):
File "Text2Scid.py", line 84, in <module>
update_if_changed(files)
File "Text2Scid.py", line 60, in update_if_changed
record = subprocess.check_output(['tail', '-1', FILES_IN + f]).decode().spli
t(',')
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 395, in check_output
**kwargs).stdout
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 1178, in _execute_chi
ld
startupinfo)
FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku

C:\Users\mkruss>


I will add that the scid files were created in the Siarrachart directory. You can load them by Open Intraday Chart

But they do not freshen up automatically
[2019-05-10 08:37:59]
Kiwi - Posts: 372
Make sure your file paths have no spaces in them. For example SierraChart or Sierra_Chart, not Sierra Chart. Some python modules have problems with them so it's easier not to include spaces in paths.
Date Time Of Last Edit: 2019-05-10 08:43:08
[2019-05-10 08:59:25]
User21223 - Posts: 89
I had Sierra Chart. I changed to SierraChart.
but the problem still occurs. Same error
[2019-05-10 09:35:47]
Kiwi - Posts: 372
After the line if name == main add this and post the result:

import sys; print(sys.version)

I'll do another version tomorrow that's more defensive to see if the problem is detectable.
[2019-05-10 09:51:56]
User21223 - Posts: 89
Microsoft Windows [Wersja 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Wszelkie prawa zastrzeżone.

C:\Users\mkruss>python Text2Scid.py
3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)]
finished base
Traceback (most recent call last):
File "Text2Scid.py", line 85, in <module>
update_if_changed(files)
File "Text2Scid.py", line 60, in update_if_changed
record = subprocess.check_output(['tail', '-1', FILES_IN + f]).decode().spli
t(',')
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 395, in check_output
**kwargs).stdout
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 1178, in _execute_chi
ld
startupinfo)
FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku
[2019-05-10 12:55:35]
Kiwi - Posts: 372
I'll do a new version tomorrow but you should be able to get it done by replacing


record = subprocess.check_output(['tail', '-1', FILES_IN + f]).decode().split(',')

with:


record = txtfile.readlines()[-1].split(',')


edit: corrected version of the file on Post #41
Date Time Of Last Edit: 2019-05-11 23:47:48

To post a message in this thread, you need to log in with your Sierra Chart account:

Login

Login Page - Create Account