Login Page - Create Account

Support Board


Date/Time: Fri, 23 Apr 2021 06:11:07 +0000



[User Discussion] - Python for Sierra Chart

[2013-06-13 03:00:17]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
For a while I've been looking at moving my development to Python rather than C++.

To enable this required a few extra bits and pieces that I'll post to the board for anyone interested. The original post is here.

The attached file runs with Python3 (might run with 2 as well). It:

1. Reads an SCID file.
2. Converts it to a pandas dataframe (for time series manipulations)
3. Writes it back to an SCID file leaving the T, BV and AV fields free so that they can be used for commands to C++ code to indicate conditions on the SC chart or to act on Sierra (place orders etc).

An entire read/convert/write loop takes under 1ms on a 3.2GHz machine so there is no appreciable lag between reading a new tick and writing it to the output file.

.
attachmentSCID_to_DF_RT.py - Attached On 2013-06-13 02:59:38 UTC - Size: 10.03 KB - 1007 views
[2013-06-13 19:24:24]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
Hello sir

Thank you for the file. Very interesting.

gd lck
[2013-06-20 23:10:17]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
Updated Version with new methods in for converting the dataframe to a longer timeframe and for mapping longer timeframe development onto the lower timeframe.

#!/usr/bin/python3
from __future__ import print_function
import numpy as np
import pandas as pd
import struct
import sys
from time import sleep, time

o = O = 'O'
h = H = 'H'
l = L = 'L'
c = C = 'C'
v = V = 'V'
x = 'x'
y = 'y'
z = 'z'

time_list = []
overrun_list = []
overruns = 0

lt = 15
mt = 5
st = 1

ohlc = {o: 'first', h: 'max', l: 'min', c: 'last',
v: 'sum', x: 'sum', y: 'sum', z: 'sum'}
cols = [O, H, L, C, V, x, y, z]
time_list = []



class SierraFile(object):
""" """
def __init__(self, filename):
self.filename = str(filename)
# self.tzAdjust = t imedelta(hours=+10).seconds/d2s
self.tzAdjust = np.timedelta64(10, 'h') / np.timedelta64(1, 'D')
self.excelDate = np.datetime64('1899-12-30')
self.sizeHeader = 0x38
self.sizeRecord = 0x28
self.pos = 0
self.last = 0

def read_existing_records(self):
with open(self.filename, 'rb') as fscid:
fscid.read(self.sizeHeader) # discard header
rows = []
ts = []
for i in range(1000000):
data = fscid.read(self.sizeRecord)
if data not in ('', b''):
d = struct.unpack('d4f4I', data)
dt = d[0] + self.tzAdjust
ts.append(self.excelDate + np.timedelta64(int(dt))
+ (np.timedelta64(int(round((dt - int(dt))
* 86400)), 's')))
datarow = [d[1], d[2], d[3], d[4], d[5], 0, 0, 0]
rows.append(datarow)
else:
break
self.pos = self.last = fscid.tell()
return (ts, rows)

def read_record(self):
global overruns, overrun_list
with open(self.filename, 'rb') as fscid:
fscid.seek(0, 2) # Go to the end of the file
self.last = fscid.tell()
if self.last == self.pos: # no new data >> nothing to do
return (-999, 0, 0)
else: # data to collect
if self.pos < self.last - self.sizeRecord: # > 1 record
print('Overrun', self.last - self.pos,
(self.last - self.pos) / self.sizeRecord)
overruns += 1
overrun_list.append(np.datetime64('now'))
late_flag = True
else:
late_flag = False
fscid.seek(self.pos, 0)
self.pos += self.sizeRecord
data = fscid.read(self.sizeRecord)
d = struct.unpack('d4f4I', data)
dt = d[0] + self.tzAdjust
new_time = (self.excelDate + np.timedelta64(int(dt))
+ (np.timedelta64(int(round((dt - int(dt))
* 86400)), 's')))
datarow = [d[1], d[2], d[3], d[4], d[5], 0, 0, 0]
return (new_time, datarow, late_flag)

def write_existing_records(self, dataframe):
with open(self.filename, 'wb') as fscid:
header = b'SCID8\x00\x00\x00(\x00\x00\x00\x01\x00'
fscid.write(header)
for i in range(21):
fscid.write(b'\x00\x00')
for i in range(dataframe.end):
da = ((dataframe.df.index.values[i] - self.excelDate)
/ np.timedelta64(1, 'D') - self.tzAdjust)
db, dc, dd, de, df, dg, dh, di = dataframe.df.iloc[i]
di = 0x11100111
df = int(df)
dg = int(dg)
dh = int(dh)
di = int(di)
wt = struct.pack('d4f4I', da, db, dc, dd, de, df, dg, dh, di)
fscid.write(wt)

def write_record(self, dataframe):
with open(self.filename, 'ab') as fscid:
i = dataframe.end - 1
da = ((dataframe.df.index.values[i] - self.excelDate)
/ np.timedelta64(1, 'D') - self.tzAdjust)
db, dc, dd, de, df, dg, dh, di = dataframe.df.iloc[i]
di = 0x88300388
df = int(df)
dg = int(dg)
dh = int(dh)
di = int(di)
record = struct.pack('d4f4I', da, db, dc, dd, de, df, dg, dh, di)
fscid.write(record)


class SierraFrame(object):
"""
DataFrame is the basic object for analysis:
init reads the .scid file into the initial object, 5 sec assumed
extend_frame adds 5000 rows to the df because appending rows is slow
add appends new data in the extended frame for real time operation
build_tf creates a new dataframe that is a multiplier of the input df
build_htf_array creates an array showing higher timeframe bars as
they develop for the lower timeframe array
countfloats is a test method
"""
def __init__(self, time_index, data):
self.df = pd.DataFrame(data, index=time_index,
columns=[O, H, L, C, V, x, y, z])
self.end = len(self.df)
self.pos = 0

def extend_frame(self):
'''
Create a 5000 row array from last time in self.df
and append it to self.df
Remove lunch break from array
'''
print('Extending DataFrame Now')
s5 = np.timedelta64(5, 's')
h1 = np.timedelta64(1, 'h')
sl = np.datetime64('today') + np.timedelta64(14, 'h')
el = np.datetime64('today') + np.timedelta64(15, 'h')
start_time = self.df.index.values[self.end - 1]
dtgen = ((start_time + i * s5) for i in range(1, 5000))
dtstrip = ((i + h1 if sl <= i < el else i) for i in dtgen)
dg = pd.DataFrame(index=dtstrip, columns=self.df.columns)
#dg.iloc[:] = 0.0
#dg[[v, x, y, z]] = dg[[v, x, y, z]].astype('int')
self.df = self.df.append(dg)
self.df = self.df.astype(np.float64)

def add(self, new_time, datarow):
'''
Add a row to an existing extended df but:
extend if its within 5 of the end
fill with last bar if its not the next bar
convert the four integer columns to float for df speed of access
'''
if self.end > len(self.df) - 5:
self.extend_frame() # not needed if first fill > day length
np_time = np.datetime64(new_time)
if np_time < self.df.index.values[self.end]:
return # new data is earlier than current
while np_time > self.df.index.values[self.end]:
self.df.iloc[self.end] = self.df.iloc[self.end - 1]
self.end += 1 # fill with prior row if new is later
for i in [4, 5, 6, 7]:
datarow[i] = float(datarow[i])
self.df.iloc[self.end] = datarow # fill when times match
#self.df.iloc[self.end] = self.df.iloc[self.end].astype(np.float64)
self.end += 1

def build_tf(self, ht):
'''
Create higher timeframe df that is a multiplier of the input, di
with ht being the high timeframe bar length in minutes
'''
return self.df.resample(str(ht)+'min', how=ohlc)[cols]

def build_htf_array(self, st, ht):
'''
Map higher timeframe development on to input df
with ht being the high timeframe bar length in minutes
'''
di = self.df.resample(str(st)+'min', how=ohlc)[cols]
dih = di.iloc[:,0:5]
for i in range(len(dih)):
if i == 0 or i//ht > (i-1)//ht:
bO = dih.iloc[i, 0]
bH = dih.iloc[i, 1]
bL = dih.iloc[i, 2]
bC = dih.iloc[i, 3]
else:
dih.iloc[i, 0] = bO
dih.iloc[i, 1] = bH = max(bH, dih.iloc[i, 1])
dih.iloc[i, 2] = bL = min(bL, dih.iloc[i, 2])
bC = dih.iloc[i, 3]
return dih

def countfloats(self):
length = len (self.df)
width = len(self.df.iloc[0])
floats = 0
nonfloats = 0
for i in range(length):
for j in range(width):
if isinstance(self.df.iloc[i,j], float):
floats += 1
else:
nonfloats += 1
return (floats, nonfloats)

def build_htf_array(di, ht):
'''
Map higher timeframe development on to input df
with ht being the high timeframe bar length in minutes
'''
dih = di.iloc[:,0:5].copy()
for i in range(len(dih)):
if i == 0 or i//ht > (i-1)//ht:
bO = dih.iloc[i, 0]
bH = dih.iloc[i, 1]
bL = dih.iloc[i, 2]
bC = dih.iloc[i, 3]
else:
dih.iloc[i, 0] = bO
dih.iloc[i, 1] = bH = max(bH, dih.iloc[i, 1])
dih.iloc[i, 2] = bL = min(bL, dih.iloc[i, 2])
bC = dih.iloc[i, 3]
return dih

def build_tf(di, ht):
'''
Create higher timeframe df that is a multiplier of the input, di
with ht being the high timeframe bar length in minutes
'''
return di.resample(str(ht)+'min', how=ohlc)[cols]



def SierraRun():
global time_list
time0 = time()
#filename = '/home/john/zRamdisk/SierraChart/Data/HSI-201306-HKFE-TD.scid'
filename = '/home/john/zRamdisk/SierraChart/Data/HSIM13-FUT-HKFE-TD.scid'
hsi = SierraFile(filename)
time_index, data = hsi.read_existing_records()
da = SierraFrame(time_index, data)
import ipdb; ipdb.set_trace() # XXX BREAKPOINT
da.extend_frame()
wtst = SierraFile('/home/john/zRamdisk/SierraChart/Data/HSI-INPUT.scid')
wtst.write_existing_records(da)
print('df ready', da.end - 1, time() - time0)
print(da.df[da.end - 1:da.end + 1])
print()
df = da.df
print('\n', np.datetime64('now'), da.end)
print(df[da.end - 5:da.end + 5])

import ipdb; ipdb.set_trace() # XXX BREAKPOINT


#time_list = []
#for i in range(4000):
#intime = df.index.values[da.end]
#time0 = time()
#da.add(intime, [1.0, 2.0, 3.0, 4.0, 5, 6, 7, 8])
#time_list.append(time() - time0)

#if time_list:
#print('TimeStats', max(time_list),
#sum(time_list) / len(time_list))
#print('\nEnd of NaN version')

# print('next', hsi.pos, hsi.last)
# jtst = SierraFile('/home/john/zRamdisk/SierraChart/Data/HSI-INPUT.scid')
# time_index, data = jtst.read_existing_records()
# ja = SierraFrame(time_index, data)
# jf = ja.df
# print('\n', ja.end)
# print(df[ja.end-5:ja.end+5])
# print('next', jtst.pos, jtst.last)
# return # ###################
counter = 0
# sys.stdout = os.fdopen(sys.stdout.fileno(), "w", newline=None)
counter_flag = False
timer_no_data = time()
timer_no_data_flag = False
overruns = 0
overrun_list = []
while True:
time0 = time()
new_time, data, late_flag = hsi.read_record()
if new_time != -999:
#time1 = time()
da.add(new_time, data)
#print("{:.6f}".format(time() - time1), end = ' ')
sys.stdout.flush()
wtst.write_record(da)
if counter > 3:
time_list.append(time() - time0)
timer_no_data = time()
#print(da.df[da.end-1:da.end], da.end)
print('.', end=' ')
sys.stdout.flush()
if timer_no_data_flag:
print('Data Restored')
timer_no_data = time()
timer_no_data_flag = False
counter += 1
counter_flag = True
if time() - timer_no_data >= 120 and not timer_no_data_flag:
timer_no_data_flag = True
print('Data lost for two minutes')
if not late_flag:
sleep_time = 0.1 - (time() - time0)
if sleep_time > 0:
sleep(sleep_time)
if counter % 12 == 0 and counter_flag:
counter_flag = False
print(' Overruns:', overruns, overrun_list, end=' ')
print('TimeStats', "{:.6f} {:.6f}".format(max(time_list),
sum(time_list) / len(time_list)), '\n', end=' ')
# print(df[da.end-1:da.end])
sys.stdout.flush()
# break
if counter % 60 == 0 and counter != 0:
import ipdb; ipdb.set_trace() # XXX BREAKPOINT


def main():
SierraRun()

if __name__ == '__main__':
"""
Takes a SierraChart scid file (input argument 1) and converts
it to a Pandas DataFrame
Timezone conversion can follow the users local timezone, or a
specified integer (input l or an integer but if the default
filename is being used, '' must be specified for the filename)
"""
print('start')
sys.stdout.flush()
main()
print('fin')
if time_list != []:
print('TimeStats', "{:.6f} {:.6f}".format(max(time_list),
sum(time_list) / len(time_list)), '\n', end=' ')

Date Time Of Last Edit: 2013-06-21 02:52:49
[2014-01-08 14:40:26]
jbutta - Posts: 85 | Ending Date: 2019-09-16 [Expired]
Kiwi, you may be just the guy I am looking for. can you tell me how I can either create a separate .bat file or program(possibly using python, to print my chartbook. I want to create an automatic way of printing some graphs at the end of the day. Thanks for the help

[2014-01-08 23:39:03]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
If your using Linux then, yes I probably can, but I abandoned Windows a while back.
[2014-01-09 04:05:37]
jbutta - Posts: 85 | Ending Date: 2019-09-16 [Expired]
I am still interested. While I use a win7 to trade, I really would like to be on linux anyway if I can. How are you graphing/trading on linux? If so I would love to see some screenshot of whatever you are using.

[2014-01-09 05:39:24]
onnb - Posts: 596 | Ending Date: 2021-11-24
this is off the original topic on python, jbutta, for what its worth, the following study will create an image for you on bar close. I used it in order to save images to a web server and it works quite well. You might need to adapt it for your needs like saving the file on session close or at a specific time of day. You would then apply this study to all charts you want saved. SC saves them for you in the images directory same as you would be saving an image manually.


  if (sc.SetDefaults)
  {
    sc.GraphName = "Save Chart Image to File";
    sc.StudyDescription = "";
    sc.AutoLoop = 1; // true
    sc.GraphRegion = 2;
    sc.HideStudy = 1;
    sc.DrawZeros = 0;
    sc.FreeDLL = 1;
    return;
  }
  
  if (sc.GetBarHasClosedStatus() == BHCS_BAR_HAS_NOT_CLOSED)
  {
    return;
  }

  sc.SaveChartImageToFile = 1;

[2014-01-09 13:44:53]
jbutta - Posts: 85 | Ending Date: 2019-09-16 [Expired]
Awesome thanks. I think this is what I needed to get where I want to go.
I appreiciate it.

[2014-01-09 13:55:57]
Hendrixon - Posts: 130 | Ending Date: 2017-03-11 [Expired]
Do you mean to develop studies in Python?
What does it give that C++ don't?
[2014-01-09 17:17:04]
jbutta - Posts: 85 | Ending Date: 2019-09-16 [Expired]
nothing, I just want to be on linux eventually for my trading platform.

As for this program, I have only programmed a little for NT7 and nothing substantial. I hope to be able to write something today for it. thanks

[2014-01-09 21:28:05]
jbutta - Posts: 85 | Ending Date: 2019-09-16 [Expired]
ONNB,
thanks. I am new to coding and sc coding for that matter. I want to create a function that save the chart at the end of the day at 16:15 each day.
I tried to work with what I saw in other codes, but It just keeps saving continuous png's. I just want it to save once at the end of the day. Can you tell me what I did wrong here? remember im a newbie.

thanks in advance!

SCDLLName("Autosave")

SCSFExport scsf_autosave(SCStudyGraphRef sc)
{
  SCInputRef Time1 = sc.Input[0];
  
   if (sc.SetDefaults)
{
sc.GraphName = "Save Chart Image to File";
sc.StudyDescription = "";
sc.AutoLoop = 1; // true
sc.GraphRegion = 2;
sc.HideStudy = 1;
sc.DrawZeros = 0;
sc.FreeDLL = 1;
  Time1.Name = "Time to Print";
  Time1.SetTime(HMS_TIME(16,15,0));

return;
}
SCDateTime Print_Time(sc.BaseDateTimeIn[sc.Index].GetDate(),Time1.GetTime() );//Set the print time for each day.
SCDateTime Current_Time(sc.BaseDateTimeIn[sc.Index].GetDate(),sc.BaseDateTimeIn[sc.Index].GetTime());//Get the current time
  sc.Subgraph[0][sc.Index]=0; //Flag signal
  if (Current_Time>=Print_Time)
  {
    sc.Subgraph[0][sc.Index]=1; //attempt to flag Print time so it only prints once per day
  }
  else
   return;
  //I want it to print/save at the change of state
  if (sc.Subgraph[0][sc.Index-1]==0 && sc.Subgraph[0][sc.Index]==1)
    {
      sc.SaveChartImageToFile = 1;
      sc.AddMessageToLog("Printed file",true);
    }
  else  
    return;  
}

[2014-01-09 23:45:45]
onnb - Posts: 596 | Ending Date: 2021-11-24
the best way to do this depends on your specific circumstances but sticking as much possible to your code...

what is happening is that the study function is like OnBarUpdate from NT. It gets called repeatably for any given bar. So the code here get called many times within the first bar encountered that starts at 16:15 or later


if (sc.Subgraph[0][sc.Index-1]==0 && sc.Subgraph[0][sc.Index]==1)
{
sc.SaveChartImageToFile = 1;
sc.AddMessageToLog("Printed file",true);
}

The simplest way I can think of to leave the rest of your approach intact and still get this to print just once would be to add a condition that you are on bar close for the study to process. That way the study only processes once.

You would add this like so:


if (sc.GetBarHasClosedStatus() == BHCS_BAR_HAS_NOT_CLOSED)
{
return;
}

SCDateTime Print_Time(sc.BaseDateTimeIn[sc.Index].GetDate(),Time1.GetTime() );//Set the print time for each day.
SCDateTime Current_Time(sc.BaseDateTimeIn[sc.Index].GetDate(),sc.BaseDateTimeIn[sc.Index].GetTime());//Get the current time
sc.Subgraph[0][sc.Index]=0; //Flag signal
if (Current_Time>=Print_Time)
{
sc.Subgraph[0][sc.Index]=1; //attempt to flag Print time so it only prints once per day
}
else
return;
//I want it to print/save at the change of state
if (sc.Subgraph[0][sc.Index-1]==0 && sc.Subgraph[0][sc.Index]==1)
{
sc.SaveChartImageToFile = 1;
sc.AddMessageToLog("Printed file",true);
}
else
return;

hope this helps

[2014-01-10 05:04:57]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
You wouldn't develop studies in Python. Python or R or Julia give you an advantage when you want to analyse data statistically or look for generic tendencies etc etc.

If you just want to create studies or executable systems (or test them) then its easiest to do it in C (there isn't much ++ in what we do for Sierra Chart fortunately). In my case, if I wanted to analyse that data in a sophisticated way then I'd export the results of the first phase in a text form and analyse it with the higher level language and its libraries.
[2014-01-10 06:20:40]
onnb - Posts: 596 | Ending Date: 2021-11-24
So you are analyzing the feed in real time and then writing back the trading actions which are then executed by a SC study? Did I get that right?
[2014-01-10 11:37:13]
norvik - Posts: 22 | Ending Date: 2014-05-11 [Expired]
Kiwi, I have a question, do you know some Python-based library like
Esper ? Esper is Complex Event Prosessing framework eventual in Java and C#.

Thanks.
[2014-01-10 13:15:29]
Hendrixon - Posts: 130 | Ending Date: 2017-03-11 [Expired]
Got you kiwi, thanks.
Can you give an example of a statistical analysis or generic tendency that is needed to be done on the raw data?
[2014-01-10 21:35:06]
jbutta - Posts: 85 | Ending Date: 2019-09-16 [Expired]
thanks to everyone here. I was able to figure it out even though I'm not a coder.

Kiwi, actually I was working was going to teach myself R so that I could use it in my business(Full Time) to do just what you are talking about. I am also interested in R do do some statistical modeling based on my trading methodology. Have you had any success modeling and has it help your trading?

[2014-01-12 23:48:18]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
onnb, yes, I'd agree with that view.

norvik, if real time event processing was a key driver I'd be using C / C++ or Java (or one of the functional languages) not Python.

Hendrixon, for generic tendencies just look at Lawrence or Ernie Chan's stuff. An example of statistical analysis would be to take your test outputs and Monte Carlo them or just use statistical packages to examine their predictive power.

jbutta, R has very powerful libraries although its an ugly language and rather slow. Modelling is an interest rather than a core of my trading. So far I find simple traditional approaches still provide the best results for me.
[2014-01-29 20:36:24]
vladaman - Posts: 1 | Ending Date: 2014-03-01 [Expired]
Here is simple Sierra Chart SCID File Reader written in Java https://gist.github.com/vladaman/8696352
[2014-03-02 23:11:31]
User15451 - Posts: 27 | Ending Date: 2018-09-15 [Expired]
Kiwi,

I have a couple of offline questions can you provide your e-mail?

Sincerly

TS
[2014-07-06 04:01:40]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
for someone who's interested in

http://www.youtube.com/watch?v=0unf-C-pBYE
Date Time Of Last Edit: 2014-07-06 05:08:44
[2014-07-06 12:32:55]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
Hi All

I'm not a programmer so this is my simple solution
to get data from *.scid and store it to *.hdf5
in order to pandas it later


#!/usr/bin/python3
import struct
import datetime as dt
import sys
import pandas as pd
import numpy as np

inputfile = open(sys.argv[1],'rb')
dt.timedelta(microseconds=1)

with inputfile as f:
  f.read(56)
  df_src=[]
  ts_src=[]
  while True:
    tick=f.read(40)
    if not tick:
      break
    src = struct.unpack('d4f4L', tick)
    ts_tmp=dt.datetime(1899, 12, 30) + dt.timedelta(src[0])
    ts_src.append(ts_tmp)
    df_tmp=[src[4],src[7],src[8]]
    df_src.append(df_tmp)
tubus = pd.HDFStore('tubus.h5')
df=pd.DataFrame(df_src, index=ts_src, columns=['Price', 'bidVol', 'askVol'])
df.to_hdf('tubus.h5','df')
print(df.index)
print(df.head())
print(tubus)
tubus.close()

1. how to run it: ~/>python3 this_script.py chart.scid

2. the script parses *.scid and creates the df DataFrame (TimeSeries): Price, bidVol, askVol

3. the script creates HDF5 file tubus.h5 and stores df

for 500MB.scid it takes 48s on i5/hugeRAM/HDD

Date Time Of Last Edit: 2014-07-06 12:34:11
[2014-07-08 00:32:15]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
Hi Ganz,

Very nice. One thing that has me confused. Why use HDF5?

I did some research after your Python post in the weekend and it seemed to me that HDF5 was about very very big data sets and random access into them. Also the ability to group different data types.

Now Sierra data is essentially sequential in nature and 2 data types (double + int) with fixed data in each column. Also the type of operations I do on them is also sequential - sometimes with some work to coerce the data into that of a higher time period before operation.

In that case would it be better to store them as CSV and/or just .scid files? Possibly the CSV files could be stored with some form of compression. I'm leaping completely out of my experience here and suggesting the bz2 sequential compression ... I probably need to try it.

https://docs.python.org/2/library/bz2.html#sequential-de-compression

Or does HDF5 do really nice compressed serialization?

My other question relates to Python vs C and how the two should fit into the Sierra Chart world so I'll address it in your other thread. I need to think about it a bit more first though.
Date Time Of Last Edit: 2014-07-08 00:40:04
[2014-07-09 14:57:04]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
Kiwi

Hi
Why use HDF5?
In that case would it be better to store them as CSV and/or just .scid files?
other question relates to Python vs C

the reason was explained there http://www.sierrachart.com/SupportBoard.php?ThreadID=8515

the idea is to get an Integrated Trading Environment for stocks, options, futures, bitcoin, etf/cfd, forex ...

to achive that the solution should be crossplatform, flexible and stores data using well known data format/scripting language at the production level

imho
[2014-07-10 03:45:46]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
OK.

I had read that and didn't see why HDF5 was chosen. Python is extremely happy reading CSVs (or even SCIDs with a little conversion).
[2014-07-10 08:09:53]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
Kiwi

The CSV is good but we need to ask SC Team to support ISO 8601.

http://pandas.pydata.org/pandas-docs/version/0.13.1/io.html#io-read-csv-table

read_csv has a fast_path for parsing datetime strings in iso8601 format, e.g “2000-01-01T00:01:02+00:00” and similar variations. If you can arrange for your data to store datetimes in this format, load times will be significantly faster, ~20x has been observed.
This has been tested and proved. (~16x in my case)

And yes, I'm happy to use scid for charting any data I need.

Another "+" for HDF5 : the format has an internal "file system". It makes much more easy to manage thousands of symbols, it makes users feel comfortable with a db, I believe.
[2014-07-10 13:14:28]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
for the record:

in order to use the script on a x64 system it needs to define the byte order for the struct function as explained there https://docs.python.org/3.0/library/struct.html for little-endian

in other words use '='or '<':

src = struct.unpack('=d4f4L', tick)

or

src = struct.unpack('<d4f4L', tick)


[2014-07-12 05:34:31]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
Kiwi

Why use HDF5?

http://pytables.github.io/FAQ.html#why-hdf5


Why HDF5?

HDF5 is the underlying C library and file format that enables PyTables to efficiently deal with the data. It has been chosen for the following reasons:

Designed to efficiently manage very large datasets.
Lets you organize datasets hierarchically.
Very flexible and well tested in scientific environments.
Good maintenance and improvement rate.
Technical excellence (R&D 100 Award).
It’s Open Source software


[2014-08-14 22:05:43]
User13778 - Posts: 24 | Ending Date: 2016-08-12 [Expired]
Excellent work Kiwi

I use Python to analyse NQ and ES and trade with Interactive Brokers.

Im looking to do similar with a python / Sierra Chart system on FXCM
[2014-10-30 14:09:26]
User85468 - Posts: 9 | Ending Date: 2014-11-27 [Expired]
Hello,

how can I have my PnF chart (or reversal bar chart), say in 15 minute periods, print and/or remove Xs or Os depending on the direction, for every price move on the last bar? It works on the Tick, but the chart is really worthless in such a short time-frame.

Kind regards
[2014-11-22 23:30:23]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
Why use HDF5?

http://youtu.be/hnhN2_TpY8g
[2015-04-01 14:27:50]
Shaharabin - Posts: 28 | Ending Date: 2016-09-26 [Expired]
Hi Guys,

Interesting stuff.

My question to you all is; can i export real time "Time and Sales" data from sierra chart and use it with any other program? using any type of language.

My end goal is to stream that data onto the PC's serial port and build a physical gauge

Thank for any replay
Shahar.
[2015-04-01 18:46:10]
Sierra Chart Engineering - Posts: 95944 | Ending Date: 2022-04-10
In regards to post #32 something that may be of help for this, is the relay server:
https://www.sierrachart.com/index.php?page=doc/doc_RelayServer.php

Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. If possible please keep your questions brief and to the point. Please be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

If your question/request has been answered and you do not have anything further, then it is easiest for us if you do not reply again to say thank you.
[2015-04-01 20:03:51]
Shaharabin - Posts: 28 | Ending Date: 2016-09-26 [Expired]
Holy-S***
This is great stuff, how did I not know about it....
I'll start working on it, which means, a whole lot of new questions coming your way :)

Thanks a lot!
[2015-04-01 22:22:56]
Sierra Chart Engineering - Posts: 95944 | Ending Date: 2022-04-10
If you have further questions for Sierra Chart support, post them in a new thread because we are not monitoring this thread.


Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. If possible please keep your questions brief and to the point. Please be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

If your question/request has been answered and you do not have anything further, then it is easiest for us if you do not reply again to say thank you.
[2017-07-17 20:26:58]
rhovega - Posts: 279 | Ending Date: 2021-04-20 [Expired]
Hi @ganz hoping you are still around this forum and get to see this!

I am new to Python; running your code from post 22 gives me error "IndexError: list index out of range", while running code "~/>python3 this_script.py chart.scid" (with the appropriate filenames) from the Python console gives me error "SyntaxError: invalid syntax". Can't figure it out.

I am aware this is a very old thread but maybe if still around you could give me some pointers to move along. Would be much appreciated.

Thank you.
[2017-07-22 09:10:57]
ganz - Posts: 1034 | Ending Date: 2019-04-28 [Expired]
rhovega

hey ...

the only way in this case is to learn what each of line exactly does
study Python step by step in order to get it working

imo :)
[2019-03-05 19:33:57]
Chad - Posts: 183 | Ending Date: 2021-09-08
@Kiwi, have you looked into adding a function within your Python script for reading market depth (.depth) files?
[2019-03-05 22:45:49]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
No CB, I trade purely on price vs time

My latest work on Python has been a program using ib_insync to connect to the twsapi directly but its for basic information presentation & order entry/management to complement TWS charting.


https://i.imgur.com/nRbivpe.png
[2019-05-10 01:25:52]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
So here's another little bit of code.

It will convert all the text files in a directory into scid files in another directory. If the constant KEEP_UPDATING was True it will then check the text files every REFRESH seconds and if they've changed it will update the scid files.

Set FILES_IN and FILES_OUT. In windows the format is "C:\\mydir\\mysubdir\\"

Also set Timezone if you wish to change the times in Sierra's scid files (Sierra saves them in the UTC time zone).

Save the code in a file named something like Text2Scid.py and run it using python Text2SCid.py.
Tested with Python 3.7.Have fun:

"""
Convert's Text files to SCID files for SierraChart
It will create a new scid file for every text file it finds
format is assumed to be D T O H L C V T with comma separated variables
It will continue to update the scid file every REFRESH seconds
generally refresh would be set to 1/2 the time period for the text file updates
"""

import os
import struct
from time import sleep
from datetime import datetime as dt, timedelta as tdelta
import subprocess

# these are linux format, windows might need to use double backslashes \\
FILES_IN = "/home/john/Python/Trading/textfiles/"
FILES_OUT = "/home/john/zRamdisk/SierraData/"
TIMEZONE = 0 # scid saved as UTC, use this if the text file is another time zone
KEEP_UPDATING = True # set to False to convert and stop; True for live updating
REFRESH = 0.25 # refresh every 1/4 second for 1/2 second updating text files
TEXT_FILE_SUFFIX = ".txt" # usually .txt or .csv

sizeHeader = 0x38
sizeRecord = 0x28
file_header = (b'SCID8\x00\x00\x00\x28\x00\x00\x00\x01\x00\x00\x00\x00\x00' +
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' +
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' +
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')


def excel_date(datetime, offset=TIMEZONE):
"""convert date&time with offset in hours to excel format for scid"""
delta = datetime - dt(1899, 12, 30) + tdelta(hours=offset)
return delta.days + delta.seconds / 86400


def python_date(exceldt, offset=TIMEZONE):
"""convert date&time with offset in hours to python format"""
excel_as_delta = tdelta(days=exceldt//1) + tdelta(seconds=exceldt%1*86400)
return dt(1899, 12, 30) + excel_as_delta + tdelta(hours=offset)

def write_struct(record, f):
if len(record) == 8 and record[0][0].isnumeric():
out = [excel_date(dt.strptime(f'{record[0]} {record[1]}', '%Y-%m-%d %H:%M:%S.%f'))]
out.extend([float(s) for s in record[2:]])
# note that SC packs dt, O, H, L, C, T, V not V, T so swap them
wt = struct.pack('d4f4I', out[0], out[1], out[2], out[3], out[4], int(out[6]), int(out[5]), 0, 0)
f.write(wt)


def create_base(f):
record = None
with open(FILES_OUT + f[: -3] +'scid', "wb") as scidfile:
scidfile.write(file_header)
with open(FILES_IN + f, "r") as txtfile:
for line in txtfile:
record = line.split(',')
write_struct(record, scidfile)
return f'{record[0]} {record[1]}' # will update when date+time changes


def update_if_changed(files):
for f in files:
with open(FILES_IN + f, "r") as txtfile:
record = txtfile.readlines()[-1].split(',')
last_dt = f'{record[0]} {record[1]}'
if files[f] == last_dt:
continue
files[f] = last_dt
with open(FILES_OUT + f[: -3] +'scid', "ab") as scidfile:
write_struct(record, scidfile)


if __name__ == '__main__':
files = sorted(k for k in os.listdir(path=FILES_IN) if k[-4:] == TEXT_FILE_SUFFIX)
files = {k: '' for k in files}
for f in files:
files[f] = create_base(f)
print('finished base')
if KEEP_UPDATING:
# now test each input file each REFRESH interval & update if datetime changed
while True:
sleep(REFRESH)
update_if_changed(files)


edit: changed so that it will run despite the extra space & linefeed added after each line. Probably better to use the attached file though.
Date Time Of Last Edit: 2019-05-11 12:42:07
attachmentText2Scid.py - Attached On 2019-05-11 12:41:30 UTC - Size: 3.31 KB - 231 views
[2019-05-10 06:02:42]
User21223 - Posts: 77 | Ending Date: 2021-05-24
Hi
Can anyone check the above code. Because I'm getting an error. I do not know if it's my part.


C:\Users\mkruss>python Text2Scid.py
File "Text2Scid.py", line 33
file_header = b'SCID8\x00\x00\x00\x28\x00\x00\x00\x01\x00\x00\x00\x00\x00\x0
0' \

^
SyntaxError: unexpected character after line continuation character



Sample text file to be converted into a scid in the attachment
Date Time Of Last Edit: 2019-05-10 06:12:59
Attachment Deleted.
attachmentSymbol2.txt - Attached On 2019-05-10 06:11:49 UTC - Size: 18.34 KB - 190 views
[2019-05-10 06:40:08]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
Yes. It seems this board adds extra characters (an extra space and a linefeed) at the end of the line when you cut & paste from it. So here's a raw version without extra characters.
Date Time Of Last Edit: 2019-05-10 06:46:10
attachmentText2Scid.py - Attached On 2019-05-10 06:39:58 UTC - Size: 3.58 KB - 196 views
[2019-05-10 08:31:55]
User21223 - Posts: 77 | Ending Date: 2021-05-24
Now. This error is shown

C:\Users\mkruss>python Text2Scid.py
finished base
Traceback (most recent call last):
File "Text2Scid.py", line 84, in <module>
update_if_changed(files)
File "Text2Scid.py", line 60, in update_if_changed
record = subprocess.check_output(['tail', '-1', FILES_IN + f]).decode().spli
t(',')
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 395, in check_output
**kwargs).stdout
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 1178, in _execute_chi
ld
startupinfo)
FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku

C:\Users\mkruss>


I will add that the scid files were created in the Siarrachart directory. You can load them by Open Intraday Chart

But they do not freshen up automatically
[2019-05-10 08:37:59]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
Make sure your file paths have no spaces in them. For example SierraChart or Sierra_Chart, not Sierra Chart. Some python modules have problems with them so it's easier not to include spaces in paths.
Date Time Of Last Edit: 2019-05-10 08:43:08
[2019-05-10 08:59:25]
User21223 - Posts: 77 | Ending Date: 2021-05-24
I had Sierra Chart. I changed to SierraChart.
but the problem still occurs. Same error
[2019-05-10 09:35:47]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
After the line if name == main add this and post the result:

import sys; print(sys.version)

I'll do another version tomorrow that's more defensive to see if the problem is detectable.
[2019-05-10 09:51:56]
User21223 - Posts: 77 | Ending Date: 2021-05-24
Microsoft Windows [Wersja 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Wszelkie prawa zastrzeżone.

C:\Users\mkruss>python Text2Scid.py
3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)]
finished base
Traceback (most recent call last):
File "Text2Scid.py", line 85, in <module>
update_if_changed(files)
File "Text2Scid.py", line 60, in update_if_changed
record = subprocess.check_output(['tail', '-1', FILES_IN + f]).decode().spli
t(',')
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 395, in check_output
**kwargs).stdout
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Users\mkruss\Anaconda3\lib\subprocess.py", line 1178, in _execute_chi
ld
startupinfo)
FileNotFoundError: [WinError 2] Nie można odnaleźć określonego pliku
[2019-05-10 12:55:35]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
I'll do a new version tomorrow but you should be able to get it done by replacing


record = subprocess.check_output(['tail', '-1', FILES_IN + f]).decode().split(',')

with:


record = txtfile.readlines()[-1].split(',')


edit: corrected version of the file on Post #41
Date Time Of Last Edit: 2019-05-11 23:47:48
[2019-06-27 15:33:37]
User183724 - Posts: 74 | Ending Date: 2021-05-11
Kiwi

Quick question...do you have a SCID to TEXT version of your Text2SCid.py that you would be willing to share? I'd like to read the SCID file into a program as a text or CSV file.

Im new to Python (used a lot of other languages)... have been using SC for a while...I kind of understand what youre doing here and would be interesting in trying to develop something.

Thank You
[2019-06-27 22:39:17]
Kiwi - Posts: 327 | Ending Date: 2021-07-25
This will probably work. It converts an scid file (named after the imports) to a tab separated variable file of the same name (but .scid becomes .tsv) & path.

Let me know if you have problems (including any error messages). It's updated for Python 3 & tested with 3.7.4.
Date Time Of Last Edit: 2019-06-27 22:39:57
attachmentSCID_to_TSV.py - Attached On 2019-06-27 22:38:54 UTC - Size: 3.55 KB - 181 views
[2019-06-28 00:23:47]
User183724 - Posts: 74 | Ending Date: 2021-05-11
Kiwi- Thanks for the file. I'll let you know how it goes
[2019-06-28 16:28:36]
User183724 - Posts: 74 | Ending Date: 2021-05-11
Has anyone worked with the SCSS spreadsheet files? Do you have py files that read/write to these files that you would be willing to share with me? Thanks in advance
[2019-07-01 20:57:55]
Sierra Chart Engineering - Posts: 95944 | Ending Date: 2022-04-10
The Spreadsheet file format is too complex to be working with in this way.
Sierra Chart Support - Engineering Level

Your definitive source for support. Other responses are from users. If possible please keep your questions brief and to the point. Please be aware of support policy:
https://www.sierrachart.com/index.php?l=PostingInformation.php#GeneralInformation

If your question/request has been answered and you do not have anything further, then it is easiest for us if you do not reply again to say thank you.
Date Time Of Last Edit: 2019-07-01 20:58:08
[2019-07-02 00:49:00]
User183724 - Posts: 74 | Ending Date: 2021-05-11
Thanks for the reply. I'm really just looking for options on how to read real time data into Python. Spread sheet SCSS files were just something I was looking at
[2019-09-16 15:29:03]
Mack - Posts: 78 | Ending Date: 2020-08-23 [Expired]
What is really missing is a Python framework for SierraChart. Something that allows to code, backtest and review strategies.

@Kiwi: in your opinion, would it be better to develop a standalone backtest system? or something that is tightly integrated with SierraChart? In theory, you could just use SC to get data, and execute signals. (but do the backtesting in your own backtester)
Date Time Of Last Edit: 2019-09-16 15:31:58

To post a message in this thread, you need to login with your Sierra Chart account:

Login

Login Page - Create Account