What I'm trying to do is speed this script up so that I can generate trading signals in a timely fashion. The issue I'm running into is that
for these symbols I have hundreds of thousands of intraday bars of historical data, and the p.Parse line just runs too slowly no matter how much I cut
down the formula that it's executing.
From my understanding, what QS is doing in that p.Parse line is executing the formula over every bar from the symbol, and because there are so many
bars it cannot help but take a while. However, for preduction use I really only need the formula executed on the last couple thousand bars or so,
and I think if I could tell QS to do this, the Parse would be MUCH faster.
So, my question is, is there any way to do this? What I would love to see is another parameter on the Parse function that specifies what bars to execute
the formula on. Do you know of a way to do this that I may have missed, or if a feature like this has been added to a recent version of QS?
You have an interesting problem. And now I question a decision I made just a couple of hours ago to increase my intraday data retention from 90 days to an unlimited retention policy. :/
I personally don't have a great answer here, but it made me wonder if QuantShare is capable of executing multiple scripts simultaneously. If so, it would be a pain, but you could spread the work among multiple scripts to grab more processing power. Still, this is just theory which could be completely bogus. If you're interested, the easiest way to test it out is to divide your list of symbols in half and perform an experiment with two scripts which launch (more or less) at the same time. If it works, then you could figure out how you further divide and improve on the process. I don't envy your problem.
I've found that QS can indeed execute multiple scripts simultaneously; if you go to your toolbar, right click, and 'add shortcut' to a script you can execute several in parallel. I do a lot of scripting, so I've found this to be very handy over the years, though it would be nice if script editor included the functionality directly.
As to my problem, breaking the task into numCpuCores scripts doesn't provide enough of a speedup. Was thinking along those lines too, so I went ahead and multithreaded the part of the script that's doing the heavy lifting (the lines above). It helps a bit, but nothing on the order of what reducing the bars processed from like 500k -> 5k would do. I'm considering creating alternate symbols for the stocks I am interested in that I only load the last few thousand bars of data into. I think that would fix it, tbh, but I'd hate to take on the extra maintenance overhead. Was hoping QS had an option on the Parse function to only load the most recent N bars, or that one could be added.
You are correct. QS is executing the formula on all bars but data is vectorized so if we are going to execute formula on fewer bars, you will certainly see speed increase but probably not in the order of magnitude that you imagine.
Unfortunately currently this is not possible as the formula needs to be execute on all data.
Using Intraday data, you can try to instruct QuantShare to load less data using "Accounts -> Intraday Settings -> Number of days worth of intraday data to load"
If you do not need all the historical data, you can also delete it (specific period) from your database using "Data -> Edit databases -> Manage Databases -> Delete databases data"
Besides reducing the number of day she retains data (or manually culling older data from the database), do you have any additional tips for Kyra? Or are there any enhancements that you might be able to perform on your side of things?
Kyra,
Good tip on the script shortcut. I need to do that, thanks!
I too was thinking of alternate symbols with reduced data, but I didn't want to recommend that... I feared that it would create too many unintended side-effects and extra work for you to deal with. (Also thought of putting it in a second user account, but that's likely to be an incredible pain just to .) It sounds like you're totally on top of the performance issue from a system/hardware perspective, or I would have dived into configuration issues and if you were on an SSD (which may or may not help).
I see that he thinks that reducing the data isn't going to be as helpful as you think, but you know what? It isn't going to be difficult to run a real-life performance test and see how much it's going to bring (before you commit to anything). In the past, I've backed up my QuantShare directory, ran a (destructive) experiment, and restored the files back with no harm done. Your situation may be different, but that might be worth a shot to see how much performance boost you get by culling your local database.
Just spitballing ideas here, I hope I 'm not getting in the way. I'm hoping to do some real-time intraday scanning and execution myself, so this topic interests me!
Thank you for the suggestion on intraday days to load! I had high hopes for this when I read it as it sounds like it would do just what I am wanting, so I tested execution speeds with this set to 100 days vs unlimited. Unfortunately, as you suggested, it didn't have any effect on report generation speed in script via parser.Parse or in the GUI backtester. It DID limit data loaded on charts, though, which can be useful if you have a very heavy formula attached to a huge intraday chart (maybe also for watchlists?). Will keep that in mind.
I also tested reducing number of intraday bars given to the parser by running my script against 500 symbols of 5000 bars and comparing that to 500 symbols of 100000 bars. This resulted in around a 20x speedup, which is what I was expecting (O(n) in number of bars) and in fact what I need for this execution to be viable in production.
I would love to have an option to the parser.Parse function (and in the report GUI tool) that truly limits the number of bars (say, to a given date range) that the parser operates on. Not sure of the internals of QS so I've no idea if this is a big ask or not. Seems like it would be straightforward, but I can imagine there may be some bar synchronization issues...
Josh:
Your ideas are welcome!. As you can see, I did go ahead and run the performance comparison, which sadly turned out as we expected it would.
It'd be great if QS develops a way to limit the number of bars processed by scripts / the report GUI tool. In the meantime I will probably go ahead with creating a "historical intraday" version of my symbols with the complete intraday data and a "production intraday" version where I keep the last N bars that I need to enable my production scripts to execute in a timely manner. Kind of a pain, and with some potential pitfalls as you mentioned, but not something that I can't manage. I just would *prefer* not to :D
Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.
1
