I'm wanting to update some of the information associated with the symbols I use in QuantShare (without removing and re-adding the symbols). I've pulled a list of all my QuantShare symbols into a text file and a CSV field. I have a source for Fullname, Sector, Industry, and a few other fields that I want to update. What's going to be my best (automated) method to import the updates into QuantShare?
Hey there. I have a script to do this regularly when my data source updates their ticker metadata. Not the prettiest code, but I'm sure you can adapt it to your needs
Pasted here: https://pastebin.com/pJC3Egmz
You need to attach the CsvHelper https://joshclose.github.io/CsvHelper/ dll to your script to use it directly, or just replace that with whatever reading code you use.
Uh... Kyra? Your reply wasn't just interesting. What you provided was ABSOLUTE GOLD!
I noticed that your script was looking for a .csv file name "SHARADAR_TICKERS", so I Googled for it and found it on Quantl, and started playing around with the data. NICE!
https://www.quandl.com/databases/SF1/data
Hmmmmm. It seems that I've already got a number of active (and verified) symbols that aren't in his database? But wait... he's got quite a number that aren't in mine (even after filtering out Canadian stocks, OTC, and de-listed symbols)! None of my ETFs are in there, but he clearly indicates that it's a database of public companies, not of stocks. (And now I see he's got an "SFP" database for funds.) He's got a custom scale for market cap and revenue for most companies? Extremely useful. **AND** there's a way to extract just the recently added companies? (I see Root, McAfee, and more?) This is absolutely beautiful.
Up until this point, I've taken my own list of symbols (companies, ETFs, mutual fund, indexes) and used a bit of shell script to extract some company information from Yahoo. The script locally stores Yahoo's JSON file for each symbol. After that (not shown), the real data is extracted from different objection, depending on the symbol type (Stock, ETF/Mutual Fund, Index, or Future). There's some absolutely fantastic (and unique) company information in the Yahoo JSON data (along with a ton of unnecessary page rendering info). Extracting the JSON from is done via a simple UNIX/Linux shell command. I choose the symbol via the $i variable, and it writes the JSON to a file named after that symbol. Here's the command:
Wow. Now I've got a lot to think about here. Yahoo provides a ton of info, but I've got to work through my own list of symbols. The SF1 database is limited to companies, but it is pretty authoritative and very useful for catching new listings. I think I want to merge the two approaches. This is AWESOME.
PS: Now that I think of it, if you didn't intend to disclose this much, let me know and I'll scrub it from my reply, okay?
Thanks again so much for your help Kyra!
Josh McCormick
Don't worry, the snippet I posted was plenty redacted, but I appreciate you mentioning it :D I certainly don't mind disclosing that I use the Sharadar data source for some of my historical data. In fact I recommend them (and I'm not affiliated). I spent a long time researching a historical provider with good coverage at a reasonable price after I decided free data was mostly more trouble than it was worth a few years back. I believe that their US Equity bundle (https://www.quandl.com/databases/SFA/data) is a particularly good value.
You mentioned that you are using some linux command line script for some of your work. Are you running Quantshare under emulation in linux, or doing data prep on a separate box? I'm curious since I haven't tried running Quantshare in linux and was wondering if you had a good experience with it if you're doing that.
As it turns out, I'm not emulating Windows on Linux. A funny thing... these days, Microsoft is hoping to get people to do just the opposite and use Windows Subsystem for Linux in order to emulate Linux on Windows! But I'm not doing that either. For what I'm using it for (data prep, custom processing, notifications, alarms), a Raspberry Pi 4B (4GB model) is overkill. I *do* understand and appreciate your choice of a commercial data provider. (Many times, free data is only "free" if your time is worth nothing.)
Last I knew about unix scripting in windows would have been Cygwin, but I haven't done any serious shell scripting in quite some time. I wasn't aware of WSL, thank you for that.
> (Many times, free data is only "free" if your time is worth nothing.)
It's worse than that. When you're developing strategies, in my experience, it's common to run into situations where they actually zero in on problems with the data, especially if you are using optimization or data mining techniques. If you have poor quality data there is a very good chance that any strategies you develop against it overstate their results dramatically because those results are based upon exploiting problems with the data. When you go to the real market with these, you're going to lose money.
Sadly there's no guarantee that if you pay for some data that it's going to be free of these problems! Data quality is something that I'm always having to think hard about.
Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.
1
