Making it easier to download JSON price data that is login protected?

Making it easier to download JSON price data that is login protected?

Josh McCormick
2020-08-18 22:25:45

As a side project, I'm downloading and analyzing some of the price data from my 401K provider. The data is in JSON format, and the fund manager requires the user to be logged into their account before their URL will provide any fund data (including price). I've got it all working, but my workflow is absolutely awful and time-consuming:

1. Log into the 401K provider's website
2. Download the 14 individual JSON files (one for each 401K fund)
3. Transfer the files to a Linux box
4. Convert the JSON price data to TXT (using 'jq')
5. Transfer the files back to the PC
6. Import the 401K price data into QuantShare with the Data --> Ascii Import tool.

I've been successful! But it is a lot of manual work. Are there any existing features or possible enhancements you'd be considering that would make this easier? Might there be some way that QuantShare's downloaders can allow the user to log into a page before retrieving data? Is it possible to directly import from (moderately complex) JSON files with the Data -> ASCII Import tool? (I didn't see any JSON options.)

Thanks,
Josh McCormick

QuantShare
2020-08-20 11:28:47

Best Answer

Yes, it is possible to import using JSON. You need to import it as CSV data then use the "Pre script" tool to ignore currently imported line and use the "ReadJsonData" to read json.

Content.Rows[x].IsIgnoreLine = true;
JsonData json = Content.Net.ReadJsonData(text);

For the automated log in, this could be possible depending on how complex it is.

Josh McCormick
2020-08-20 17:50:01

I'll give the prescript a try, thanks. I'll send a private email with more of the login details.

Get more messages

No more messages

Back


Trading Items Consumer Price Index - Historical data Price to Book Ratio Historical Data - Market to Book ISE index options historical data Bursa Malaysia EOD data RTS RU data	How-to Lessons How to download historical EOD data for penny stocks How to quickly download the most recent EOD data for your stocks How to download earnings calendar data for various stocks How to download and use U.S. stocks earnings data How to download EOD quotes for active and valid stocks only
Related Forum Threads Downloader - Login and Download a file how do you check if the price data is up to date? ETF data download limited to 2011 DownloaderTask - How to download data for a watchlist Download fundamental data for U.S. stocks	Blog Posts Day Trading: A trading system that combines intraday and EOD data 6 ways to download free intraday and tick data for the U.S. stock... 6 places to download historical intraday Forex quotes data for fr... How to make sure your Historical Price Data is Split Adjusted Download Trading Data using the Post-Script