Downloaders - Trading Software

Downloaders

David
2020-04-29 02:04:06

I appear to be having an issue with the downloaders, that seems to be based on the GMT of the exchange where the data is coming from.
I only pull down data from the NZX and ASX (NZ and Australia) and also monitor some key US indices like the S&P500 DJI etc

Recently with my ASX symbols I needed to know the days of the week.
Used DayOfWeek() and noticed some days were listed as 0 and some 6 i.e. Sunday and Saturday.
Thinking there was an error in the data, I downloaded the entire database again with the "Historical Stock Market Data" downloader,
again it shows up this inconsistency occasionally with Sunday and Saturday listed as trading days, also tried with "Yahoo EOD Quotes" and same thing.

Created a new account with just ^NZ50,^AORD,^GSPC
Used Yahoo EOD quotes to download from April 1st to April the 24th 2020
For the NZ50 16 bars are downloaded with dates from the 31st March to the 23rd of April , but with the data from the 1st of April to the 24th of April
For the AORD It starts at the 31st of March and ends at the 24th, also 16 bars. Pricing after the 6th of April is correct, before is following day pricing.
Also it jumps from the 2nd of April to the 6th of April, Friday the 3rd is missing.
For the ^GSPC it is perfect.

NZ is GMT + 12, AU is GMT + 10, US-New York is GMT -4

Had a look at the Downloaders URL script, and tried adjusting the P1 and P2 values with UNIX time offsets, but not much luck.

Mucked around a bit with the generated URL from yahoo EOD Quotes (AORD below) and cant seem to alter the outcome.

Putting the below into a browser like Firefox will get the data nicely formatted in JSON, and you can check the first timestamp is 31st Mar 2020

https://query2.finance.yahoo.com/v8/finance/chart/^aord?formatted=true&crumb=&lang=en-US&region=US&period1=1585699200&period2=1587735173&interval=1d&events=div%7Csplit&corsDomain=finance.yahoo.com

Also had a look at indices closer to the GMT like a number of the European ones and these seemed correct

So just a "too far from everything" issue? Easy fix?

Cheers

David
2020-05-08 21:37:33

Ok, been trying alternate downloaders like YLoader etc to get around this date issue, but found other small quirks using these packages.

Had a closer look at the Data Yahoo was presenting for download via say "Yahoo EOD" and it seems the timestamp has been adjusted for the exchanges GMT offset.

So all I have done is altered the "Pre Script" to add the exchanges GMT offset to the timestamps and this appears to have fixed the problem.

All my ASX and NZX data is now correct, i.e. no Sunday or Saturday trading days. Also checked the S&P500/DJI and also appears correct. (note, was not wrong previously, but this didn't break it)

Not 100% sure this will work with every exchange, hence not uploading, but for GMT > 8ish exchanges like Australia and New Zealand. the data and dates, line up correctly now.

Pre Script below , altered lines have //*

System.Threading.Thread.CurrentThread.CurrentCulture = System.Globalization.CultureInfo.InvariantCulture;
for(int i=0;i<Content.Rows.Length;i++)
{
Content.Rows[i].IsIgnoreLine = true;
}

string content = Content.GetContent();
JsonData j = Content.Net.ReadJsonData(content);

//Content.AddRow(j["chart"]["result"][0]["indicators"]["quote"][0]["close"].ToString());

DateTime unix = new DateTime(1970, 1, 1);
JsonData r = j["chart"]["result"][0];
JsonData q = r["indicators"]["quote"][0];
JsonData t = r["timestamp"];
JsonData c = q["close"];
JsonData o = q["open"];
JsonData h = q["high"];
JsonData l = q["low"];
JsonData v = q["volume"];
JsonData gmt = r["meta"]["gmtoffset"]; //*
for(int i=0;i<c.Count;i++)
{
int tt,os; //*
if(int.TryParse(t[i].ToString(), out tt))
{
try{
string open = (o[i] == null) ? "" : o[i].ToString();
string high = (h[i] == null) ? "" : h[i].ToString();
string low = (l[i] == null) ? "" : l[i].ToString();
string close = (c[i] == null) ? "" : c[i].ToString();
string volume = (v[i] == null) ? "" : v[i].ToString();
int.TryParse(gmt.ToString(), out os); //*
tt += os; //*
Content.AddRow(unix.AddSeconds(tt).ToString("MM/dd/yyyy", System.Globalization.CultureInfo.InvariantCulture), open, high, low, close, volume);
}
catch
{
}
}
}

QuantShare
2021-04-05 11:46:34

Best Answer

Thank you David.

The downloader was updated.
Historical Stock Market Data

Historical Stock Market Data (by QuantShare, uploaded several months ago)

No notes

Get more messages

No more messages

Back


Trading Items Toronto Stock Exchange - historical data Splits Calendar - Historical data for U.S. Stocks EOD Data - End of day data for U.S. Stock Market Israel Stock Market Update the Industry Name of your U.S. Stocks	How-to Lessons How to quickly select stocks based on the last value of a databas... How to download EOD quotes for active and valid stocks only How to quickly download the most recent EOD data for your stocks How to generate buy/sell signals from a trading system How to speed up watchlist and screener plug-ins when working with...
Related Forum Threads downloaders using google trends data Daily and Historical Downloaders SecureChannelFailure on downloaders Download fundamental data for U.S. stocks Symbol Information	Blog Posts Creating a download item: Initial Jobless Claims How to Download Trading Data for Certain Securities Only Running QuantShare on Amazon's EC2 Cloud & Automate Strategies Si... German Stock Exchanges: 5 free ways to download historical EOD/In... Trading Forex with the commitments of traders report