|
Let us say you want to automate the download process of an economic indicator, you know where you can download the data, you searched for a similar downloader in the QuantShare Sharing server, but unfortunately, you didn't found what you were looking for. Your only choice is to create your own downloader so you can automate the downloading process. For someone who never used the downloader and haven't read the documentation, it may seem difficult to create a download item. Don't worry, we will create a series of posts introducing each time an example and addressing the main difficulty that someone may find while creating a download item. For the first post, we are going to show you how to create the following item: Initial Jobless Claims - Historical data. This downloader retrieves historical data for Initial Jobless Claims. The data is available in the St. Louis Fed website and the URL where the data can be downloaded is: https://research.stlouisfed.org/fred2/series/IC4WSA/downloaddata The download item we want to create will automatically download historical data for the Initial Jobless Claims Series, adds the parsed data into the Quotes database, and associates it with the symbol: ^Initial_Jobless_Claims Downloading Process What we notice first is that when we click on the "Download Data" button (in the URL page), a file is downloaded and the URL in the browser does not change. This is because the "Download Data" button sends POST data (Post Data is data sent to the website server without being displayed in the URL). The first thing we have to do is to detect what parameters this POST data contains. This can be done using the "Detect Post Data" feature of QuantShare. Create a new download item (Download Manager -> Add), then click on "Detect" (at the bottom). Add the following URL (https://research.stlouisfed.org/fred2/series/IC4WSA/downloaddata) in the URL field then click on "Navigate". Now select the file format (Text, Comma Separated) then click on the "Download Data" button, the "Detect Post Data" feature of QuantShare will display a popup that includes all the parameters passed to the server by the POST request. Click on "Add URL", and then close the "Detect Post Data" form. The "Update Download Item" form should have a row that contains the previously used URL and the Field button should display "6 Field(s)". There are three things we need to specify before updating the parser settings: - Specify that the parameters should be passed with a POST request: Click on "Settings" in the URL row, then check the first checkbox - Set the content type: In the URL row, double click on the cell under the "Content Type" column and select "CSV_EXCEL". - Specify the symbol that should be associated with the data (this is because the data doesn't a reference to any symbol): In the URL row, click on the cell under the "Symbol" column and type "^Initial_Jobless_Claims" - Allow the user to specify the end date: Click on "6 Field(s)" and set the type of the "obs_end_date" row to Custom. Parsing Process This is it for the URL settings; now let us do the parsing part. Under the "Parser" column, click on the "Parser" button. The application will download a sample that you can use to make sure that your parser settings are correct. Here is a subset of the sample: DATE,VALUE 1967-01-28,209000 1967-02-04,211000 1967-02-11,216500 1967-02-18,219500 It is easy to notice that: The separator is a comma The date format is YMD The date separator is "-" We should add two columns (Quotes-Date and Quotes-Close) There are 12 lines to skip (You will notice this later) Now click on "Next", then click on the fields (Quotes -> Date) and (Quotes -> Close) to see how the data was parsed. To validate the parsing, click again on "Next". Make sure there are no errors then click on "Finish". We hope this post will help you create more downloaders and why not share them later.
|
|
