Help With Daily Data Import Script

Help With Daily Data Import Script

Dave W.
2013-09-12 02:56:19

I want to use the process defined in the article "How to Automatically Import Local Data Into QuantShare" here:
Read more: http://www.quantshare.com/sa-552-how-to-automatically-import-local-data-into-quantshare#ixzz2ecwjLDcz

Buy my situation is a bit different, and I'm hoping you can help. After each trading day, my data service software downloads three files I need to import, and each is stored in a separate directory: 1 file for AMEX market in one directory, one file for NASDAQ in another, and one file for NYSE in another. Each day, the download service software adds a new file to each directory with a name in the format: AMEX_20130911.txt. Since there could be hundreds or potentially thousands of daily files in each directory, I need to only grab and import the most recent file in each of the three directories.

I'm guessing there is an easy modification to the script (reposted below) that would allow me to only grab the most recent .txt file in each directory by last modified date. Can you help me with that?

Here's the original code for reference:

This script allows you to dynamically create URLs or file paths to parse.
- Type the following script then click on "OK"

string[] files = System.IO.Directory.GetFiles(@"C:\data\", "*.csv");
int count = files.Length;
for(int i=0;i < files.Length;i++)
{
string file = files[i];
Functions.AddURL(file, "Parsing: " + i + "/" + count);
}

Read more: http://www.quantshare.com/sa-552-how-to-automatically-import-local-data-into-quantshare#ixzz2ecxvVwX4

QuantShare
2013-09-12 13:35:40

An easier solution for this, would be to create a new downloader, create 3 URLs (one for each directory).

Example:
c:\directory1\AMEX_[DATE].txt

Click on "Fields" button under "Fields" column and add a date field (type). Make sure the checkbox (last column) is checked.

Test the parsing by clicking on "Parser" button.

qTester
2013-10-09 14:54:53

file-by-file import is very(!) slow (707 requests in 30 minute(s) and 50 second(s).)
is any other ability to bulk update several hundreds of files, which (externaly)refreshing every 1 minute inside some local directory on HDD?

QuantShare
2013-10-09 21:42:05

What is the size of each file?
In the downloader, you can define the number of threads to use ("Threads" option).
You can define the maximum number of threads to allow, by selecting the download item then selecting "Update -> Download Settings".

Get more messages

No more messages

Back


Trading Items EODData Daily Import Daily Stock Quotes - Stock Market EOD Data Daily Historical Data - Alphavantage Intraday Trading System with Daily Ranking Daily US Stocks Data	How-to Lessons How to import trading data from CSV files How to use QuantShare with Metastock data How to speed up watchlist and screener plug-ins when working with... How to screen for stocks having a high correlation with the Dow J... How to get fundamental data for U.S. Stocks
Related Forum Threads Need help with a MM Script Clear Symbol data stored in memory in global script ASCII IMPORT -HOW CANCWE DELETE DATA FOR A WRONG DATE Access intraday data while working on daily chart Importing daily metastock data	Blog Posts How to Import Metastock data into QuantShare How to Automatically Import Local Data Into QuantShare How to deal with StockTwits data QS Trading Software: Global Script How to make sure your Historical Price Data is Split Adjusted