Click here to Login





                                                   Help With Daily Data Import Script

  1

0
Dave W.
2013-09-12 02:56:19


I want to use the process defined in the article "How to Automatically Import Local Data Into QuantShare" here:
Read more: http://www.quantshare.com/sa-552-how-to-automatically-import-local-data-into-quantshare#ixzz2ecwjLDcz

Buy my situation is a bit different, and I'm hoping you can help. After each trading day, my data service software downloads three files I need to import, and each is stored in a separate directory: 1 file for AMEX market in one directory, one file for NASDAQ in another, and one file for NYSE in another. Each day, the download service software adds a new file to each directory with a name in the format: AMEX_20130911.txt. Since there could be hundreds or potentially thousands of daily files in each directory, I need to only grab and import the most recent file in each of the three directories.

I'm guessing there is an easy modification to the script (reposted below) that would allow me to only grab the most recent .txt file in each directory by last modified date. Can you help me with that?



Here's the original code for reference:

This script allows you to dynamically create URLs or file paths to parse.
- Type the following script then click on "OK"

string[] files = System.IO.Directory.GetFiles(@"C:\data\", "*.csv");
int count = files.Length;
for(int i=0;i < files.Length;i++)
{
string file = files[i];
Functions.AddURL(file, "Parsing: " + i + "/" + count);
}

Read more: http://www.quantshare.com/sa-552-how-to-automatically-import-local-data-into-quantshare#ixzz2ecxvVwX4






QuantShare
2013-09-12 13:35:40

  0

An easier solution for this, would be to create a new downloader, create 3 URLs (one for each directory).

Example:
c:\directory1\AMEX_[DATE].txt

Click on "Fields" button under "Fields" column and add a date field (type). Make sure the checkbox (last column) is checked.

Test the parsing by clicking on "Parser" button.



qTester
2013-10-09 14:54:53

  0

file-by-file import is very(!) slow (707 requests in 30 minute(s) and 50 second(s).)
is any other ability to bulk update several hundreds of files, which (externaly)refreshing every 1 minute inside some local directory on HDD?



QuantShare
2013-10-09 21:42:05

  0

What is the size of each file?
In the downloader, you can define the number of threads to use ("Threads" option).
You can define the maximum number of threads to allow, by selecting the download item then selecting "Update -> Download Settings".



No more messages
0




Reply:

No html code. URLs turn into links automatically.

Type in the trading objects you want to include: - Add Objects
To add a trading object in your message, type in the object name, select it and then click on "Add Objects"










QuantShare

Trading Items
EODData Daily Import
Daily Stock Quotes - Stock Market EOD Data
Daily Historical Data - Alphavantage
Intraday Trading System with Daily Ranking
Daily US Stocks Data

How-to Lessons
How to import trading data from CSV files
How to use QuantShare with Metastock data
How to speed up watchlist and screener plug-ins when working with...
How to screen for stocks having a high correlation with the Dow J...
How to automate the data downloading process

Related Forum Threads
Need help with a MM Script
Clear Symbol data stored in memory in global script
ASCII IMPORT -HOW CANCWE DELETE DATA FOR A WRONG DATE
Access intraday data while working on daily chart
Importing daily metastock data

Blog Posts
How to Import Metastock data into QuantShare
How to Automatically Import Local Data Into QuantShare
How to deal with StockTwits data
QS Trading Software: Global Script
How to make sure your Historical Price Data is Split Adjusted









QuantShare
Product
QuantShare
Features
Create an account
Affiliate Program
Support
Contact Us
Trading Forum
How-to Lessons
Manual
Company
About Us
Privacy
Terms of Use

Copyright 2021 QuantShare.com
Social Media
Follow us on Facebook
Twitter Follow us on Twitter
Google+
Follow us on Google+
RSS Trading Items



Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.