Sensible data management tool for importing and exporting of large data files

Question

I am looking for some sensible suggestions on data management tools for large data files.

I have been experimenting aith excel and access only to find that access has limitations on import files size which caught me off guard.

I've been looking at dbase and some other out of the box solutions, but just hoping to find something that best fits my needs.

My typical data processing procedure is as follows:

1. Import large .txt files in common, pipe or other common delimited formats.

2. Run queries on the data once imported such as merging cells, and

3. Export my final queries into MySQL format...

The tables I am dealing with are millions of records large and as I said Access can no longer handle them..

Thanks for any ideas or suggestions on the best software for the job.

Answer 1

It sounds like you should check out MySQL's load data infile .... The docs claim that it is a very fast way to load rows. (I personally have never loaded millions of rows at once).

Once the data is loaded, you may need/want to add appropriate indexes to your tables to help speed up query performance.

For getting data out of MySQL, check out select ... into outfile.