0

I am looking for some sensible suggestions on data management tools for large data files.

I have been experimenting aith excel and access only to find that access has limitations on import files size which caught me off guard.

I've been looking at dbase and some other out of the box solutions, but just hoping to find something that best fits my needs.

My typical data processing procedure is as follows:

  1. Import large .txt files in common, pipe or other common delimited formats.

  2. Run queries on the data once imported such as merging cells, and

  3. Export my final queries into MySQL format...

The tables I am dealing with are millions of records large and as I said Access can no longer handle them..

Thanks for any ideas or suggestions on the best software for the job.

1

It sounds like you should check out MySQL's load data infile .... The docs claim that it is a very fast way to load rows. (I personally have never loaded millions of rows at once).

Once the data is loaded, you may need/want to add appropriate indexes to your tables to help speed up query performance.

For getting data out of MySQL, check out select ... into outfile.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.