Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I am looking for some sensible suggestions on data management tools for large data files.

I have been experimenting aith excel and access only to find that access has limitations on import files size which caught me off guard.

I've been looking at dbase and some other out of the box solutions, but just hoping to find something that best fits my needs.

My typical data processing procedure is as follows:

  1. Import large .txt files in common, pipe or other common delimited formats.

  2. Run queries on the data once imported such as merging cells, and

  3. Export my final queries into MySQL format...

The tables I am dealing with are millions of records large and as I said Access can no longer handle them..

Thanks for any ideas or suggestions on the best software for the job.

share|improve this question
add comment

1 Answer

It sounds like you should check out MySQL's load data infile .... The docs claim that it is a very fast way to load rows. (I personally have never loaded millions of rows at once).

Once the data is loaded, you may need/want to add appropriate indexes to your tables to help speed up query performance.

For getting data out of MySQL, check out select ... into outfile.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.