What techniques do people use to upload data from a flat file into a normalised database? We are planning on using MySQL or MariaDB.

I have a number of large excel spreadsheets that I would like to import into a database. The files are in slightly different formats. We had thought of importing to a temporary table and preforming validation and data transformations on this temporary table. Then coding an sql script to insert the records in the the tables and child tables.

Is there anyway of speeding up or streamlining this process. This is a situation that I am going to have to repeat on a constant basis and I do not have any control over the format that the data is going to be provided in.

Any suggestions would be very much appreciated

closed as off-topic by Paul White, Kin Shah, Mark Storey-Smith, Michael - sqlbot, ypercubeᵀᴹ Jan 7 '14 at 16:33

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Shopping list question - questions about which tool, library, product or resource you should use are off-topic here because they quickly become obsolete and often are just about the preferences of the answerer. If you have an issue with or a question about a specific tool, please revise your question to conform to that scope." – Paul White, Kin Shah, Mark Storey-Smith, Michael - sqlbot, ypercubeᵀᴹ
If this question can be reworded to fit the rules in the help center, please edit the question.


You could investigate the MariaDB storage engine 'CONNECT'. It's in beta but could simplify your process. It is not OSX, however.



This storage engine permits you to read and write to 'other' data files. Take a look as it's a very interesting component.

Not the answer you're looking for? Browse other questions tagged or ask your own question.