Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm php developer so dont be strict. I have a big table ~5.5gb dump. Our PM decided to make new column in it to perform new feature. Table is InnoDB So what i tried:

  1. Alter table in screen with table lock. Took ~30hours and nothing. So i just stopped it. First i made mistake cause didn't end all transactions but 2nd time was no multilock. Status was copy to tmp table.

  2. Since i also need to apply partitioning for this table we decide to make dump, rename and make table with same name and new structure. But dump is making strict copy(atleast i didnt found something else). So i added to dump a new column with sed and query it. But some strange errors began. I believe it was caused by charset. Table in utf-8 and file became us-ascii after sed. So i got errors(unknown command '\'') on 30% of data. So this is also a bad way.

What are other options to accomplish this and speed performance(i can do it with php script, but it will took ages). What will be performance of INSERT SELECT in this case.

Thanks for any advance.

share|improve this question

migrated from Jun 18 '13 at 17:45

This question came from our site for professional and enthusiast programmers.

yeap . thanks . – ineersa Jun 18 '13 at 13:00
up vote 7 down vote accepted

Use MySQL Workbench. You can right-click a table and select "Send to SQL Editor" --> "Create Statement". This way no table "properties" will be forgotten to add (including CHARSET or COLLATE).
With this huge amount of data I'd recommend cleaning up either the table or the data structure you use (a good DBA comes handy). If not possible:

  • rename the table (ALTER) and create a new one with the CREATE script you get from Workbench. You can also extend that query with the new field you need
  • BULK LOAD the data from the old table to the new one:
    INSERT INTO new_table (fieldA, fieldB, fieldC, ..., fieldN)
       SELECT fieldA, fieldB, fieldC, ..., fieldN
       FROM old_table

    This way you avoid indexing/etc to run record by record. The "update" to the table still will be slow (as the amount of data is huge) but this is the fastest way I can think of.

    EDIT: Read this article to get details about the commands used in the above sample query ;)
share|improve this answer
My options is fine. And i got SET NAMES utf8 and COLLATION.But meh idk why 30% of data corrupted after sed. I think bulk load will be the fastest but maybe something more exist that i'm missing. Thank you Mark – ineersa Jun 18 '13 at 13:13
@ineersa data corruption can have many reasons: e.g. you opened the file with an editor that doesn't support all chars and saved it. Or, the way you try to import from dump corrupts the data (it's buggy and can't read the file properly). Or, same guy may identify part of some data as an expression (e.g. "james\robin" == "\r" as expression) or command, etc. This is why I never recommend using dump, not even with binary data dump tool only, not even with (or BCP for MS SQL Server). It goes wrong far too many times... – Mark Jun 18 '13 at 13:22
yeap i tried with hex-blob. it doesnt help. Also you right after using sed mysql identify \' as command in some names(not in all). Thats strange and buggy. Will try bulk load tonight. Hope it will be done atleast in 10-15 hrs. – ineersa Jun 18 '13 at 13:38
@ineersa hope it will. you can also try adding only part of the data, let's say 10% of it to see how much time that takes - and have an estimate for the whole transaction. It'll be a very rough estimate though, things can go slow if caches/memory/whatever gets filled up / overloaded. – Mark Jun 18 '13 at 13:42
Thanks Mark. Worked awesome. Even faster then restore from dump. Took ~ 5 hrs. – ineersa Jun 19 '13 at 12:30

Your sed idea is a decent method, but without the errors or the command you ran, we can't help you.

However, a well known method for making online changes to large tables is pt-online-schema-change. The simplistic overlook of what this tool does is copied from the documentation:

pt-online-schema-change works by creating an empty copy of the table to alter, modifying it as desired, and then copying rows from the original table into the new table. When the copy is complete, it moves away the original table and replaces it with the new one. By default, it also drops the original table.

This method might also take a while to complete, but during the process the original table will be completely useable.

share|improve this answer
I will try bulk loading later tonight. If it wont work will need this tool probably. Errors are caused by inetifieng some symbols after using sed as commands. For example 'D\'agostini' will cause error unknown command '\''. But not always , like in 30% of cases. Thats strange and buggy. Same comes even with hex-blob dumps. Thank you Derek. – ineersa Jun 18 '13 at 13:46

I think mydumper/myloader is a good tool for operations like this: Is getting better every day. You can utilise your CPUs and can load data in parallel

I have managed to load hundreds of gigabytes mysql tables in hours.

Now, when it comes to adding a new column, is tricky as MySQL copies the whole table across to memory TMP area with ALTER TABLE... Although MySQL 5.6 says it can do online schema changes, I haven't managed to do them online for massive tables with no lock contention as yet.

share|improve this answer

alter table add column, algorithm=inplace, lock=none will alter a MySQL 5.6 table without copying the table and without locking impact.

Just tested this yesterday, mass inserted 70K rows into a 280K row 7 partition table, 10K rows into each partition, with 5 seconds sleep in between to allow other throughput.

Started the mass inserts, then in separate session started the online alter statement above in MySQL Workbench, the alter finished before the inserts, two new columns were added, and no rows resulted from the alter meaning MySQL did not copy any rows.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.