Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I'm php developer so dont be strict. I have a big table ~5.5gb dump. Our PM decided to make new column in it to perform new feature. Table is InnoDB So what i tried:

  1. Alter table in screen with table lock. Took ~30hours and nothing. So i just stopped it. First i made mistake cause didn't end all transactions but 2nd time was no multilock. Status was copy to tmp table.

  2. Since i also need to apply partitioning for this table we decide to make dump, rename and make table with same name and new structure. But dump is making strict copy(atleast i didnt found something else). So i added to dump a new column with sed and query it. But some strange errors began. I believe it was caused by charset. Table in utf-8 and file became us-ascii after sed. So i got errors(unknown command '\'') on 30% of data. So this is also a bad way.

What are other options to accomplish this and speed performance(i can do it with php script, but it will took ages). What will be performance of INSERT SELECT in this case.

Thanks for any advance.

share|improve this question
    
yeap . thanks . –  ineersa Jun 18 '13 at 13:00
add comment

migrated from stackoverflow.com Jun 18 '13 at 17:45

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 4 down vote accepted

Use MySQL Workbench. You can right-click a table and select "Send to SQL Editor" --> "Create Statement". This way no table "properties" will be forgotten to add (including CHARSET or COLLATE).
With this huge amount of data I'd recommend cleaning up either the table or the data structure you use (a good DBA comes handy). If not possible:

  • rename the table (ALTER) and create a new one with the CREATE script you get from Workbench. You can also extend that query with the new field you need
  • BULK LOAD the data from the old table to the new one:
    SET FOREIGN_KEY_CHECKS = 0;
    SET UNIQUE_CHECKS = 0;
    SET AUTOCOMMIT = 0;
    INSERT INTO new_table (fieldA, fieldB, fieldC, ..., fieldN)
       SELECT fieldA, fieldB, fieldC, ..., fieldN
       FROM old_table
    SET UNIQUE_CHECKS = 1;
    SET FOREIGN_KEY_CHECKS = 1;
    COMMIT;
    

    This way you avoid indexing/etc to run record by record. The "update" to the table still will be slow (as the amount of data is huge) but this is the fastest way I can think of.

    EDIT: Read this article to get details about the commands used in the above sample query ;)
share|improve this answer
    
My options is fine. And i got SET NAMES utf8 and COLLATION.But meh idk why 30% of data corrupted after sed. I think bulk load will be the fastest but maybe something more exist that i'm missing. Thank you Mark –  ineersa Jun 18 '13 at 13:13
1  
@ineersa data corruption can have many reasons: e.g. you opened the file with an editor that doesn't support all chars and saved it. Or, the way you try to import from dump corrupts the data (it's buggy and can't read the file properly). Or, same guy may identify part of some data as an expression (e.g. "james\robin" == "\r" as expression) or command, etc. This is why I never recommend using dump, not even with binary data dump tool only, not even with dev.mysql.com/doc/refman/5.6/en/mysqldump.html (or BCP for MS SQL Server). It goes wrong far too many times... –  Mark Jun 18 '13 at 13:22
    
yeap i tried with hex-blob. it doesnt help. Also you right after using sed mysql identify \' as command in some names(not in all). Thats strange and buggy. Will try bulk load tonight. Hope it will be done atleast in 10-15 hrs. –  ineersa Jun 18 '13 at 13:38
    
@ineersa hope it will. you can also try adding only part of the data, let's say 10% of it to see how much time that takes - and have an estimate for the whole transaction. It'll be a very rough estimate though, things can go slow if caches/memory/whatever gets filled up / overloaded. –  Mark Jun 18 '13 at 13:42
1  
Thanks Mark. Worked awesome. Even faster then restore from dump. Took ~ 5 hrs. –  ineersa Jun 19 '13 at 12:30
show 1 more comment

Your sed idea is a decent method, but without the errors or the command you ran, we can't help you.

However, a well known method for making online changes to large tables is pt-online-schema-change. The simplistic overlook of what this tool does is copied from the documentation:

pt-online-schema-change works by creating an empty copy of the table to alter, modifying it as desired, and then copying rows from the original table into the new table. When the copy is complete, it moves away the original table and replaces it with the new one. By default, it also drops the original table.

This method might also take a while to complete, but during the process the original table will be completely useable.

share|improve this answer
    
I will try bulk loading later tonight. If it wont work will need this tool probably. Errors are caused by inetifieng some symbols after using sed as commands. For example 'D\'agostini' will cause error unknown command '\''. But not always , like in 30% of cases. Thats strange and buggy. Same comes even with hex-blob dumps. Thank you Derek. –  ineersa Jun 18 '13 at 13:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.