2013. Lost connection to MySQL server during query error

Question

I have a table with close to 20 million rows.

In phpMyAdmin, I attempt to add a new column:

ALTER TABLE `myTable` ADD `newColumn` TINYINT( 1 ) UNSIGNED NOT NULL DEFAULT '0';

After about 60 secs (every time), I get the error.

I've had my server guy increase both net_read_timeout and net_write_timeout from 60-600 seconds, but the error still occurs after 60 secs.

Just wanted to see if I'm going in the right direction here, or should be chasing something else?

The error log from just prior to the crash says (nothing before this for a long time):

150209 05:03:39 mysqld_safe Number of processes running now: 0
150209 05:03:39 mysqld_safe mysqld restarted
2015-02-09 05:03:40 0 [Warning] option 'interactive_timeout': unsigned value 9999999999 adjusted to 31536000
2015-02-09 05:03:40 0 [Warning] option 'wait_timeout': unsigned value 9999999999 adjusted to 31536000
2015-02-09 05:03:40 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-02-09 05:03:40 6744 [Note] Plugin 'FEDERATED' is disabled.
2015-02-09 05:03:40 6744 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2015-02-09 05:03:40 6744 [Note] InnoDB: The InnoDB memory heap is disabled
2015-02-09 05:03:40 6744 [Note] InnoDB: Mutexes and rw_locks use InnoDB's own implementation
2015-02-09 05:03:40 6744 [Note] InnoDB: Memory barrier is not used
2015-02-09 05:03:40 6744 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-02-09 05:03:40 6744 [Note] InnoDB: Using Linux native AIO
2015-02-09 05:03:40 6744 [Note] InnoDB: Not using CPU crc32 instructions
2015-02-09 05:03:40 6744 [Note] InnoDB: Initializing buffer pool, size = 54.0M
2015-02-09 05:03:40 6744 [Note] InnoDB: Completed initialization of buffer pool
2015-02-09 05:03:40 6744 [Note] InnoDB: Highest supported file format is Barracuda.
2015-02-09 05:03:40 6744 [Note] InnoDB: The log sequence numbers 108217366409 and 108217366409 in ibdata files do not match the log sequence number 108487328890 in the ib_logfiles!
2015-02-09 05:03:40 6744 [Note] InnoDB: Database was not shutdown normally!
2015-02-09 05:03:40 6744 [Note] InnoDB: Starting crash recovery.
2015-02-09 05:03:40 6744 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-02-09 05:03:41 6744 [Note] InnoDB: Restoring possible half-written data pages 
2015-02-09 05:03:41 6744 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 394549, file name sbdb-bin.000108
2015-02-09 05:03:42 6744 [Note] InnoDB: 128 rollback segment(s) are active.
2015-02-09 05:03:42 6744 [Note] InnoDB: Waiting for purge to start
2015-02-09 05:03:42 6744 [Note] InnoDB: 5.6.22 started; log sequence number 108487328890
2015-02-09 05:03:42 6744 [Note] Recovering after a crash using sbdb-bin
2015-02-09 05:03:42 6744 [Note] Starting crash recovery...
2015-02-09 05:03:42 6744 [Note] Crash recovery finished.
2015-02-09 05:03:43 6744 [Note] Server hostname (bind-address): '*'; port: 3306
2015-02-09 05:03:43 6744 [Note] IPv6 is available.
2015-02-09 05:03:43 6744 [Note]   - '::' resolves to '::';
2015-02-09 05:03:43 6744 [Note] Server socket created on IP: '::'.
2015-02-09 05:03:43 6744 [Note] Event Scheduler: Loaded 0 events
2015-02-09 05:03:43 6744 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.22-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)

Thanks for your time and help.

Answer 1

So, if I've understood your log file correctly, you attempt to add a new column to your 20 million row table, the system crashes and then is rebooted by the mysqld_safe daemon? Please tell me that you're not really going to call it "newColum" in production.

What's strange (to me at least) is that your crash doesn't appear to be related to a time-out - you have it set to 9999999999 and it gets adjusted to 31536000 by the server. That's the number of seconds in a year (which should hopefully be long enough for any operation! :-) ).

What I would recommend is that you try two tools on your test system and then use the one that works best for you on production. There is a third tool from Facebook, but it's written in PHP, I know nothing about it and it doesn't seem to get much "airtime" in any case.

The first tool is from Percona whose reputation in the MySQL sphere is stellar and is written in Perl. The second is from Shlomi Noach (a big hitter in the MySQL world) and is written in Python. Thankfully, you're using Unix of some sort - these tools (AFAICS) only work on Unix environments - the Facebook one may or may not work on Windows - as I said, I know nothing about it.

1. pt-online-schema-change "ALTER tables without locking them."

2. oak-online-alter-table "Perform a non-blocking ALTER TABLE operation"

3. Facebook Online Schema Change (OSC) "This is a small but complex utility to perform online schema change for MySQL. We call it OSC and the source is here".

Let us know how you get on.

Answer 2

Check your memory usage or, more probably, your disk usage (is there enough free space during the process?). Altering a table may require either a large amount of memory or a copy on disk of your table. Changing the alter algorithm from INPLACE to COPY can be even faster in your particular case.

You may also be hitting the innodb_online_alter_log_max_size limit, although in that case, only the query should fail, not the entire server. It is possible that the crash may be happening due to the ROLLBACK, and not the operation itself, though.

Finally, some application configurations or hosting servers cancels a query/http request that is taking too long, I recommend you to execute the same query on the command line client for testing purposes.