DISCLAIMER : Not Familiar With TokuDB Storage Engine
My guess for the growth would that data and indexes are being written contiguously. This is what InnoDB experiences when running
OPTIMIZE TABLE against a table with
innodb_file_per_table disabled. This makes
ibdata1 grow. The Undo Logs grow with. Although I do not know TokuDB internals, optimization would lead to file growth.
Create the table from scratch. For a TokuDB table
mydb.mytable do the following:
CREATE TABLE mytablenew LIKE mytable;
INSERT INTO mytablenew SELECT * FROM mytable;
ALTER TABLE mytable RENAME mytableold;
ALTER TABLE mytablenew RENAME mytable;
mytable is smaller and is DB accessible, then get rid of the old one with
DROP TABLE mytableold;
Give it a Try !!!
As for TokuDB itself, I wrote an earlier post stating how the Storage Engine spends a lot of its time doing data compression (See my post for the question TokuDB not much faster than MySQL and another answer from a TokuDB Engineer). IMHO I suppose running data compression on a table already optimized may just result in more CPU and Server Load.