Synchronise data from one table to another & vice-versa using triggers

Question

We are migrating from one database structure to another - very slowly. The system is MySQL. There are two databases. For example's sake, we will call the databases old_db and new_db. Both databases have tables that will need to be synchronised for around a year whilst the business's application is migrated from the old db to the new db.

Again to slim down the example, we have one table in the old database, called: old_db . myoldertable, new_db . mynewtable.

Structure:

old_db.myoldertable
primary_id, Date, amount, unneedded_column1, unneedded_column2 ... and so-on

new_db.mynewtable
primary_id, date, amount, old_table_primary_id

They look similar in structure, but this is simplified.

Problem:

• If I put an insert/update/delete trigger on both tables, I will end up in a recursion loop. This is because the tables need to be synchronised, so the insert / update / delete triggers need to be on both tables.

Requirements

• Must be 100% MySQL. Cannot use a programming language for this.
• The new and old database tables must be synchronised immediately if either have rows that are inserted/updated/deleted.
• Daily / hourly updates are NOT an option. Again, must be synchronised immediately.

Solutions?

• Ideas to break possibility of recursion for all three insert / update / delete scenarios.
• Any other ideas.

EDIT :

So far I have looked at

• One solution I have explored is replacing the old table with a view. However, our new database has two tables (or more) where the old database had only one table. Views, in MySQL cannot be writeable to more than one base table. Also views cannot have triggers in MySQL.

  • I also looked at using a UUID that represents the last update. I would create a column in both tables to contain a Unique ID. Making sure that both tables contain the same UUID as part of the triggers. However, this does not stop recursion.

Answer 1

I concur with @NathanJolly, this is a potential data nightmare and seems ill-advised... but there is a way to implement this that is, at least conceptually, fairly straightforward.

You need not worry about the possibility of recursion in triggers, because this won't happen... MySQL doesn't support it.

I built this fiddle that creates tables t1 and t2, each of which has an AFTER INSERT trigger that tries to insert the record into the other table... then tries to do an insert. Theoretically, there's a potential for recursion, but MySQL prevents this:

ERROR 1442 (HY000): Can't update table 't1' in stored function/trigger because it is already
                    used by statement which invoked this stored function/trigger.

So it won't run in a loop, but it still throws an exception. Technically, it's not trigger recursion causing the exception, because we haven't actually gotten that far... it's the fact that the trigger in t2 tries to modify table t1, which was involved in the currently-executing statement.

Then again, technically, it's not just the recursion of the triggers we need to avoid -- we have to stop one step before that -- if the client modifies t1, the t1 trigger modifies t2, then t2's trigger needs to not even try to do anything to t1 (including, but not limited to, firing the trigger again on t1).

Implementing this is actually fairly straightforward... in each trigger, we toggle the value of a session to tell the other trigger not to run.

CREATE TRIGGER t1_ai AFTER INSERT ON t1 FOR EACH ROW BEGIN
  IF @__disable_trigger_t1t2 = 1 THEN
    SET @__disable_trigger_t1t2 = NULL;
  ELSE
    SET @__disable_trigger_t1t2 = 1;
    -- trigger logic goes in here
    INSERT INTO t2 (id,things) VALUES (NEW.id,NEW.stuff);
  END IF;
END //

CREATE TRIGGER t2_ai AFTER INSERT ON t2 FOR EACH ROW BEGIN
  IF @__disable_trigger_t1t2 = 1 THEN
    SET @__disable_trigger_t1t2 = NULL;
  ELSE
    SET @__disable_trigger_t1t2 = 1;
    -- trigger logic goes in here
    INSERT INTO t1 (id,stuff) VALUES (NEW.id,NEW.things);
  END IF;
END //

A test case for this logic working can be seen in this fiddle.

Essentially, when a trigger fires, it checks the value of the @__disable_trigger_t1t2 variable (this variable name is, of course, arbitrary). If '1', it sets it back to NULL (the default for session variables) and does no further processing, because the '1' means this row is being manipulated by the trigger on the other table. If already NULL, then it sets the value to 1 and executes the trigger logic -- which will update the other table, whose trigger will see the '1', will not execute its own logic, and will reset the variable back to NULL, so that the next execution of either trigger -- whether for the next row in the same statement, or for a subsequent statement -- will see the NULL and execute normally.

In MySQL, triggers are only supported FOR EACH ROW, meaning that we can define a MySQL trigger as "a stored program that is executed before or after each row is inserted, updated, or deleted"... which means that no matter how many rows your query affects, the trigger will run once for each row. Actions against other tables will cause the triggers on those tables to fire, and the execution of the trigger on t1 for the first row will not complete until all of the triggers it caused to fire have also completed... and this is why we can use this variable like this -- only one row is actually being handled at a time, so we set and clear this variable for each row, always leaving it NULL after each statement terminates.

A significant caveat is that this approach fails if you need to trigger on cascading deletes or updates related to foreign keys, because:

"Currently, cascaded foreign key actions do not activate triggers."

^{-- http://dev.mysql.com/doc/refman/5.6/en/create-trigger.html}

So, while I have misgivings about the long-term viability of maintaining database synchronization with this approach because of the number of things that could go wrong with two applications manipulating each other's data or logical errors in the trigger definitions... it seems technically possible.

You also mentioned:

views cannot have triggers in MySQL

No, but updating an updatable view will fire the triggers on the underlying base table as if the table had been directly manipulated.

Answer 2

To avoid the loop and MySQL exception, just add a field that marks the origin of the row. Then send to corresponding tables only rows that came from themselves and ignore the others. It's a simple workaround but it works perfectly.