5

We're in the process of phasing out an old system and migrating onto a new one. The last time that we phased out an old system, we ran both systems in parallel and integrated data between both until everything was fully migrated out in our field. During that process, I was able to build an integration between our legacy system and our new system that leveraged SQL Server's Change Data Capture to track changes and integrate those over incrementally.

For this next migration, the legacy system that we will be phasing out is based on MySQL v5.1.69 instead of SQL Server. I am not familiar with MySQL and I was wondering if there are any technologies similar to CDC that can be leveraged on MySQL in our current version or a newer version that would be worth migrating towards?

  • Apache Nifi is your friend.... there is processor CapureChangeMySQL – Izzy Fayon Jul 29 at 17:21
4

LinkedIn has open-sourced their CDC tool Databus, which supports MySQL.

http://engineering.linkedin.com/data-replication/open-sourcing-databus-linkedins-low-latency-change-data-capture-system

Another open source CDC tool (implemented in PHP) was written as a part of the FlexViews project.

http://swanhart.livejournal.com/132117.html

4

Recently I had an academic project where I needed to use CDC with MySQL. There are a few tools available in the market. I was able to create a POC by using Maxwell CDC.

A unique feature of Maxwell is it can parse DDL statements as well. Here is link to the page of Maxwell http://maxwells-daemon.io

Here is complied list of all CDC's : https://github.com/wushujames/mysql-cdc-projects/wiki

3

I recommend you take a look at Debezium an open-source CDC platform which, amongst others, also supports MySQL.

You can use Debezium to stream changes into Apache Kafka, but by means of its embedded mode you can also use the Debezium connectors as a library in your Java applications and easily propagate data changes to other streaming APIs such as Kinesis etc.

Disclaimer: I'm the Debezium project lead.

0

You could try brew something based on the Triggers (AFTER INSERT, DELETE, UPDATE). I would think it depends on the Target System. How did CDC work? Did it propagate all the data and structure to a MySQL Database?

You can use the Triggers to fill a kind of work queues, so that if an entry EntryARownrX from table A gets changed, you stuff that info (table A, row nr X, Update/Insert/Delete) in a Queue Table and then transform that change to your new System (with reading the actual value of data directly from A for Insert or Update).

  • Yeah, triggers are definitely an option. The nice thing about CDC is that it would automatically create and propagate tables based on inserts, updates and deletes without manually having to create the triggers. – njkroes Apr 22 '14 at 20:49
0

If you don't mind using a database proxy, then have a look at the CDC protocol for MariaDB MaxScale which is utilising the MariaDB MaxScale AVRO router to convert binlog events into AVRO files. The CDC protocol then makes the change data capture events available to registered clients. (MaxScale should be compatible with MySQL, or at least the older MySQL versions.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.