0

I need to stream the changes to elasticsearch to keep in sync with my database(mysql). What will be best way to do this ? I have thought for below approaches

  1. Using trigger( but my table will have billions of data and the application will have frequent update so trigger may be costly) .

    2 Using delta/history table (this will add the extra effort in development and this will cause the lot many joins ) .

Is there any out of box solution or any other way which will have the better performance and less complexity.

  • billions of data... that sounds impressive. Billions of what? – Hogan Jan 14 '16 at 19:22
1

This topic is very interesting for me now, so because it Up, some last information:

Replication is good, but what if You need replicate data to different technology Target System (for example Redshift)

I found and test few implementations for change data capture:

1) MariaDB MaxScale CDC - my personal test was unsuccessful (and builded from source and binary distribution return link for missed plugin). Ask a questions, wait for answer.

2) StreamSets - simulate slave, ship changes to Kafka MQ. Tested on real loading, work perfect

3) Yelp

4) Maxwell Daemon - under test now, example with NiFi - NiFI examples

5) Python Library - also simulate a slave for remote server

6) Lapidus - NodeJS implementations

Few other links also founded, some under active maintenance, some look like forgotten.

From the tests, StreamSets look very good, Maxwell also very interesting MaxScale - sure it work, but may be available only under subscription support.

From "other" side - Talend as primary data integration tools, Redshift, memSQL.

0

I will go by using the binary log, it only reflects the changes on data or schema and its already integrated in mysql.

  • 1
    Can you provide a little more information on how one would accomplish this? One line answers aren't usually too helpful – LowlyDBA Jun 15 '15 at 14:58
  • 1
    The question is so general that the one liner fits, it just provide anoher option to investigate. Anyway, you need to translate mysql changes to elasticsearch, this part is going to be done with specific code related to the application using case 1, case 2 or my case 3. Binary log its used in replication to move changes from one mysql to another mysql. You can use mysqlbinlog utility to decode the binary part and use external code to translate to elasticsearch in a asyncronous way, even in other machine. And thats the general idea – PerroVerd Jun 15 '15 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.