I need to set up the replication from existing MySQL database into a an empty database in an existing Clickhouse cluster. Since the most simple solution, clickhouse-mysql seems to be broken (stumbles on NULL values and crashes), I'm trying to get used to something more complicated, for instance several ways to to this using Kafka (currently I'm playing with the maxwell.io). I've set up a demo mysql-kafka-clickhouse chain, seems to be generally working (although I need to polish it a bit to make the nested JSONs to work properly when updating the Clickhouse MWs); but I still have no clue about the major step: how do I sync the initial state of the MySQL database ?

As far as I understand, the Clickouse reads Kafka topic, then transforms it using a materialized view and ... a table. The relationship between the MW and this table is unclear to me though: do I need to fill this table with the initial state of the MySQL database contents ?

All of the solutions tend to focus on how the changes will be transported, they explain how binlogs are read, and even how do you position the pointer inside the binlogs to start reading from. But how do I transport the data that already exists inside the MySQL database ? Because as long as it's not updated it will be missing from Clickhouse.

Seems like I'm lacking some essential part, I guess it will be enough to just to point out the part of the documentation I've missed either in Debezium or in Maxwell.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.