Lost data in bulk Insert on Galera Cluster

Question

I have a 2-node fresh Galera Cluster in 2 DCs. I Import a 3GB dump into one node and the other starts to sync. After importing dump, I check the size and rows on both nodes and they are not equal. This is my galera.cnf ( Customized for WAN env )

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="my-company"
wsrep_cluster_address="gcomm://XXXX,YYYY"

# Galera Synchronization Configuration
wsrep_sst_method=rsync

# Galera Node Configuration
wsrep_node_address="XXXX"
wsrep_node_name="Node-1"

# Galera Custome setting for WAN
wsrep_provider_options="evs.suspect_timeout=PT1M"
wsrep_provider_options="evs.inactive_timeout=PT1M"
wsrep_provider_options="evs.send_window=512"
wsrep_provider_options="evs.user_send_window=512"
wsrep_provider_options="evs.inactive_check_period=PT30S"
wsrep_provider_options="gcs.fc_factor=1"
wsrep_provider_options="gcs.fc_limit=260"
wsrep_provider_options="gcs.max_packet_size=2097152"

This is Databases size after import dump:

Node1: 27329.4 MB
Node2: 17570.1 MB

This is bandwidth usage on the node that I imported dump:

Why many record did't insert? It's about mysql tuning or something else?

Ubuntu 18.0.4
mysql-wsrep-server-5.7
8 Core cpu, 16GB Ram

I hope someone can help, Thanks a lot.

Answer 1

From the documentation:

Options need to be provided as a semicolon (;) separated list on a single line.

Only wsrep_provider_options="gcs.max_packet_size=2097152" is applied, the rest of the specified options are ignored. - https://dba.stackexchange.com/u/217784

Answer 2

There a few factors that can provoke a missmatch between datasets size (the most often one is when many UPDATES or DELETE happens in one of the servers). What you can do to ensure data consistency across nodes is table checksum. If you don't have many tables this should be simple to do. Of course you need the tables without traffic to do this, not sure if this is possible.