It's impossible to tell without investigation. Potentially it could be mistakes when you created the slave, it could be direct writes to the slave, bugs in replication, bugs in an app with
SET binlog=0, etc.
Worse, it's very hard to investigate post-factum. You need to prepare for the data drift troubleshooting.
pt-table-checksum would be your first step. It's a mature tool, safe to use on prod, but problems are possible, too. Most likely, locking issues, deadlocks. If need to be extra paranoid, I set
--chunk-time to something like 0.1 or even less.
Find differences in the data, review it (you may want to use twindb_table_compare to find which rows are missing/extra/different), and fix the mismatches (with
pt-table-sync or rebuild the slave depending how big the data drift is).
pt-table-checksum in cron. If one run takes long time (say 1 - 2 days) I recommend to split discovery and alerting parts (i.e. the heavy
checksums and a light script checks
checksums and alerts if inconsistencies are found). On my systems I run
Then, when an inconsistency is discovered inspect the data, the binlog and then you'll figure out where it comes from.