We seem to have a strange issue from time to time during heavy write workload , a slave in a different data center starts to lag behind the master but when we STOP and START the slave the it starts to catch up quickly. We have a slave in the same data center as the master running the same configuration and does not suffer from this issue as replication lag at maximum reaches 1 second between these two but on the different data center it goes past 10-15 minutes until we have to restart the slave which recovers in a couple of minutes.
The issue occurs during a specific batch job we perform daily and checked that no slow queries are the culprit, CPU and I/O seem to be utilized therefore we know MySQL is doing something but are trying to figure out why a stop and start of the slave would make it work more "efficiently" ?
Update: Issue still seems to occur till this day and only a restart of the slave thread or mysql process mitigates the issue for a couple of days until problem starts to arise again. One thing we noticed is that one of the CPU's on the server (which most likely the slave thread is running on) has a high constant usr time which we find odd when we know it is not currently processing any data.
Would you recommend any tools or commands on how to investigate what exactly the thread might be doing? There doesn't seem to be any slow queries as we set the long time to 0 and log_slow_slave_statements 1 .