Guess what ? You have a nasty little bug (Replication stall with multi-threaded replication)
This bug was first reported in MySQL 5.6.17 on 20 Jun 2014 15:45
#SUGGESTION #1
You need to look over all the release notes from MySQL 5.6.18
up to MySQL 5.6.22
to see if it was resolved. If any of those release notes claim the problem was fixed, then someone missed a patch. I do advise that you do not regress to anything before MySQL 5.6.21 (Any known issues upgrading from MySQL 5.1.73 to 5.6.21?Any known issues upgrading from MySQL 5.1.73 to 5.6.21? under ISSUE #3 : Security Issues
)
#SUGGESTION #2
Don't use multithreaded replication until this bug is fixed.
#SUGGESTION #3
In the aforementioned bug report, it says this somewhere in the middle
When MySQL comes back up again, the log is rotated, and IO thread starts writing from the event group which was partially written.
. . .
You can see that the "end_log_pos" values are the same for the partial event I showed earlier and this one. It is the same UPDATE transaction that was partially written.
Now comes the interesting part. When the coordinator reads the relay log, it sends the partial event to the worker, since the event is partial hence the worker never commits the transaction and the transaction is kept open. The coordinator reads the next event (which is the full version of the partial event) but the coordinator cannot assign the next event to another worker because of one of the workers having an open transaction.
And apparently, MTS waits for workers to commit transactions when it sees log rotated.
I would suggest not using --flush-logs
with mysqldump, just in case there are some issues with group commits that are all-of-a-sudden split by the mysqldump and this bug isn't viewing incoming binlog events correctly.