I have been struggling with this issue for a while now. I have a simple Master-Slave setup with MySQL 8.0.15 installed on both systems. In day-to-day operations the slave keeps up the Master with no issue, but unfortunately with our network, the connection between the two server will sometimes be down for a couple hours or more. When this happens and the connection finally comes back up, it appears the relay logs are written on the slave at the expected speed (network speed), but the application of these logs into the MySQL database is not reaching the full potential of the physical disks.
The RAM and CPU allotted are not anywhere near capped either. When replication is catching up on the slave, the disks only reach around 600KB/s, but in typical operations (Windows operations) these SSDs can reach > 500MB/s. I know these are more 'random' writes, but I was not expecting this much of a performance drop off. This is actually the same speed as a similar setup we have that use only spinning disks spec'ed at far less IOPS.
The hardware/software is the same on the master and slave: The data partition is setup in a separate RAID-10 of SSDs with the OS and application on a separate RAID 1, 64GB RAM, 88 'CPU's, both are running Windows 2016 with MySQL 8.0.15, and they are using similar configuration files. Here are the startup configs I am currently using:
All of our databases use solely innodb tables. I have attempted various optimizations from my research online. I have tried adjusting slave-parallel settings and various innodb settings, none of which seem to have an affect on disk throughput when the slave is catching up. Does anyone see anything here I need to adjust or any recommendations on how to go about troubleshooting this? Or even if you think I should be looking at hardware vs. software? Thank you!
Please let me know if you need any more information.