We currently have a 3 node percona xtradb cluster running on premises. We are trying to add a 4th, remote node ( Obviously split-brain is a concern in this situation but it is only temporary while we get the new node running ).
The node starts fine and completes the SST, but then the wsrep_local_recv_queue continues to grow in size until the server crashes with an Error: semaphore wait has lasted > 600 seconds
I have tried setting wsrep_desync=ON which helps by not putting the rest of the cluster into flow control mode but the queue still continues to grow on the new node. From what I've read, it should be applying the queries despite being in desync mode.
I have also tried setting wsrep_provider_options="evs.send_window=512;evs.user_send_window=512" but that didn't appear to make any noticeable difference in this case.
Is there something I'm missing here? I haven't been able to get much information out of the logs but I'm not sure what to look for.
