0

I had created a redis cluster of 6 instances running on 2 RHEL-7.4 servers some time ago. Today I found that 3 of those instances(on 10.32.129.79) were down. Since I've restarted those 3 instances, I've observed that the command "cluster nodes" is not giving the same output on different nodes. Given below is one of the output:

dcfc502ffae3810b61368e774187406f774114d8 10.32.129.77:6379@16379 myself,master - 0 1571299699000 1 connected 0-5460
274c70586d2ac7ce2723d8a23b4f0126ddae4174 10.32.129.77:6378@16378 master - 0 1571299701559 8 connected 5461-10922
e8b9f87f369da10a3b5f4b9453aaae24c744a156 10.32.129.79:6380@16380 slave,fail 66649746bbd41f06748063cd268f71f4be063aca 1571290310404 1571290310404 5 connected
66649746bbd41f06748063cd268f71f4be063aca 10.32.129.77:6380@16380 master - 0 1571299700000 2 connected 10923-16383
40503860d939a2d70e47c48050e6eaaf109e7f3f 10.32.129.79:6379@16379 slave,fail 274c70586d2ac7ce2723d8a23b4f0126ddae4174 1571290310404 1571290310404 8 connected
8e5c582a49a868ace78d10b8b0d24c4647e1b2dc 10.32.129.79:6378@16378 slave,fail dcfc502ffae3810b61368e774187406f774114d8 1571290310404 1571290310404 6 connected

And given below is the other one:

274c70586d2ac7ce2723d8a23b4f0126ddae4174 10.32.129.77:6378@16378 master,fail? - 1571297986302 1571297981292 8 connected 5461-10922
40503860d939a2d70e47c48050e6eaaf109e7f3f 10.32.129.79:6379@16379 slave 274c70586d2ac7ce2723d8a23b4f0126ddae4174 0 1571299542894 8 connected
66649746bbd41f06748063cd268f71f4be063aca 10.32.129.77:6380@16380 master - 0 1571299538884 2 connected 10923-16383
e8b9f87f369da10a3b5f4b9453aaae24c744a156 10.32.129.79:6380@16380 myself,slave 66649746bbd41f06748063cd268f71f4be063aca 0 1571299541000 5 connected
8e5c582a49a868ace78d10b8b0d24c4647e1b2dc 10.32.129.79:6378@16378 slave dcfc502ffae3810b61368e774187406f774114d8 0 1571299543895 6 connected
dcfc502ffae3810b61368e774187406f774114d8 10.32.129.77:6379@16379 master,fail? - 1571297991309 1571297985000 1 connected 0-5460

Is there any way of recovering from this inconsistent state without losing my data? Also, in which type of scenarios can this happen?

Redis Version: 5.0.4

0

It turns out there was a firewall issue on both the servers. After stopping the firewall(systemctl stop firewalld), the cluster is consistent.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.