1

I am trying to design a fully redundant MySQL Cluster implementation for a service provider's network. Nodes within this cluster will be geographically separated between towns in the state that the service provider resides. Though I understand that using Geographic Replication for MySQL Cluster requires an asynchronous replication configuration, since the two data centers that the cluster nodes will reside in are connected via a high bandwidth (10Gbps), low latency pipe (~2ms), my goal is to have geographic diversity and synchronous replication across these data centers.

The nodes per data center break down like this:

Data Center A:

  • Management Node 1
  • SQL(API) Nodes 1 and 2
  • Data Node 1 (Node Group 1) and Data Node 2 (Node Group 2)

Data Center B:

  • Management Node 2
  • SQL Node 3
  • Data Node 3 (Node Group 1) and Data Node 4 (Node Group 2)

In this scenario, data node redundancy is strong (I believe), considering that a node from each node group will reside at each data center and a given data node in node group 1 (in either data center A or B) will have half of the data while the data node from the other node group in this same data center will have the other half of the data. In the instance of an isolation event or other type of failure that breaks apart the data centers, each location will have at least a full copy of the data.

My concern now becomes having a management node located in each data center. I am not sure if redundancy for the management nodes is best implemented this way or not. From what I understand about MySQL Cluster, if an isolation event occurs at one of the data centers (let's say data center A) and the connection between these two locations is lost, data nodes in data center B would start looking at management node 2 in this data center as the arbitrator, instead of the management node 1 in data center A, which would have originally been the arbitrator.

In my mind, this situation creates a split brain operation, where each data center is looking at its own arbitrator and, as a result, you still have all data nodes turned up in the cluster, yet only half of the storage locations are being written to. You break your synchronous environment.

Should I instead have the redundant management node also located in data center A on a separate physical resource, or does this change nothing about the logic of the cluster? Should I go ahead and instead create an asynchronous replication environment for the database and have each data center house its own cluster instead of breaking a cluster out between two geographic locations?

Warning: This is my first database project ever, so I am quite new to this stuff. I may be completely butchering this design implementation.

0

If you have split exactly in half brain situation MySQL Cluster resolves it designating one node as master. If you look into ndb_mgm output below you will notice asterisk at the end of node 10 line. It marks master.

It means that if your cluster will be split in two equal parts the half which contains master will continue running. Nodes from the other half will be shut down.

ndb_mgm> show
Connected to Management Server at: 10.1.1.10:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=10   @10.1.1.20  (mysql-5.6.24 ndb-7.3.9, Nodegroup: 0, *)
id=11   @10.1.1.21  (mysql-5.6.24 ndb-7.3.9, Nodegroup: 0)
  • Thank you for your response, Mindaugas. What happens in the instance where the data center who's hosting the master node is isolated and I still need availability in the other data center? This implementation will be used for RADIUS, btw. So authentication would still need to be available via the other data center, but from what I understand the non-master node community would shut down if the master node community was isolated. Currently, our RADIUS database is hosted in one location and if there is an isolation event at this location all subscribers are affected. – Chris Telinde Oct 23 '15 at 15:09
  • If you restart not master node part nodes after brain split after StartPartialTimeout+StartPartitionedTimeout half of the cluster will start working [almost] normally. But MySQL Cluster is not designed for such operations and of course nodes will not resync after isolation will end. Did not try such scenario but I believe that then again you will have to stop half of the cluster and restart data node with --inital. – Mindaugas Riauba Oct 23 '15 at 17:25
  • Thanks again for the feedback. It seems like multi-master geographic asynchronous replication between the sites may be the way to go with this, then... – Chris Telinde Oct 23 '15 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.