1

Right now our setup is CloudFlare > 1x HAProxy (Dallas,TX), 3x App servers (Dallas,TX), 3x MariaDB nodes in a Galera multi-master cluster (Dallas,TX). The App servers auto-scale horizontally to handle whatever traffic gets thrown at them. HAProxy and DB servers are scaled up when needed. The App servers also maintain individual SQL read caches to ease the load on the DB.

Our main selling point is keeping our product online no matter what happens, as our solution caters to government emergency services, and since we're a start-up, our budget is really tight.

It was unexpected that our current cloud provider (which we cannot change in the following months) is experiencing DDOS for around 30minutes a month. For our situation not even one minute of downtime is acceptable.

So I am thinking of going with something like CloudFlare > 3x HAProxy (Dallas,TX; Freemont,CA; Newark,NJ) > 6x App servers (Dallas,TX; Freemont,CA; Newark,NJ) > 3 Galera Multi-master + 2 Galera Arbitrator (Dallas,TX; Freemont,CA; Newark,NJ; Atlanta,GA; San Francisco,SF).

This will increase our costs quite a bit, and that is why I want two Galera Arbitrators if I can skimp on costs while at the same time increase stability.

My main question is, what are the resource requirements for Galera Arbitrator, and would it be worth having three MariaDB nodes and two Galera Arbitrator nodes, instead of only three MariaDB nodes in the cluster? Another way to put it is, is it worth using garbd on setups over three nodes in size?

Provided that route-wise our HAProxy servers will choose the App servers in the same region, which subsequently choose the DB server in the same region. The choices are based on a blend of connect time and server load.

Improve this question
4
  • I thought Linode was quite good at DoS mitigation and recovery. Galera Arbitrator should be forgotten about. It doesn't provide any non-toy functionality. What functionally do you want?
    – danblack
    Commented Feb 19, 2019 at 0:29
  • @danblack, funny enough, they're doing a quite good job at DoS mitigation on our other servers from other projects, which are non-critical. Never had issues with them until now. In any case, now we have all the eggs in one basket (Dallas,TX), and as I wrote, we will spread the servers between multiple regions, but I was asking if there is any merit in adding 2 more Galera Arbitrator nodes, so that we will be able to support 2 DB nodes losing connectivity instead of only one. Or is this overkill, and we should only stick with 3 MariaDB nodes in 3 separate regions?
    – Alex Protopopescu
    Commented Feb 19, 2019 at 9:45
  • with two nodes down, which other two nodes are going to doing a SST donor job? Garbs can't do that.
    – danblack
    Commented Feb 19, 2019 at 10:52
  • @danblack we are using a non-blocking SST method for the donor, so the only MariaDB server that would be left in the cluster is capable of being the donor at the cost of degraded performance. How is this situation resolved in configurations of 2x MariaDB + 1 Garbd nodes, as the one described here galeracluster.com/documentation-webpages/arbitrator.html ? I'm not bent on using Garbd if it has no advantage, but if possible, I'm trying to find a way to have the cluster survive with only one MariaDB node up from a total of 3x MariaDB nodes.
    – Alex Protopopescu
    Commented Feb 19, 2019 at 18:43

1 Answer 1

Reset to default
0

Overall I'd say its best to widen the flow control and send/recv windows, and time-outs between nodes.

If you get down to one node, you can use set global wsrep_provider_options="pc.bootstrap=TRUE" to turn it into an active node without garbs. Lacks automation however is more durable overall.

The nature of the DoS might have an impact in the design. If the app server for the site is down is there sufficiently reduce load on the galera node? Or is the network throughput/latency compromised?

Recommending sticking with a 3 node cluster initially. Its a big step up from a one site.

Improve this answer
1
  • Thank you for your comments and answer! I'll abandon the Garbd idea. We already have a 3x node cluster, just have to move our 3 nodes to separate regions, increase the timeouts as needed and tweak the load balancing for the app servers in one region to prefer their region DB node. The DoS is not happening on our servers directly, but on the infrastructure of the cloud provider, and is affecting us too. The resources are fine, as we have close to no load on all servers, we heavily optimised everything early, and in peak time there are around 2k req/h and our servers can take around 10M req/h.
    – Alex Protopopescu
    Commented Feb 20, 2019 at 3:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.