Right now our setup is CloudFlare > 1x HAProxy (Dallas,TX), 3x App servers (Dallas,TX), 3x MariaDB nodes in a Galera multi-master cluster (Dallas,TX). The App servers auto-scale horizontally to handle whatever traffic gets thrown at them. HAProxy and DB servers are scaled up when needed. The App servers also maintain individual SQL read caches to ease the load on the DB.
Our main selling point is keeping our product online no matter what happens, as our solution caters to government emergency services, and since we're a start-up, our budget is really tight.
It was unexpected that our current cloud provider (which we cannot change in the following months) is experiencing DDOS for around 30minutes a month. For our situation not even one minute of downtime is acceptable.
So I am thinking of going with something like CloudFlare > 3x HAProxy (Dallas,TX; Freemont,CA; Newark,NJ) > 6x App servers (Dallas,TX; Freemont,CA; Newark,NJ) > 3 Galera Multi-master + 2 Galera Arbitrator (Dallas,TX; Freemont,CA; Newark,NJ; Atlanta,GA; San Francisco,SF).
This will increase our costs quite a bit, and that is why I want two Galera Arbitrators if I can skimp on costs while at the same time increase stability.
My main question is, what are the resource requirements for Galera Arbitrator, and would it be worth having three MariaDB nodes and two Galera Arbitrator nodes, instead of only three MariaDB nodes in the cluster? Another way to put it is, is it worth using garbd on setups over three nodes in size?
Provided that route-wise our HAProxy servers will choose the App servers in the same region, which subsequently choose the DB server in the same region. The choices are based on a blend of connect time and server load.