As you may know, the problem with master-master setup is conflicts. If the same row is (for example) deleted and updated at the same time on different masters, this will result in a replication breakage. You mention that you want to do this only for failover, but then this risk remains during failover. For example, you want to upgrade master1. You switch the DNSs. New writes are directed to master2. A long transaction completes on master1, it's replicated to master2, and a conflict happens.
This can also happen if whatever you are using to automate the failover cannot connect to master1 for some seconds, but master1 is still running. It switches. New writes go to master2. Some clients are still connected to master1, they write, conflict happens.
Of course reusing the same connections (for example via a proxy) will increase this risk - but still, I consider it as a good optimization.
Other possible disaster causes include human errors. You do something manually with your data, but you do it on master2 by mistake, et voila' a conflict appears.
This should answer your question "what could go wrong", but there is one more important point: someday you may need to scale more, and not only reads. In most cases, one cannot simply distribute the writes and maybe even add one more master - because of conflicts.
I am not saying that master-master is bad. I use it. But I suggest you to evaluate 2 alternatives, in case you didn't:
- Cluster (like Galera). Conflicts are not persisted: if 2 simultaneous queries on different servers produce a conflict, one of them will fail. Well, there are edge cases actually. But the worse thing I've seen in edge cases is one single node crashing. You still have 2. Of course clusters have also a lot of disadvantages, like expensive writes and DBAs committing suicide whenever an SST happens. So I'm not suggesting it, I'm just pointing out that I would consider this option.
- More traditional environment with 1 master and N slaves, plus Orchestrator or similar software. If Orchestrator cannot connect to the master, it checks if slaves can. If they can't, it elects one of them for failover. It keeps into account several aspects, like server version and binary log format. And replication lag of course - but instead of relying on Seconds_behind_master (which is unreliable and has ridiculously low granularity) it checks the last replicated transaction. Also, it uses its own implementation of GTID - which is useful for you because 5.5 doesn't have it, and for lots other people because it's the more error-prone feature ever appeared in a piece of software (not to mention its bugs).
If you eventually decide to use master-master, I suggest to regularly test failover during normal work hours with a non-crazy traffic. So you will avoid bad surprises when the situation is bad.
Hope this helps.
