0

Setup:
I have a single RDS cluster. This cluster has two instances, one in each zone. Naturally, one "replication role" reader and one writer.

What happened:
I manually triggered a failover of the main instance from the RDS console "Instance actions" menu. After doing so, the console eventually showed this change including the swap of reader and writer roles. However, the replica instance remained read only so all writes were failing. This replica instance's Parameter Group settings have read_only:TrueIfReplica.

Questions:
What other information can I provide to clarify my configuration or issue?
Any clues on why this replica instance stayed read only?
If this is the expected behavior of read_only:TrueIfReplica, where else should I look to fix my configuration?

Cheers!

0

One thing I need to answer this question. Which endpoint you are using in the application Instance or Cluster?

In Aurora there are two types of endpoints. 1. Instance endpoint 2. Cluster Endpoint

Instance Endpoint:

This endpoint will always points to that particular instance whether it is read or writer. It never care about the role, it'll point that particular instance.

Clusrer Endpoint:

Actually it has 2 endpoints. One is for Writer and another one os for reader. This is the endpoint we should use in the application. Because unlike Normal RDS, Aurora's failover and MultiAZ is different. So these cluster endpoints will point bases on the role.

Writer endpoint will always point to the writer role instance, after failover this endpoint will go to the new writer. Like the same for Read endpoint.

Answers for your three questions

  1. What other information can I provide to clarify my configuration or issue?

Check which endpoint you are using.

  1. Any clues on why this replica instance stayed read only?

If you are using instance endpoint, after the failover the application still refers the same instance for write. Thats why its failing.

  1. If this is the expected behavior of read_only:TrueIfReplica, where else should I look to fix my configuration?

Nothing, just again check which endpoint you are using.

  • This is helpful, thanks, and I should have added this detail from the beginning. We are actually using the cluster endpoint. So, when the failover occurred (and DNS propagated), some services initiated new connections with the failover instance. We observed failed writes. I'll double check with my team that we're sure the failed writes were directed at the failover instance and not connections that had not yet transitioned; I did not force restart the primary instance. Does Aurora kill existing connections on failover? – mdtheory Aug 12 '18 at 10:07
  • yes, during the failover, the existing connections will be dropped. My suggestion is, get the writer cluster endpoint and login. create a test table. insert few records. Then do the failover. again login with same endpoint and insert one more. then you will confirm aurora is fine. use this command to check which is primary select server_id from aurora_replica_status() where session_id='MASTER_SESSION_ID'; Let me know if you found anything strange. – Bhuvanesh Aug 12 '18 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.