How do I prevent split brain scenarios with the Bitnami PostgreSQL HA chart?

Question

We've been trying to use the Bitnami PostgreSQL HA chart (GitHub repo) in our Kubernetes environment to meet our high availability needs, but keep running into issues where we'd eventually get into a split brain scenario.

What we noticed was that when the primary replica's pod would be terminated, regardless of why, be it by the Kubernetes scheduler, manually, etc., eventually (if not the first time) a standby replica would takeover the role of primary, but when the primary pod would be restarted, it would still maintain the role of primary, thus a split brain situation.

This could be confirmed looking at the output from the repmgr's cluster show command output, as well as querying the repmgr's tables directly. The three tells from the cluster show command output included:

• The value of the Role column,
• The value of the Status column, and
• That the Upstream column didn't point to the primary.

Of course, this led to various data mismatch issues, i.e. client A writes to -0 and client B reads from -1 expecting to the see the data from A, only to not find it.

Eventually we checked to see if the issues we're experiencing could be reproduced using the out-of-the-box chart, and to our surprise, they could be.

We assume that this chart does work for PostgreSQL HA needs and the out-of-the-box experience requires some configuration. So, the question is, what's the necessary configuration to make it work?

So far, we're coming up short reading through the Bitnami documentation. There are several related and open issues on the Bitnami repo, which we've commented on here, but we're hoping this is something we're doing wrong, i.e. configuration, and others can help.

Steps to reproduce the issue

Install the chart

➜  ~ helm install pg-ha-test bitnami/postgresql-ha

This assumes you've already added the Bitnami repo: helm repo add bitnami https://charts.bitnami.com/bitnami.

Verify the cluster status

➜  ~ kubectl exec pg-ha-test-postgresql-ha-postgresql-0 -- /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 14:03:35.44
postgresql-repmgr 14:03:35.45 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 14:03:35.45 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 14:03:35.45 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 14:03:35.45

 ID   | Name                                  | Role    | Status    | Upstream                              | Location | Priority | Timeline | Connection string
------+---------------------------------------+---------+-----------+---------------------------------------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1000 | pg-ha-test-postgresql-ha-postgresql-0 | primary | * running |                                       | default  | 100      | 1        | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-0.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
 1001 | pg-ha-test-postgresql-ha-postgresql-1 | standby |   running | pg-ha-test-postgresql-ha-postgresql-0 | default  | 100      | 1        | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-1.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5

Delete the primary replica

To simulate a future deployment that would need to terminate a pod, as well as simulate a real pod termination due to unforeseen issues, we delete the primary pod manually.

➜  ~ kubectl delete pod pg-ha-test-postgresql-ha-postgresql-0
pod "pg-ha-test-postgresql-ha-postgresql-0" deleted

Oddities

• In this particular test, we needed to delete the primary pod twice in order to reproduce the eventual error state being reported, but we have experienced this same issue sometimes on the first try and on other attempts it took 3 or more tries. Regardless, we could eventually reproduce the error state.
• The expectation is that the standby replica would take over as the primary. However, we often noticed that after the primary was terminated, it would restart as the primary, even if the error wasn't immediately reproduced. Or said another way, the primary always seemed to restart as the primary and only when we got into the error state would the standby report that it was a primary.

Verify cluster status

From the original primary replica:

➜  ~ kubectl exec pg-ha-test-postgresql-ha-postgresql-0 -- /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 14:09:07.02
postgresql-repmgr 14:09:07.02 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 14:09:07.03 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 14:09:07.03 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 14:09:07.03

 ID   | Name                                  | Role    | Status               | Upstream | Location | Priority | Timeline | Connection string
------+---------------------------------------+---------+----------------------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1000 | pg-ha-test-postgresql-ha-postgresql-0 | primary | * running            |          | default  | 100      | 1        | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-0.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
 1001 | pg-ha-test-postgresql-ha-postgresql-1 | standby | ! running as primary |          | default  | 100      | 2        | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-1.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5

WARNING: following issues were detected
  - node "pg-ha-test-postgresql-ha-postgresql-1" (ID: 1001) is registered as standby but running as primary

command terminated with exit code 25 

From the original standby replica:

➜  ~ kubectl exec pg-ha-test-postgresql-ha-postgresql-1 -- /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 14:09:32.42
postgresql-repmgr 14:09:32.42 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 14:09:32.42 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 14:09:32.42 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 14:09:32.43

WARNING: following issues were detected
  - node "pg-ha-test-postgresql-ha-postgresql-0" (ID: 1000) is running but the repmgr node record is inactive

 ID   | Name                                  | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string
------+---------------------------------------+---------+-----------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1000 | pg-ha-test-postgresql-ha-postgresql-0 | primary | ! running |          | default  | 100      | 1        | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-0.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
 1001 | pg-ha-test-postgresql-ha-postgresql-1 | primary | * running |          | default  | 100      | 2        | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-1.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5

command terminated with exit code 25

NOTE that each replica reports both as primary, albeit with slight verbiage difference from the original primary replica in the Role and Status columns than it reports for the original standby replica, and that neither replica is communicating with the other per the Upstream column.

Answer 1

You did not provide any details on how exactly you have configured repmgrd, but the default chart doesn't appear to do a good job of it. A two-node cluster without a tiebreaker (witness) is practically a guarantee of split brains -- you can't form a proper quorum with only two nodes.

If the decision to fail over and promote the standby is made, you also need to fence off the old primary and reintegrate it as a standby when it is back up.

The fact that Kubernetes pods can be restarted fairly quickly adds more complexity, as you need to be able to tolerate brief primary outages without failing over, and if you do fail over, fence the failed server before it has a chance to wreak havoc.