We've been trying to use the Bitnami PostgreSQL HA chart (GitHub repo) in our Kubernetes environment to meet our high availability needs, but keep running into issues where we'd eventually get into a split brain scenario.
What we noticed was that when the primary replica's pod would be terminated, regardless of why, be it by the Kubernetes scheduler, manually, etc., eventually (if not the first time) a standby replica would takeover the role of primary, but when the primary pod would be restarted, it would still maintain the role of primary, thus a split brain situation.
This could be confirmed looking at the output from the repmgr
's cluster show
command output, as well as querying the repmgr
's tables directly. The three tells from the cluster show
command output included:
- The value of the
Role
column, - The value of the
Status
column, and - That the
Upstream
column didn't point to the primary.
Of course, this led to various data mismatch issues, i.e. client A writes to -0 and client B reads from -1 expecting to the see the data from A, only to not find it.
Eventually we checked to see if the issues we're experiencing could be reproduced using the out-of-the-box chart, and to our surprise, they could be.
We assume that this chart does work for PostgreSQL HA needs and the out-of-the-box experience requires some configuration. So, the question is, what's the necessary configuration to make it work?
So far, we're coming up short reading through the Bitnami documentation. There are several related and open issues on the Bitnami repo, which we've commented on here, but we're hoping this is something we're doing wrong, i.e. configuration, and others can help.
Steps to reproduce the issue
Install the chart
➜ ~ helm install pg-ha-test bitnami/postgresql-ha
This assumes you've already added the Bitnami repo: helm repo add bitnami https://charts.bitnami.com/bitnami
.
Verify the cluster status
➜ ~ kubectl exec pg-ha-test-postgresql-ha-postgresql-0 -- /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 14:03:35.44
postgresql-repmgr 14:03:35.45 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 14:03:35.45 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 14:03:35.45 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 14:03:35.45
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
------+---------------------------------------+---------+-----------+---------------------------------------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1000 | pg-ha-test-postgresql-ha-postgresql-0 | primary | * running | | default | 100 | 1 | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-0.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
1001 | pg-ha-test-postgresql-ha-postgresql-1 | standby | running | pg-ha-test-postgresql-ha-postgresql-0 | default | 100 | 1 | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-1.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
Delete the primary replica
To simulate a future deployment that would need to terminate a pod, as well as simulate a real pod termination due to unforeseen issues, we delete the primary pod manually.
➜ ~ kubectl delete pod pg-ha-test-postgresql-ha-postgresql-0
pod "pg-ha-test-postgresql-ha-postgresql-0" deleted
Oddities
- In this particular test, we needed to delete the primary pod twice in order to reproduce the eventual error state being reported, but we have experienced this same issue sometimes on the first try and on other attempts it took 3 or more tries. Regardless, we could eventually reproduce the error state.
- The expectation is that the standby replica would take over as the primary. However, we often noticed that after the primary was terminated, it would restart as the primary, even if the error wasn't immediately reproduced. Or said another way, the primary always seemed to restart as the primary and only when we got into the error state would the standby report that it was a primary.
Verify cluster status
From the original primary replica:
➜ ~ kubectl exec pg-ha-test-postgresql-ha-postgresql-0 -- /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 14:09:07.02
postgresql-repmgr 14:09:07.02 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 14:09:07.03 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 14:09:07.03 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 14:09:07.03
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
------+---------------------------------------+---------+----------------------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1000 | pg-ha-test-postgresql-ha-postgresql-0 | primary | * running | | default | 100 | 1 | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-0.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
1001 | pg-ha-test-postgresql-ha-postgresql-1 | standby | ! running as primary | | default | 100 | 2 | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-1.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
WARNING: following issues were detected
- node "pg-ha-test-postgresql-ha-postgresql-1" (ID: 1001) is registered as standby but running as primary
command terminated with exit code 25
From the original standby replica:
➜ ~ kubectl exec pg-ha-test-postgresql-ha-postgresql-1 -- /opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show
postgresql-repmgr 14:09:32.42
postgresql-repmgr 14:09:32.42 Welcome to the Bitnami postgresql-repmgr container
postgresql-repmgr 14:09:32.42 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql-repmgr
postgresql-repmgr 14:09:32.42 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues
postgresql-repmgr 14:09:32.43
WARNING: following issues were detected
- node "pg-ha-test-postgresql-ha-postgresql-0" (ID: 1000) is running but the repmgr node record is inactive
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
------+---------------------------------------+---------+-----------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1000 | pg-ha-test-postgresql-ha-postgresql-0 | primary | ! running | | default | 100 | 1 | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-0.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
1001 | pg-ha-test-postgresql-ha-postgresql-1 | primary | * running | | default | 100 | 2 | user=repmgr password=0EPD7026uM host=pg-ha-test-postgresql-ha-postgresql-1.pg-ha-test-postgresql-ha-postgresql-headless.default.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5
command terminated with exit code 25
NOTE that each replica reports both as primary, albeit with slight verbiage difference from the original primary replica in the Role and Status columns than it reports for the original standby replica, and that neither replica is communicating with the other per the Upstream column.