0

I recently set up a Postgres15 master-slave (Primary-secondary) cluster on 2 nodes. At the time of starting, I ensured that data files are exactly the same on both the nodes. The size of the DB is 1.5TB. The directory structure on both nodes looks as follows:

  - /PROD/datadg/
   |
   |-> /PROD/datadg/tablespace
   |   |-> /PROD/datadg/tablespace/tablespace1
   |   |-> /PROD/datadg/tablespace/tablespace2
   |
   |-> /PROD/datadg/data
   |   |-> /PROD/datadg/data/pg_tblspc
   |   |   | -> /PROD/datadg/data/pg_tblspc/16432 -> /PROD/datadg/tablespace/tablespace1
   |   |   | -> /PROD/datadg/data/pg_tblspc/16433 -> /PROD/datadg/tablespace/tablespace2

Almost a week later now, I see almost 2GB size difference in the tablespace folders on the 2 nodes. I also see some file count difference on both the nodes.

autovacuum is on on both the nodes and there isn't any idle in transaction queries on the slave node. Also there hasn't been any disruption on the streaming replication. I did not get any error like WAL segment already removed or so on the slave node. pg_stat_replication on the master node also doesn't show anything out of the odd and the sent_lsn, write_lsn and flush_lsn are regularly updated. I can not see difference in counts of most tables either, haven't verified for all of them. So my first question is:

  • Why is there a difference in the files in the tablespace folder? I can understand the difference in the modification timestamps, but some files are just missing on the slave node.

Now if I were to run vacuumdb on the master node, there are chances that the slave node will break and give an error like this

PANIC,XX000,"WAL contains references to invalid pages",,,,,"WAL redo at 875E/21A70BD0 for
 Heap2/VISIBLE: cutoff xid 60350476 flags 0x01; blkref #0: rel 16405/16419/533716933, fork 2, blk 26; blkref #1: rel 16405/16419/533716933, blk 853758",,,,"","startup

In the case when slave node breaks, these are the steps I usually do to bring the slave node back:

  1. Start pg_backup_start('backup') on the master node
  2. rsync the files from master to slave by running the following on the slave node:
rsync -av --delete master_node:/PROD/datadg/data/ /PROD/datadg/data --exclude 'pg_log' --exclude 'pg_replslot'

  1. Stop pg_backup_stop() on master node

  2. Start the slave node again and it usually works, even though the tablespace files might not still be the same.

Second question:

  • What is the best way to bring the slave node back? Is the rsync between tablespaces required? And if yes, what is the best method to do it for very large databases, something maybe as big as 30TB or more. I don't want to rsync all the files even if only the timestamp on them is different. So is a command like this safe to do? Or should an option like --checksum be used?
rsync -av --delete master_node:/PROD/datadg/tablespace/ /PROD/datadg/tablespace --size-only

Third question:

  • Is it advised to run vacuumdb before or after bringing the slave node back again?

Edit 1:

How I created the master-slave nodes (with no data except the one from initdb on master node)?

  1. Config file for master node
wal_level = logical
synchronous_commit = remote_write
archive_mode = on
archive_command = 'rsync -a %p /PROD/wal_archive/%f'
#archive_command = ':'
max_wal_senders=10
hot_standby = on
restart_after_crash = off
wal_sender_timeout = 5000
wal_receiver_status_interval = 2
max_standby_streaming_delay = -1
max_standby_archive_delay = -1
hot_standby_feedback = on
random_page_cost = 1.5
wal_keep_size = 0
  1. On master node, create replica user
CREATE ROLE replica WITH Replication Login Superuser;
  1. Whitelist replica user on master node and reload
  2. Stop the already running PG instance on the slave node
  3. Issue pg_backup_start('backup') on master node
  4. rsync to standby node using
rsync -av --delete master_node:/PROD/datadg/data/ /PROD/datadg/data --exclude 'pg_log' --exclude 'pg_replslot'
  1. Issue pg_backup_stop() on master node
  2. touch standby.signal on slave node
  3. Change the postgresql.conf on slave node and add
primary_conninfo = 'host=master_node's ip port=5432 user=replica application_name=host_name keepalives_idle=60 keepalives_interval=5 keepalives_count=5'
restore_command = 'rsync -a /PROD/wal_archive/%f %p'
recovery_target_timeline = 'latest'
  1. Start the slave node
  2. Check the replication status on master node.
Improve this question
4
  • Did you follow the instructions about the 'backup label file' and the 'tablespace map file' given under pg_backup_stop? You don't describe doing so, and I'm guessing that that is the problem.
    – jjanes
    Jul 25 at 16:06
  • I don't know why, but you are experiencing data corruption. Delete the standby and start from scratch. If you want an answer, please edit the question and describe how exactly you created the standby server.
    – Laurenz Albe
    Jul 25 at 20:41
  • @jjanes I looked at the pg_backup_stop, but didn't understand the meaning of "must be written to files in the backup area". I issued pg_backup_start('backup') and pg_backup_stop()` commands, but they didn't create any backup label file. I only saw the output on the psql client.
    – Abhishek B
    Jul 26 at 2:03
  • @LaurenzAlbe I edited to add the details about how I created the standby server. I understand from jjanes comment that I might be missing the backup label file and tablespace map, but I don't know where they are and what should be the rsync command for that. I used to do these steps as per my understanding of this
    – Abhishek B
    Jul 26 at 2:32

1 Answer 1

Reset to default
2

Your procedure is wrong, and the backup is missing the backup_label file that contains the checkpoint that you need to recover the standby from. Consequently, you recovered from the wrong checkpoint (the one in the control file). Unfortunately you did not encounter an error, so you ended up with a corrupted server.

The correct way to call pg_backup_stop() is

SELECT labelfile FROM pg_backup_stop();

You have to capture the output and create a file backup_label in the data directory of the standby with these contents. Note that pg_backup_start() and pg_backup_stop() must be called in the same database session.

For further reading, look at the documentation and this article.

Improve this answer
3
  • 1
    Thanks! Btw I guess you mean pg_backup_start() and pg_backup_stop() in the same session? So if I create the labelfile and put in the data directory for my slave node, and start my PG instance, then the data would not be corrupted? And I should remove that labelfile afterwards? Any advice on using rsync --size-only option or any other option to make it faster and not sync the whole 1TB?
    – Abhishek B
    Jul 26 at 5:33
  • Correct. But you never manually remove a file in the data directory. PostgreSQL will remove (actually rename) the file automatically when it is appropriate.
    – Laurenz Albe
    Jul 26 at 6:45
  • 2
    @AbhishekB You can't use --size-only. You also can't reliably use timestamps, either, you need the checksum feature.
    – jjanes
    Jul 26 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.