0

My postgres is deployed on k8s using the timescale helm chart. It's managed by patroni and the backup is managed by pgbackrest.

It's been stuck in this state for a day now (removed timestamps etc. from log to reduce clutter):

2022-12-02 19:00:02,932 DEBUG: Starting postgres: postgres -D /var/lib/postgresql/data --config-file=/var/lib/postgresql/data/postgresql.conf --listen_addresses=0.0.0.0 --port=5432 --cluster_name=my-cluster --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=150 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
LOG:  starting PostgreSQL 14.6 (Ubuntu 14.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, 64-bit
LOG:  listening on IPv4 address "0.0.0.0", port 5432
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG:  database system was interrupted while in recovery at log time 2022-12-02 17:33:49 UTC
HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
INFO: postmaster pid=2470
...
...
FATAL:  WAL was generated with wal_level=minimal, cannot continue recovering
DETAIL:  This happens if you temporarily set wal_level=minimal on the server.
HINT:  Use a backup taken after setting wal_level to higher than minimal.
LOG:  startup process (PID 2471) exited with exit code 1
LOG:  aborting startup due to startup process failure
LOG:  database system is shut down

Some things I've tried:

  1. Manual restore with patroni but it always fails due to Error: Provided config file /etc/patroni.yml not existing or no read rights. Check the -c/--config-file parameter Since the db is running through docker on k8s from a predefined chart, I have less control over installing custom stuff.
  2. Deleting/restarting the pod. This gives me the same result i.e. stuck in startup

How do I fix this error?

7
  • What does "manual restore with patroni" mean? Have you tried restoring with pgBackRest?
    – mustaccio
    Dec 2, 2022 at 20:04
  • Yes, I've tried to restore with pgbackrest but I am not sure what --stanza but I am unable to find the stanza name
    – ritratt
    Dec 2, 2022 at 20:25
  • Was wal_level ever set minimal? Was it always set to minimal? And what you are trying to do? Are you trying to set up a standby, or restore the master from a backup, or just have the master start back up again after a power failure? Or something else?
    – jjanes
    Dec 2, 2022 at 21:09
  • The wal is set to replica and I've confirmed this by looking at pgbackrest.conf. My cluster was running fine. I restarted my pods and since then it's been stuck in this state. So I guess the answer to your question is getting the master to startup after a power failure?
    – ritratt
    Dec 2, 2022 at 21:37
  • If the error is complaining about wal_level being set to minimal when it never has been, I would say you have some data corruption. The db system wrote something to disk, but then when it read it back from disk it said something else. I would say the error must be in the 'global/pg_control' file.
    – jjanes
    Dec 2, 2022 at 22:02

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.