My postgres is deployed on k8s using the timescale helm chart. It's managed by
patroni and the backup is managed by
It's been stuck in this state for a day now (removed timestamps etc. from log to reduce clutter):
2022-12-02 19:00:02,932 DEBUG: Starting postgres: postgres -D /var/lib/postgresql/data --config-file=/var/lib/postgresql/data/postgresql.conf --listen_addresses=0.0.0.0 --port=5432 --cluster_name=my-cluster --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=150 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on LOG: starting PostgreSQL 14.6 (Ubuntu 14.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, 64-bit LOG: listening on IPv4 address "0.0.0.0", port 5432 LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" LOG: database system was interrupted while in recovery at log time 2022-12-02 17:33:49 UTC HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target. INFO: postmaster pid=2470 ... ... FATAL: WAL was generated with wal_level=minimal, cannot continue recovering DETAIL: This happens if you temporarily set wal_level=minimal on the server. HINT: Use a backup taken after setting wal_level to higher than minimal. LOG: startup process (PID 2471) exited with exit code 1 LOG: aborting startup due to startup process failure LOG: database system is shut down
Some things I've tried:
- Manual restore with
patronibut it always fails due to
Error: Provided config file /etc/patroni.yml not existing or no read rights. Check the -c/--config-file parameterSince the db is running through docker on k8s from a predefined chart, I have less control over installing custom stuff.
- Deleting/restarting the pod. This gives me the same result i.e. stuck in startup
How do I fix this error?