Postgres stuck in wal recovery crashloop

Question

My postgres is deployed on k8s using the timescale helm chart. It's managed by patroni and the backup is managed by pgbackrest.

It's been stuck in this state for a day now (removed timestamps etc. from log to reduce clutter):

2022-12-02 19:00:02,932 DEBUG: Starting postgres: postgres -D /var/lib/postgresql/data --config-file=/var/lib/postgresql/data/postgresql.conf --listen_addresses=0.0.0.0 --port=5432 --cluster_name=my-cluster --wal_level=replica --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=150 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
LOG:  starting PostgreSQL 14.6 (Ubuntu 14.6-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, 64-bit
LOG:  listening on IPv4 address "0.0.0.0", port 5432
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG:  database system was interrupted while in recovery at log time 2022-12-02 17:33:49 UTC
HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
INFO: postmaster pid=2470
...
...
FATAL:  WAL was generated with wal_level=minimal, cannot continue recovering
DETAIL:  This happens if you temporarily set wal_level=minimal on the server.
HINT:  Use a backup taken after setting wal_level to higher than minimal.
LOG:  startup process (PID 2471) exited with exit code 1
LOG:  aborting startup due to startup process failure
LOG:  database system is shut down

Some things I've tried:

1. Manual restore with patroni but it always fails due to Error: Provided config file /etc/patroni.yml not existing or no read rights. Check the -c/--config-file parameter Since the db is running through docker on k8s from a predefined chart, I have less control over installing custom stuff.
2. Deleting/restarting the pod. This gives me the same result i.e. stuck in startup

How do I fix this error?