Crunchydata - PGO - Postgres(v 14.x) restore not working as expected

Question

We have set up PostgreSQL, including its backup and recovery processes based on crunchy data Postgres Operator, as per the documentation.

The backups are occurring as expected. However, when we simulate scenarios of accidental data deletion or similar issues, we are unable to restore the data correctly. Although the restore command or action completes successfully, the recovery does not yield the desired results.

For instance to mimick accidental data deletion and recovery, we are following these steps to test the recovery:

1. Insert the table or data.
2. Check if archive mode is on: - SELECT name, setting, unit FROM pg_settings WHERE name IN ('archive_mode', 'archive_command', 'archive_timeout');
3. Force data to be archived: SELECT pg_switch_wal();
4. On the repository host pod, perform a full backup: pgbackrest backup --stanza=db --log-level-console=info --type=full
5. Check the backup: pgbackrest info
6. Note the current timestamp: SELECT now(); -- 2024-07-01 07:14:45.118038+00
7. Delete data from the table: DELETE FROM abc WHERE partition_id = 'someId1';
8. Remove the data directory: rm -rf pgdata/pg15/*
9. Restore the data: pgbackrest --delta --log-level-console=detail --stanza=db --type=time --target="2024-07-01 12:26:26.118038+00" restore or pgbackrest --stanza=db restore

After doing expectation was that data deleted row should have recovered as the data was present when the original backup was done.

Seems like some data from somewhere is also getting recoverd which was not actually backed up and is not expected to be recovered.

Can you please help us understand what might be going wrong? Your help and time are greatly appreciated, as this issue is a priority and has become a blocker for our team.

Postgressql.conf

    archive_command = 'pgbackrest --stanza=db archive-push "%p"'
    archive_mode = 'on'
    archive_timeout = '60s'
    cluster_name = 'postgres-ha'
    hot_standby = 'on'
    jit = 'off'
    listen_addresses = '*'
    max_connections = '100'
    max_locks_per_transaction = '64'
    max_prepared_transactions = '0'
    max_replication_slots = '10'
    max_wal_senders = '10'
    max_worker_processes = '8'
    password_encryption = 'scram-sha-256'
    pgnodemx.kdapi_path = '/etc/database-containerinfo'
    port = '5432'
    shared_preload_libraries = 'pg_stat_statements,pgnodemx,pgaudit'
    ssl = 'on'
    ssl_ca_file = '/pgconf/tls/ca.crt'
    ssl_cert_file = '/pgconf/tls/tls.crt'
    ssl_key_file = '/pgconf/tls/tls.key'
    track_commit_timestamp = 'off'
    unix_socket_directories = '/tmp/postgres'
    wal_keep_size = '128MB'
    wal_level = 'logical'
    wal_log_hints = 'on'
    hba_file = '/pgdata/pg15/pg_hba.conf'
    ident_file = '/pgdata/pg15/pg_ident.conf'

On a side note, not sure if this is related, but when i run the verify command I see some error

pgbackrest verify --stanza=db
ERROR: [039]: client state is 'data-get' but expected 'idle'