Postgres restore from WAL-E backup never finishes

Question

Doing a restore on a WAL-E backup to S3, postgres never seems to complete the restore process. The log repeats the following:

< 2016-04-08 13:41:55.801 PDT >LOG:  connection received: host=[local]
< 2016-04-08 13:41:55.801 PDT >FATAL:  the database system is starting up
< 2016-04-08 13:41:56.809 PDT >LOG:  connection received: host=[local]
< 2016-04-08 13:41:56.809 PDT >FATAL:  the database system is starting up
< 2016-04-08 13:41:57.812 PDT >LOG:  connection received: host=[local]
< 2016-04-08 13:41:57.812 PDT >FATAL:  the database system is starting up
< 2016-04-08 13:41:58.815 PDT >LOG:  connection received: host=[local]
< 2016-04-08 13:41:58.815 PDT >FATAL:  the database system is starting up
< 2016-04-08 13:41:59.817 PDT >LOG:  connection received: host=[local]
< 2016-04-08 13:41:59.818 PDT >FATAL:  the database system is starting up
< 2016-04-08 13:42:00.820 PDT >LOG:  connection received: host=[local]
< 2016-04-08 13:42:00.820 PDT >FATAL:  the database system is starting up
lzop: <stdin>: not a lzop file
wal_e.blobstore.s3.s3_util WARNING  MSG: could no longer locate object while performing wal restore
        DETAIL: The absolute URI that could not be located is s3://groupsiopostgres/main/wal_005/0000000100000A8000000071.lzo.
        HINT: This can be normal when Postgres is trying to detect what timelines are available during restoration.
        STRUCTURED: time=2016-04-08T20:42:01.071643-00 pid=28784

The URI does increase approximately every minute. I understand these aren't errors. But psql continues to in this loop until systemctl kills it for taking too long.

Here's my recovery.conf file:

restore_command = '/usr/local/bin/envdir /etc/wal-e.d/env /usr/bin/wal-e --terse wal-fetch %f %p'
standby_mode = 'on'
trigger_file = '/var/lib/pgsql/9.3/data/failover'

How can I get postgres to complete before systemctl kills it?

Answer 1

standby_mode = on

Tharrr's yer problem. You've configured it as a continuous (warm or hot) standby.

standby_mode in the manual:

standby_mode (boolean)

Specifies whether to start the PostgreSQL server as a standby. If this parameter is on, the server will not stop recovery when the end of archived WAL is reached, but will keep trying to continue recovery by fetching new WAL segments using restore_command and/or by connecting to the primary server as specified by the primary_conninfo setting.