Postgres 10 replica service error on startup

Question

I have migrated a database promoting the read-replica in a server with more capacity, this process worked smoothly. But I'm trying to create a read-replica now from this promoted server to have a similar setup than I had before. The problem is when I try to start Postgres server this happens.

● postgresql-10.service - PostgreSQL 10 database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql-10.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since mar 2018-10-16 07:07:42 UTC; 10s ago
   Docs: https://www.postgresql.org/docs/10/static/
   Process: 22746 ExecStart=/usr/pgsql-10/bin/postmaster -D ${PGDATA} 
   (code=exited, status=2)
   Process: 22740 ExecStartPre=/usr/pgsql-10/bin/postgresql-10-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
   Main PID: 22746 (code=exited, status=2)

I'm going to describe my process as accurate as I can, I'm running on CentOS RHEL 7.

First I install Postgres 10:

rpm -Uvh https://yum.postgresql.org/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
yum install postgresql10-server postgresql10
yum install postgresql10-contrib
su - postgres
/usr/pgsql-10/bin/initdb
systemctl start postgresql-10.service

This works, the server is running so the installation worked, let's start with the replication. I stop the service and continue with this:

 rm -r /var/lib/pgsql/10/data
 su - postgres
 /usr/pgsql-10/bin/pg_basebackup -h <database host ip> -D /var/lib/pgsql/10/data/ -P -U replicate

replicate is the user with replication privileges in the main database.

The pg_basebackup finishes correctly, then I rename the recovery.done to recovery.conf.

drwx------. 6 postgres postgres   54 oct 16 06:45 base
-rw-------. 1 postgres postgres   30 oct 16 06:55 current_logfiles
drwx------. 2 postgres postgres 4,0K oct 16 06:45 global
drwx------. 2 postgres postgres  188 oct 16 06:53 log
drwx------. 2 postgres postgres    6 oct 16 06:45 pg_commit_ts
drwx------. 2 postgres postgres    6 oct 16 06:45 pg_dynshmem
-rw-------. 1 postgres postgres 4,5K oct 16 06:55 pg_hba.conf
-rw-------. 1 postgres postgres 1,6K oct 16 06:55 pg_ident.conf
drwx------. 4 postgres postgres   68 oct 16 06:53 pg_logical
drwx------. 4 postgres postgres   36 oct 16 06:45 pg_multixact
drwx------. 2 postgres postgres   18 oct 16 06:45 pg_notify
drwx------. 2 postgres postgres    6 oct 16 06:53 pg_replslot
drwx------. 2 postgres postgres    6 oct 16 06:45 pg_serial
drwx------. 2 postgres postgres    6 oct 16 06:45 pg_snapshots
drwx------. 2 postgres postgres    6 oct 16 06:53 pg_stat
drwx------. 2 postgres postgres    6 oct 16 06:53 pg_stat_tmp
drwx------. 2 postgres postgres    6 oct 16 06:45 pg_subtrans
drwx------. 2 postgres postgres    6 oct 16 06:53 pg_tblspc
drwx------. 2 postgres postgres    6 oct 16 06:45 pg_twophase
-rw-------. 1 postgres postgres    3 oct 16 06:55 PG_VERSION
drwx------. 3 postgres postgres 4,0K oct 16 06:45 pg_wal
drwx------. 2 postgres postgres 4,0K oct 16 06:53 pg_xact
-rw-------. 1 postgres postgres   88 oct 16 06:55 postgresql.auto.conf
-rw-------. 1 postgres postgres  23K oct 16 06:55 postgresql.conf
-rw-------. 1 postgres postgres   58 oct 16 06:55 postmaster.opts
-rw-r--r--. 1 postgres postgres  154 oct 16 06:53 recovery.conf

But when running systemctl start postgresql-10.service, I get

Job for postgresql-10.service failed because the control process exited with error code. See "systemctl status postgresql-10.service" and "journalctl -xe" for details.

I have not been able to find more clues than this single message:

postgresql-10.service: main process exited, code=exited, status=1/FAILURE

What am I missing? I have tried to run with default postgres.conf and pg_hba.conf to avoid misconfiguration errors but that didn't work either.

Full journalctl output:

oct 17 09:15:11 database-replica systemd[1]: Starting PostgreSQL 10 database server...
-- Subject: Unit postgresql-10.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-10.service has begun starting up.
oct 17 09:15:11 database-replica postmaster[19514]: 2018-10-17 09:15:11.834 UTC [19514] LOG:  listening on IPv6 address "::1", port 5432
oct 17 09:15:11 database-replica postmaster[19514]: 2018-10-17 09:15:11.834 UTC [19514] LOG:  listening on IPv4 address "127.0.0.1", port 5432
oct 17 09:15:11 database-replica postmaster[19514]: 2018-10-17 09:15:11.836 UTC [19514] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
oct 17 09:15:11 database-replica postmaster[19514]: 2018-10-17 09:15:11.840 UTC [19514] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
oct 17 09:15:11 database-replica postmaster[19514]: 2018-10-17 09:15:11.865 UTC [19514] LOG:  redirecting log output to logging collector process
oct 17 09:15:11 database-replica postmaster[19514]: 2018-10-17 09:15:11.865 UTC [19514] HINT:  Future log output will appear in directory "log".
oct 17 09:15:11 database-replica systemd[1]: postgresql-10.service: main process exited, code=exited, status=1/FAILURE
oct 17 09:15:11 database-replica systemd[1]: Failed to start PostgreSQL 10 database server.
-- Subject: Unit postgresql-10.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-10.service has failed.
--
-- The result is failed.
oct 17 09:15:11 database-replica systemd[1]: Unit postgresql-10.service entered failed state.
oct 17 09:15:11 database-replica systemd[1]: postgresql-10.service failed.
oct 17 09:15:11 database-replica polkitd[676]: Unregistered Authentication Agent for unix-process:19502:16930756 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale es_ES.UTF-8) (disconnected from bus)

Answer 1

Finally, I noticed where the problem was. The pg_basebackup copied a recovery.conf file that was originally from the source database, and this caused the break. Removing recovery.conf and creating it from scratch solved the issue.

The server is now running.