Unexpected PostgreSQL restart

Question

My PostgreSQL server unexpectedly restarted with such messages:

2017-08-16 03:44:34 GMT LOG:  received fast shutdown request
2017-08-16 03:44:34 GMT LOG:  aborting any active transactions
2017-08-16 03:44:34 GMT FATAL:  terminating connection due to administrator command
2017-08-16 03:44:34 GMT FATAL:  terminating connection due to administrator command
2017-08-16 03:44:34 GMT LOG:  autovacuum launcher shutting down
2017-08-16 03:44:34 GMT LOG:  shutting down
2017-08-16 03:44:34 GMT LOG:  database system is shut down
2017-08-16 03:46:04 GMT LOG:  incomplete startup packet
2017-08-16 03:46:04 GMT LOG:  database system was shut down at 2017-08-16 03:44:34 GMT
2017-08-16 03:46:04 GMT LOG:  MultiXact member wraparound protections are now enabled
2017-08-16 03:46:04 GMT LOG:  database system is ready to accept connections
2017-08-16 03:46:04 GMT LOG:  autovacuum launcher started

what could be the reason?

Configuration details:

OS: Ubuntu 16.04.1 LTS (Windows Azure VM)
PostgreSQL version: 9.5.8
listen address: * (my bad, but Azure have to block another ports from external access.
                   I didn't enable PG 5432 port for external access) 
HBA config:
local   all             postgres                                ident
host    all             all             127.0.0.1/32            md5
host    all             all             ::1/128                 md5
host    all             all             10.0.0.0/24             md5
host    replication     app_replicator 10.0.0.5/32              md5
local   all             all                                     peer

Answer 1

2017-08-16 03:44:34 GMT LOG: received fast shutdown request

Someone did a pg_ctl -m fast stop then pg_ctl start, either manually or via the Windows service controller (services.msc, net service, ...).

If it wasn't you, it was a scheduled job, automatic update for some tool related to PostgreSQL from some unofficial 3rd party, etc. Because unless your system got hit by the most mystically lucky cosmic ray in history, that's what happened.

There's no way to restart PostgreSQL from within the PostgreSQL SQL environment (a SQL connection via PgAdmin, psql, etc). You could do it by sending a signal from an untrusted procedural language like plpythonu or plperlu if you're connected as the database superuser and these languages are installed. But otherwise it'd be done at the command-shell level with pg_ctl, by signalling the postmaster process, etc.

Answer 2

In my case, this happened automatically due to the system was out of memory.

Found logs about oom_reaper and dotnet oom tasks in Syslog, but not from Postgres. I only got the message "received fast shutdown request" in the Postgres csvlog.

PS: Since my Postgres is running inside of Docker i can't mutually exclude that Docker sent a signal and forced this shutdown (but I don't assume this)

Answer 3

When this happened to me it was automatic updates to PostgreSQL. I validated this by looking at the /var/log/dpkg.log file and seeing that PostgreSQL was updated at the time of the shutdown.