PostgreSQL 9.6 won't start: "FATAL: could not write lock file "postmaster.pid": Input/output error"

Question

I am using PostgreSQL 9.6 running on Debian 9.2.

I cannot get my PostgreSQL up and running after the disk it was writing to became read only (it's on a separate disk from the OS). Here is the log from when it happened:

2019-01-05 14:03:17.592 CET [621] ERROR:  could not fsync file "pg_xlog/xlogtemp.621": Input/output error
2019-01-05 14:03:18.613 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:18.613 CET [621] DETAIL:  Could not write to file "pg_clog/0001" at offset 155648: Read-only file system.
2019-01-05 14:03:19.614 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:19.614 CET [621] DETAIL:  Could not open file "pg_clog/0001": Read-only file system.
2019-01-05 14:03:20.616 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:20.616 CET [621] DETAIL:  Could not open file "pg_clog/0001": Read-only file system.
2019-01-05 14:03:21.617 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:21.617 CET [621] DETAIL:  Could not open file "pg_clog/0001": Read-only file system.
2019-01-05 14:03:21.801 CET [12300] puppetdb@puppetdb PANIC:  could not write to log file 00000001000000EC00000023 at offset 14868480, length 106496: Read-only file system
2019-01-05 14:03:21.801 CET [12300] puppetdb@puppetdb STATEMENT:  COMMIT
2019-01-05 14:03:21.817 CET [590] LOG:  server process (PID 12300) was terminated by signal 6: Aborted
2019-01-05 14:03:21.817 CET [590] DETAIL:  Failed process was running: COMMIT
2019-01-05 14:03:21.817 CET [590] LOG:  terminating any other active server processes
2019-01-05 14:03:21.817 CET [16116] puppetdb@puppetdb WARNING:  terminating connection because of crash of another server process
2019-01-05 14:03:21.817 CET [16116] puppetdb@puppetdb DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2019-01-05 14:03:21.817 CET [16116] puppetdb@puppetdb HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-01-05 14:03:21.818 CET [16079] puppetdb@puppetdb WARNING:  terminating connection because of crash of another server process
2019-01-05 14:03:21.818 CET [16079] puppetdb@puppetdb DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2019-01-05 14:03:21.818 CET [16079] puppetdb@puppetdb HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-01-05 14:03:21.818 CET [15721] puppetdb@puppetdb WARNING:  terminating connection because of crash of another server process
2019-01-05 14:03:21.818 CET [15721] puppetdb@puppetdb DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
...
...
2019-01-05 14:03:21.913 CET [15852] puppetdb@puppetdb HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-01-05 14:03:21.945 CET [16148] puppetdb@puppetdb FATAL:  the database system is in recovery mode
2019-01-05 14:03:22.201 CET [16149] puppetdb@puppetdb FATAL:  the database system is in recovery mode
2019-01-05 14:03:22.580 CET [16151] puppetdb@puppetdb FATAL:  the database system is in recovery mode
2019-01-05 14:03:22.631 CET [590] LOG:  all server processes terminated; reinitializing
2019-01-05 14:03:22.655 CET [590] LOG:  could not open file "postmaster.pid": Read-only file system
2019-01-05 14:03:22.664 CET [590] PANIC:  could not open control file "global/pg_control": Read-only file system

I noticed today when I came in as our PuppetDB process was not returning results. I check the PostgreSQL service status and tried restarting. That's when I saw this in the new log file:

pg_ctl: another server might be running; trying to start server anyway
2019-01-07 14:33:50.084 CET [18561] FATAL:  could not remove old lock file "postmaster.pid": Read-only file system
2019-01-07 14:33:50.084 CET [18561] HINT:  The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
pg_ctl: another server might be running; trying to start server anyway
2019-01-07 14:39:56.606 CET [20276] FATAL:  could not remove old lock file "postmaster.pid": Read-only file system
2019-01-07 14:39:56.606 CET [20276] HINT:  The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.

I stopped the PostgreSQL service, unmounted the disk and ran fsck to repair and I can now write to disk again. However, I cannot start PostgreSQL. The only thing printing to the log file is this:

2019-01-07 14:44:11.567 CET [21038] FATAL:  could not write lock file "postmaster.pid": Input/output error

It will not print any more details even when I try to start it with debug level 5.

postgres@pve01vm107-stage:~$ /usr/lib/postgresql/9.6/bin/postgres "-d" "5" "-D" "/puppetdb/postgresql/9.6/main" "-c" "config_file=/etc/postgresql/9.6/main/postgresql.conf"
2019-01-07 15:11:38.474 CET [4275] DEBUG:  postgres: PostmasterMain: initial environment dump:
2019-01-07 15:11:38.474 CET [4275] DEBUG:  -----------------------------------------
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LANG=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   USER=postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PWD=/var/lib/postgresql
2019-01-07 15:11:38.474 CET [4275] DEBUG:   HOME=/var/lib/postgresql
2019-01-07 15:11:38.474 CET [4275] DEBUG:   MAIL=/var/mail/postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   SHELL=/bin/bash
2019-01-07 15:11:38.474 CET [4275] DEBUG:   TERM=xterm
2019-01-07 15:11:38.474 CET [4275] DEBUG:   SHLVL=1
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LANGUAGE=en_US:en
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LOGNAME=postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/puppetlabs/bin
2019-01-07 15:11:38.474 CET [4275] DEBUG:   _=/usr/lib/postgresql/9.6/bin/postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PGLOCALEDIR=/usr/share/locale
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PGSYSCONFDIR=/etc/postgresql-common
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_COLLATE=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_CTYPE=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_MESSAGES=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_MONETARY=C
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_NUMERIC=C
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_TIME=C
2019-01-07 15:11:38.474 CET [4275] DEBUG:  -----------------------------------------
2019-01-07 15:11:38.476 CET [4275] FATAL:  could not write lock file "postmaster.pid": Input/output error
2019-01-07 15:11:38.476 CET [4275] DEBUG:  shmem_exit(1): 0 before_shmem_exit callbacks to make
2019-01-07 15:11:38.476 CET [4275] DEBUG:  shmem_exit(1): 0 on_shmem_exit callbacks to make
2019-01-07 15:11:38.476 CET [4275] DEBUG:  proc_exit(1): 0 callbacks to make
2019-01-07 15:11:38.476 CET [4275] DEBUG:  exit(1)

Since this is (thankfully) only a staging system and our setup is still in the early testing stages, I could reboot the server but it still did not help.

Does anyone have any idea how I can fix this? I can't find this exact error in all of my Googling. Thanks.