0

I am using PostgreSQL 9.6 running on Debian 9.2.

I cannot get my PostgreSQL up and running after the disk it was writing to became read only (it's on a separate disk from the OS). Here is the log from when it happened:

2019-01-05 14:03:17.592 CET [621] ERROR:  could not fsync file "pg_xlog/xlogtemp.621": Input/output error
2019-01-05 14:03:18.613 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:18.613 CET [621] DETAIL:  Could not write to file "pg_clog/0001" at offset 155648: Read-only file system.
2019-01-05 14:03:19.614 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:19.614 CET [621] DETAIL:  Could not open file "pg_clog/0001": Read-only file system.
2019-01-05 14:03:20.616 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:20.616 CET [621] DETAIL:  Could not open file "pg_clog/0001": Read-only file system.
2019-01-05 14:03:21.617 CET [621] ERROR:  could not access status of transaction 0
2019-01-05 14:03:21.617 CET [621] DETAIL:  Could not open file "pg_clog/0001": Read-only file system.
2019-01-05 14:03:21.801 CET [12300] puppetdb@puppetdb PANIC:  could not write to log file 00000001000000EC00000023 at offset 14868480, length 106496: Read-only file system
2019-01-05 14:03:21.801 CET [12300] puppetdb@puppetdb STATEMENT:  COMMIT
2019-01-05 14:03:21.817 CET [590] LOG:  server process (PID 12300) was terminated by signal 6: Aborted
2019-01-05 14:03:21.817 CET [590] DETAIL:  Failed process was running: COMMIT
2019-01-05 14:03:21.817 CET [590] LOG:  terminating any other active server processes
2019-01-05 14:03:21.817 CET [16116] puppetdb@puppetdb WARNING:  terminating connection because of crash of another server process
2019-01-05 14:03:21.817 CET [16116] puppetdb@puppetdb DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2019-01-05 14:03:21.817 CET [16116] puppetdb@puppetdb HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-01-05 14:03:21.818 CET [16079] puppetdb@puppetdb WARNING:  terminating connection because of crash of another server process
2019-01-05 14:03:21.818 CET [16079] puppetdb@puppetdb DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2019-01-05 14:03:21.818 CET [16079] puppetdb@puppetdb HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-01-05 14:03:21.818 CET [15721] puppetdb@puppetdb WARNING:  terminating connection because of crash of another server process
2019-01-05 14:03:21.818 CET [15721] puppetdb@puppetdb DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
...
...
2019-01-05 14:03:21.913 CET [15852] puppetdb@puppetdb HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-01-05 14:03:21.945 CET [16148] puppetdb@puppetdb FATAL:  the database system is in recovery mode
2019-01-05 14:03:22.201 CET [16149] puppetdb@puppetdb FATAL:  the database system is in recovery mode
2019-01-05 14:03:22.580 CET [16151] puppetdb@puppetdb FATAL:  the database system is in recovery mode
2019-01-05 14:03:22.631 CET [590] LOG:  all server processes terminated; reinitializing
2019-01-05 14:03:22.655 CET [590] LOG:  could not open file "postmaster.pid": Read-only file system
2019-01-05 14:03:22.664 CET [590] PANIC:  could not open control file "global/pg_control": Read-only file system

I noticed today when I came in as our PuppetDB process was not returning results. I check the PostgreSQL service status and tried restarting. That's when I saw this in the new log file:

pg_ctl: another server might be running; trying to start server anyway
2019-01-07 14:33:50.084 CET [18561] FATAL:  could not remove old lock file "postmaster.pid": Read-only file system
2019-01-07 14:33:50.084 CET [18561] HINT:  The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.
pg_ctl: another server might be running; trying to start server anyway
2019-01-07 14:39:56.606 CET [20276] FATAL:  could not remove old lock file "postmaster.pid": Read-only file system
2019-01-07 14:39:56.606 CET [20276] HINT:  The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again.

I stopped the PostgreSQL service, unmounted the disk and ran fsck to repair and I can now write to disk again. However, I cannot start PostgreSQL. The only thing printing to the log file is this:

2019-01-07 14:44:11.567 CET [21038] FATAL:  could not write lock file "postmaster.pid": Input/output error

It will not print any more details even when I try to start it with debug level 5.

postgres@pve01vm107-stage:~$ /usr/lib/postgresql/9.6/bin/postgres "-d" "5" "-D" "/puppetdb/postgresql/9.6/main" "-c" "config_file=/etc/postgresql/9.6/main/postgresql.conf"
2019-01-07 15:11:38.474 CET [4275] DEBUG:  postgres: PostmasterMain: initial environment dump:
2019-01-07 15:11:38.474 CET [4275] DEBUG:  -----------------------------------------
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LANG=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   USER=postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PWD=/var/lib/postgresql
2019-01-07 15:11:38.474 CET [4275] DEBUG:   HOME=/var/lib/postgresql
2019-01-07 15:11:38.474 CET [4275] DEBUG:   MAIL=/var/mail/postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   SHELL=/bin/bash
2019-01-07 15:11:38.474 CET [4275] DEBUG:   TERM=xterm
2019-01-07 15:11:38.474 CET [4275] DEBUG:   SHLVL=1
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LANGUAGE=en_US:en
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LOGNAME=postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/puppetlabs/bin
2019-01-07 15:11:38.474 CET [4275] DEBUG:   _=/usr/lib/postgresql/9.6/bin/postgres
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PGLOCALEDIR=/usr/share/locale
2019-01-07 15:11:38.474 CET [4275] DEBUG:   PGSYSCONFDIR=/etc/postgresql-common
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_COLLATE=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_CTYPE=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_MESSAGES=en_US.UTF-8
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_MONETARY=C
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_NUMERIC=C
2019-01-07 15:11:38.474 CET [4275] DEBUG:   LC_TIME=C
2019-01-07 15:11:38.474 CET [4275] DEBUG:  -----------------------------------------
2019-01-07 15:11:38.476 CET [4275] FATAL:  could not write lock file "postmaster.pid": Input/output error
2019-01-07 15:11:38.476 CET [4275] DEBUG:  shmem_exit(1): 0 before_shmem_exit callbacks to make
2019-01-07 15:11:38.476 CET [4275] DEBUG:  shmem_exit(1): 0 on_shmem_exit callbacks to make
2019-01-07 15:11:38.476 CET [4275] DEBUG:  proc_exit(1): 0 callbacks to make
2019-01-07 15:11:38.476 CET [4275] DEBUG:  exit(1)

Since this is (thankfully) only a staging system and our setup is still in the early testing stages, I could reboot the server but it still did not help.

Does anyone have any idea how I can fix this? I can't find this exact error in all of my Googling. Thanks.

  • I'd start by checking permissions on postmaster.pid and its parent directories. Also try removing the file manually if it's there. – mustaccio Jan 7 at 17:57
  • @mustaccio there is no postmaster.pid file in the data directory and I doublechecked that everything in the data directory, including the parent directory, is owned by postgres. Still no luck. – David Jacob Jan 8 at 8:57
  • What happens if you log in to the Linux account named "postgres", and try to manually create a file in the data directory? – jjanes Jan 8 at 18:07
  • Check the output of dmesg for hardware/driver errors. The "Input/output error" part of the message comes from the system, not postgres. This happens when your disk is dying. – Daniel Vérité Jan 9 at 13:20

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.