1

We are having 9.4.4 version running on linux( PostgreSQL 9.4.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit) which is crashing unexpectedly and coming up on its own.
What can be wrong, at time it fails while executing a query (update) at times at delete query and at times and checkpoint. Looks like OOM Killer is getting triggered by I am not sure why.

Second is that I am having some partition tables(almost 2000 + Partitioned Tables) . Can that be a problem if my query is trying to update or delete from the partitioned table.

Following is one of the sample logs.

2015-08-28 09:58:14.534 UTC >LOG:  checkpointer process (PID 5901) was terminated by signal 9: Killed
2015-08-28 09:58:14.582 UTC >LOG:  terminating any other active server processes
2015-08-28 09:58:14.608 UTC >WARNING:  terminating connection because of crash of another server process
2015-08-28 09:58:14.608 UTC >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-28 09:58:14.608 UTC >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-08-28 09:58:14.608 UTC >WARNING:  terminating connection because of crash of another server process
2015-08-28 09:58:14.608 UTC >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-28 09:58:14.608 UTC >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-08-28 09:58:14.608 UTC >WARNING:  terminating connection because of crash of another server process
2015-08-28 09:58:14.608 UTC >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-28 09:58:14.608 UTC >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-08-28 09:58:14.608 UTC >WARNING:  terminating connection because of crash of another server process
2015-08-28 09:58:14.608 UTC >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-28 09:58:14.608 UTC >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-08-28 09:58:14.608 UTC >WARNING:  terminating connection because of crash of another server process
2015-08-28 09:58:14.608 UTC >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-28 09:58:14.608 UTC >HINT:  In a moment you should be able to reconnect to the database and repeat your command.

2015-08-28 09:58:14.639 UTC >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-08-28 09:58:14.639 UTC >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2015-08-28 09:58:15.390 UTC >LOG:  all server processes terminated; reinitializing
2015-08-28 09:58:19.334 UTC >LOG:  database system was interrupted; last known up at 2015-08-28 09:52:39 UTC
2015-08-28 09:58:19.883 UTC >LOG:  database system was not properly shut down; automatic recovery in progress
2015-08-28 09:58:19.893 UTC >LOG:  redo starts at 6D/AE079F48
2015-08-28 09:58:32.298 UTC >LOG:  unexpected pageaddr 6D/88426000 in log segment 000000010000006D000000E7, offset 4349952
2015-08-28 09:58:32.298 UTC >LOG:  redo done at 6D/E7425DB0
2015-08-28 09:58:32.299 UTC >LOG:  last completed transaction was at log time 2015-08-28 09:55:04.414434+00
2015-08-28 09:58:38.717 UTC >LOG:  MultiXact member wraparound protections are now enabled
2015-08-28 09:58:38.746 UTC >LOG:  autovacuum launcher started
2015-08-28 09:58:38.746 UTC >LOG:  database system is ready to accept connections
  • 3
    was terminated by signal 9: so it does not crash, it's killed from outside. OOM is a plausible candidate indeed. You need to figure out what processes eat too much RAM; start with the kernel logs at the moment of the kill. – Daniel Vérité Aug 28 '15 at 12:45
  • 1
    What is the output of dmesg (which should have the OOMKiller output in it)? How much RAM is available on the machine? What items have you changed in your postgresql.conf? What items have you changed in your sysctl.conf? Do you have swap enabled? These would definitely be helpful in diagnosing the reason for the OOMKiller firing. – Kassandry Aug 29 '15 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.