I have a PostgreSQL 10.12 server running on an Ubuntu 18.04 Server. Since today, I am suddenly getting errors when querying data from a Python script running on the same machine, and in some edge cases also in pgAdmin 4.
In about 50% of the cases, my Python script will fail to return any data because the DB server closed the connection. I am using SQLAlchemy and it is giving me the following error:
psycopg2.errors.AdminShutdown: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Interestingly, the same queries (which are simple "SELECT * FROM table" statements) run perfectly fine in pgAdmin. However, when I try to run the following query in pgAdmin, I get the message Connection to the server has been lost.
select * from mytable where mycolumn <> ''
This query specifically worked fine a few days ago. I couldn't find any other examples where this happens in pgAdmin, but there is surely something wrong.
What is weird is that the database has been running for over a month without any issues, and this only started today. I did try to restart the service to no avail.
The logs are not making much sense to me, but there are several things wrong.
The last few days, it constantly gave me the following message:
2020-04-06 19:43:26.959 CEST [7820] postgres@postgres FATAL: password authentication failed for user "postgres"
2020-04-06 19:43:26.959 CEST [7820] postgres@postgres DETAIL: Password does not match for user "postgres".
Connection matched pg_hba.conf line 102: "host all all all md5"
I am certain that I am using the correct password, as it does work 50% of the time of the queries. It is showing this message for the default postgres database and not my own, but it still confuses me why this happens as I'm using the default postgres user for both databases. I got this message a lot during the night where I was not doing anything with it either.
For the actual error regarding the query from pgAdmin, this is what I get:
2020-04-07 11:25:18.891 CEST [5497] LOG: server process (PID 20569) was terminated by signal 11: Segmentation fault
2020-04-07 11:25:18.891 CEST [5497] DETAIL: Failed process was running: SELECT * FROM transactionoutputs WHERE fileheader <> ''
2020-04-07 11:25:18.891 CEST [5497] LOG: terminating any other active server processes
2020-04-07 11:25:18.891 CEST [20567] postgres@bitcoin WARNING: terminating connection because of crash of another server process
2020-04-07 11:25:18.891 CEST [20567] postgres@bitcoin DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-04-07 11:25:18.891 CEST [20567] postgres@bitcoin HINT: In a moment you should be able to reconnect to the database and repeat your command.
I can't seem to find any errors in the logs for the regular SELECT queries from my Python scripts, so I don't know why the connection is closed for this all the time. What is so confusing about this is that it worked perfectly fine just a yesterday, and I have not changed anything. Restarting PostgreSQL also did not help.
I'm not an expert with databases or Linux, so any help why this might happen and how I can figure out what exactly is happening is highly appreciated, thank you.
