2020-12-20 21:06:04 UYT [25922]: [4-1] user=,db=,app=,client= LOG: server process (PID 19303) was terminated by signal 11: Segmentation fault 2020-12-20 21:06:04 UYT [25922]: [5-1] user=,db=,app=,client= DETAIL: Failed process was running: COPY public.act_hi_varinst (id_, proc_def_key_, proc_def_id_, proc_inst_id_, execution_id_, act_inst_id_, case_def_key_, case_def_id_, case_inst_id_, case_execution_id_, task_id_, name_, var_type_, rev_, bytearray_id_, double_, long_, text_, text2_, tenant_id_, state_, create_time_, root_proc_inst_id_, removal_time_) TO stdout; 2020-12-20 21:06:04 UYT [25922]: [6-1] user=,db=,app=,client= LOG: terminating any other active server processes 2020-12-20 21:06:04 UYT [1910]: [3-1] user=dbusr,db=dbdatabase ,app=[unknown],client= WARNING: terminating connection because of crash of another server process 2020-12-20 21:06:04 UYT [1910]: [4-1] user=dbusr,db=dbdatabase ,app=[unknown],client= DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-12-20 21:06:04 UYT [1910]: [5-1] user=dbusr,db=dbdatabase ,app=[unknown],client= HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-12-20 21:06:04 UYT [27108]: [3-1] user=dbusr,db=dbdatabase ,app=[unknown],client= WARNING: terminating connection because of crash of another server process 2020-12-20 21:06:04 UYT [27108]: [4-1] user=dbusr,db=dbdatabase ,app=[unknown],client= DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

Error found at /var/log/messages

Dec 20 21:06:04 server-name kernel: [20278231.774013] postgres[19303]: segfault at 7f7740354704 ip 00007f772c8ac700 sp 00007ffdac7ff4e0 error 4 in postgres[7f772c813000+5a1000]

2020-12-20 21:06:04 UYT [25922]: [4-1] user=,db=,app=,client= LOG:  server process (PID 19303) was terminated by signal 11: Segmentation fault
2020-12-20 21:06:04 UYT [25922]: [5-1] user=,db=,app=,client= DETAIL:  Failed process was running: COPY public.act_hi_varinst (id_, proc_def_key_, proc_def_id_, proc_inst_id_, execution_id_, act_inst_id_, case_def_key_, case_def_id_, case_inst_id_, case_execution_id_, task_id_, name_, var_type_, rev_, bytearray_id_, double_, long_, text_, text2_, tenant_id_, state_, create_time_, root_proc_inst_id_, removal_time_) TO stdout;
2020-12-20 21:06:04 UYT [25922]: [6-1] user=,db=,app=,client= LOG:  terminating any other active server processes
2020-12-20 21:06:04 UYT [1910]: [3-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= WARNING:  terminating connection because of crash of another server process
2020-12-20 21:06:04 UYT [1910]: [4-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-12-20 21:06:04 UYT [1910]: [5-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2020-12-20 21:06:04 UYT [27108]: [3-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= WARNING:  terminating connection because of crash of another server process
2020-12-20 21:06:04 UYT [27108]: [4-1] user=dbusr,db=dbdatabase     ,app=[unknown],client= DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

Error found at /var/log/messages

Dec 20 21:06:04 server-name kernel: [20278231.774013] postgres[19303]: segfault at 7f7740354704 ip 00007f772c8ac700 sp 00007ffdac7ff4e0 error 4 in postgres[7f772c813000+5a1000]

We have a master/slave postgresql cluster with streaming replication and pgpool.

Versions in both postgresql servers are.

postgres: 9.4
OS: Debian GNU/Linux 8 (jessie)

PGPOOL server: version pgpool: pgpool-II version 3.3.3 os: Red Hat Enterprise Linux Server release 6.1

Problem:

We suffer a crash down of the postgresql service at master node with subsequent failover done by pgpool

Could not find the reason of this behavior in master node. We want to know if someone coult tell us, based on the recorded log messages, what could have caused this crash. Following, some relevant lines from the postgresql and linux log.

postgresql.log

2020-12-20 21:06:04 UYT [25922]: [4-1] user=,db=,app=,client= LOG: server process (PID 19303) was terminated by signal 11: Segmentation fault 2020-12-20 21:06:04 UYT [25922]: [5-1] user=,db=,app=,client= DETAIL: Failed process was running: COPY public.act_hi_varinst (id_, proc_def_key_, proc_def_id_, proc_inst_id_, execution_id_, act_inst_id_, case_def_key_, case_def_id_, case_inst_id_, case_execution_id_, task_id_, name_, var_type_, rev_, bytearray_id_, double_, long_, text_, text2_, tenant_id_, state_, create_time_, root_proc_inst_id_, removal_time_) TO stdout; 2020-12-20 21:06:04 UYT [25922]: [6-1] user=,db=,app=,client= LOG: terminating any other active server processes 2020-12-20 21:06:04 UYT [1910]: [3-1] user=dbusr,db=dbdatabase ,app=[unknown],client=10.97.67.70 WARNING: terminating connection because of crash of another server process 2020-12-20 21:06:04 UYT [1910]: [4-1] user=dbusr,db=dbdatabase ,app=[unknown],client=10.97.67.70 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-12-20 21:06:04 UYT [1910]: [5-1] user=dbusr,db=dbdatabase ,app=[unknown],client=10.97.67.70 HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-12-20 21:06:04 UYT [27108]: [3-1] user=dbusr,db=dbdatabase ,app=[unknown],client=10.97.67.70 WARNING: terminating connection because of crash of another server process 2020-12-20 21:06:04 UYT [27108]: [4-1] user=dbusr,db=dbdatabase ,app=[unknown],client=10.97.67.70 DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

Error found at /var/log/messages

Dec 20 21:06:04 vmlx-postgresql-n1-pom-prod kernel: [20278231.774013] postgres[19303]: segfault at 7f7740354704 ip 00007f772c8ac700 sp 00007ffdac7ff4e0 error 4 in postgres[7f772c813000+5a1000]

Thank you all.

We have a master/slave postgresql cluster with streaming replication and pgpool.

Versions in both postgresql servers are.

postgres: 9.4
OS: Debian GNU/Linux 8 (jessie)

PGPOOL server: version pgpool: pgpool-II version 3.3.3 os: Red Hat Enterprise Linux Server release 6.1

Problem:

We suffer a crash down of the postgresql service at master node with subsequent failover done by pgpool

Could not find the reason of this behavior in master node. We want to know if someone coult tell us, based on the recorded log messages, what could have caused this crash. Following, some relevant lines from the postgresql and linux log.

postgresql.log

2020-12-20 21:06:04 UYT [25922]: [4-1] user=,db=,app=,client= LOG: server process (PID 19303) was terminated by signal 11: Segmentation fault 2020-12-20 21:06:04 UYT [25922]: [5-1] user=,db=,app=,client= DETAIL: Failed process was running: COPY public.act_hi_varinst (id_, proc_def_key_, proc_def_id_, proc_inst_id_, execution_id_, act_inst_id_, case_def_key_, case_def_id_, case_inst_id_, case_execution_id_, task_id_, name_, var_type_, rev_, bytearray_id_, double_, long_, text_, text2_, tenant_id_, state_, create_time_, root_proc_inst_id_, removal_time_) TO stdout; 2020-12-20 21:06:04 UYT [25922]: [6-1] user=,db=,app=,client= LOG: terminating any other active server processes 2020-12-20 21:06:04 UYT [1910]: [3-1] user=dbusr,db=dbdatabase ,app=[unknown],client= WARNING: terminating connection because of crash of another server process 2020-12-20 21:06:04 UYT [1910]: [4-1] user=dbusr,db=dbdatabase ,app=[unknown],client= DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-12-20 21:06:04 UYT [1910]: [5-1] user=dbusr,db=dbdatabase ,app=[unknown],client= HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-12-20 21:06:04 UYT [27108]: [3-1] user=dbusr,db=dbdatabase ,app=[unknown],client= WARNING: terminating connection because of crash of another server process 2020-12-20 21:06:04 UYT [27108]: [4-1] user=dbusr,db=dbdatabase ,app=[unknown],client= DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

Error found at /var/log/messages

Dec 20 21:06:04 server-name kernel: [20278231.774013] postgres[19303]: segfault at 7f7740354704 ip 00007f772c8ac700 sp 00007ffdac7ff4e0 error 4 in postgres[7f772c813000+5a1000]

Thank you all.