Symptoms:
WARNING: terminating connection because of crash of another server processDETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.HINT: In a moment you should be able to reconnect to the database and repeat your command.FATAL: the database system is in recovery mode
Check the Postgres database logs first to find out more details about when and why a server process is crashed. Here checkpointer process (PID 1095) got killed but there is no clear message why it got killed. There is a hint related to memory "possibly corrupted shared memory". To get a confirmation we can check /var/log/messages for possible error messages related to memory. We can see in the below log the same PID(1095) is recorded there as well and it got killed due to out of memory issue.
postgres.log
2022-07-30 00:01:25.290 [14388] LOG: checkpointer process (PID 1095) was terminated by signal 9: Killed[root@hostname user]# grep -i "Out of memory" /var/log/messages-20220731
2022-07-30 00:01:25.290 [14388] LOG: terminating any other active server processes
2022-07-30 00:01:25.300 [19233] WARNING: terminating connection because of crash of another server process
2022-07-30 00:01:25.300 [19233] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-07-30 00:01:25.300 [19233] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2022-07-30 00:01:25.300 [19170] WARNING: terminating connection because of crash of another server process
Jul 28 01:10:39 hostname kernel: Out of memory: Kill process 22876 (postmaster) score 238 or sacrifice child
Jul 30 00:01:23 hostname kernel: Out of memory: Kill process 1095 (postmaster) score 238 or sacrifice child
Jul 30 00:01:23 hostname kernel: Out of memory: Kill process 18242 (postmaster) score 221 or sacrifice child
Jul 30 00:01:24 hostname kernel: Out of memory: Kill process 9107 (postmaster) score 203 or sacrifice child
Jul 30 00:01:27 hostname kernel: Out of memory: Kill process 18215 (postmaster) score 193 or sacrifice child
Conclusion: Eventually the database logs record a message "the database system is in recovery mode" for a few seconds and perform the automatic recovery. It is clearly evident that this is caused by out-of-memory error as per /var/log/message statements. We can increase the memory and monitor the server for a few days.
──────────────────
Tags: postgres issues, postgres primary db is in readonly mode, postgres WARNING: terminating connection because of crash of another server process,WARNING: terminating connection because of crash of another server process postgres, postgres DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory, postgres HINT: In a moment you should be able to reconnect to the database and repeat your command, postgres FATAL: the database system is in recovery mode, postgres kernel: Out of memory: Kill process (postmaster) score or sacrifice child,kernel: Out of memory: Kill process
DISCLAIMER
The purpose of sharing the content on this website is to Educate. The author/owner of the content does not warrant that the information provided on this website is fully complete and shall not be responsible for any errors or omissions. The author/owner shall have neither liability nor responsibility to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the contents of this website. So, use the content of this website at your own risk.
This content has been shared under Educational And Non-Profit Purposes Only. No Copyright Infringement Intended, All Rights Reserved to the Actual Owner.
For Copyright Content Removal Please Contact us by Email at besttechreads[at]gmail.com