0

I will start this post by stating that I am not very experienced in database administration, please accept my apologies for any unclear explanation you may find below.

We have a replica postgresql hosted on AWS RDS, which stopped replicating last week. The instance was flagged as Storage-full.

However, while looking at the free storage space on CloudWatch, we realized there were still roughly 46Gb available on the 50Gb-allocated instance. We increased the allocated space to 60Gb and everything went back to normal, but we knew the issue would come back and it did.

The main instance on which this one is replicated is auto vacuumed. I think any writes resulting on the main will be written on the replica, so this is probably not a vacuum issue.

I could not find any indication of problem in CloudWatch metrics.

It appears that the logs could be the issue here but I don't know where to look to investigate this option.

I will edit the question with any relevant information you might suggest in the comments. Thanks for your help.

Improve this question
9
  • Hi, and welcome to dba.se! Can you run du -h from the various directories under root on your AWS instance to check the space usage of the various systems?
    – Vérace
    Commented Aug 5, 2021 at 13:08
  • Hi Vérace, thanks for taking a look at this. I can ssh into the db instance via psql but I can't find the information to ssh into the instance itself. It should be no more difficult than ssh-ing into an ec2 instance but I can't find proper information on that matter
    – Vincent Rolea
    Commented Aug 5, 2021 at 13:56
  • Still - it's a bit wierd - when you pay for 50GB, shouldn't you get 50GB of useful space - i.e. not including the OS itself? Just a heads up - not a cloud person really - am planning to move though!
    – Vérace
    Commented Aug 5, 2021 at 15:26
  • That was my initial thought.. But it appears the free storage metric from AWS RDS represents the storage taken by the tables of the database and not the volume itself. (I checked using SELECT pg_database_size('mydb'); and numbers match) Logs might be the issue but I can't get access to the information..
    – Vincent Rolea
    Commented Aug 5, 2021 at 15:31
  • 45GB of logs seems a bit excessive! :-) Don't they have an email helpline? If you do get a response, post back here with the answer - it might help others with the same issue! You can answer your own question and also mark it correct - no points though! :-) Just had a thought - you can run shell commands from psql! Shell excitement with the \! psql meta-command.?
    – Vérace
    Commented Aug 5, 2021 at 15:50

1 Answer 1

Reset to default
1

Got an answer back from AWS support.

It appears the problem came from not having enough space left for inodes. We cannot control allocated space for inodes on RDS so the only solution here is to get a bigger instance.

The OOM Killer allowed to get our instance running again.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.