AWS RDS Serverless-V2 Postgres restart issues

Question

We have an RDS Serverless-V2 Postgres cluster setup with two instances (one read/write, and the other a read). Both instances are in the same AZ for now, although we plan on moving one to a different AZ soon. Additionally we are using an RDS-Proxy for our Lambdas to connect to the db (so shared conns can be used). The ACUs are set to run 0.5 to 8

We are currently perf-testing this setup for possible use in an application (replacing our current DynamoDB db). During these perf tests we are processing (writing) to about 3 million records, while at the same time we are measuring our API (REST GET calls) performance on a few of our main queries. The writes (updates) are very simple where a single field in a record is being updated (a small string field). At the same time our queries are relatively simple (a few tables with simple joins - nothing complex). The queries are being run 10 at a time asynchronously using Jmeter.

The problem is that we are getting intermittent restarts of one (and sometimes both) of the instances. During these restarts the other instance takes over as expected however this has a noticeable impact on our queries which are starting to time-out (15 second timeouts on the handling Lambdas) until the failed instance comes back online (usually a few minutes).

Here are some of the log outputs when one of these restarts happens:

postgres@test:[23685]:DETAIL:  The postmaster has commanded this
   server process to roll back the current transaction and exit, because
   another server process exited abnormally and possibly corrupted
   shared memory.

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC:10.10.253.103(40115):rdsproxyadmin@postgres:[23355]:FATAL: the database system is starting up

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC:10.10.254.218(49269):rdsproxyadmin@postgres:[23348]:FATAL: the database system is starting up

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC:10.10.254.218(38211):rdsproxyadmin@postgres:[23354]:FATAL: the database system is starting up

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC:[local]:rdsadmin@rdsadmin:[23352]:FATAL: the database system is starting up

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC:[local]:rdsadmin@rdsadmin:[23347]:FATAL: the database system is starting up

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC:10.10.253.103(32051):rdsproxyadmin@postgres:[23346]:FATAL: the database system is starting up

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC::@:[23345]:LOG: database system was interrupted; last known up at 2022-06-16 16:48:21 UTC

2022-06-16T13:48:48.000-03:00   2022-06-16 16:48:48 UTC::@:[22011]:LOG: In Aurora for PostgreSQL read only hot standby mode. Database system is ready to accept read only connections.

2022-06-16T13:48:50.000-03:00   2022-06-16 16:48:50 UTC:10.10.253.103(7795):postgres@test:[24095]:LOG: unexpected EOF on client connection with an open transaction

We're very pleased so far with the performance, but this stability issue has us very concerned. Any insights are appreciated.

Thanks

Answer 1

Turns out that our issue was related to our minimum ACU being set to 0.5. This did not allow enough cpu or memory capacity when high-level write and read demands were made.

As per this AWS best-practices guide:

"If there is a sudden spike in requests, you can overwhelm the database. Aurora Serverless might not be able to find a scaling point and scale quickly enough due to a shortage of resources. This is especially true when your cluster is at 1 ACU capacity, which corresponds to approximately 2 GB of memory. Typically 1 ACU is not adequate for production workloads."

We have since rerun our perf test using minimum 6 and maximum 8 ACUs and had no issues. From here we'll start to fine-tune our range to minimize costs and increase reliability.