Aurora / MySQL 5.6 process list thousands of threads in state 'cleaned up'

Question

I have a scaling issue with an application, it uses Aurora as the DB and uses Slick as the ORM. I am not sure where the issue lies but I have a question releated to MySQL / Aurora that I hope will shed some light on my issue.

The application needs a significant number of connections in the pool otherwise the application starts failing. However the when I look at INFORMATION_SCHEMA.PROCESSLIST I can see a significant number of threads in the 'cleaned up' state.

I can't see 'cleaned up' on the list of general states - https://dev.mysql.com/doc/refman/5.7/en/general-thread-states.html

What does 'cleaned up' mean exactly? ( I can guess but I can't find any documentation )

below is a count of the number of threads total vs the number in 'cleaned up'

mysql> select count(*) from INFORMATION_SCHEMA.PROCESSLIST where STATE = 'cleaned up' order by time desc ;
+----------+
| count(*) |
+----------+
|    11982 |
+----------+
1 row in set (0.08 sec)

mysql> select count(*) from INFORMATION_SCHEMA.PROCESSLIST order by time desc ;
+----------+
| count(*) |
+----------+
|    12007 |
+----------+
1 row in set (0.08 sec)

Answer 1

Finally found the answer, its an Aurora specific state that has the same meaning as the blank state in MySQL.

Cleaned Up state means it has completed and cleaned up and waiting for the connection to close.

https://forums.aws.amazon.com/thread.jspa?messageID=708499

Answer 2

(Not an Answer, but a course of action...)

What is the value of max_connections? It sounds like it is terribly large. It is a very rare system that needs more than 1000 for that setting.

So.. Lower the setting (in my.cnf) to 500 and restart mysqld. This will either fix the problem of push it back to the client, which will then complaint that it cannot get connection to the server.

So, then, ... figure out why there are so many clients clamoring for database activity.

And, find out what queries are so slow that they can't get in and out quickly. It may be as simple as a missing composite index.