1

I'm suddenly having intermittent 2003 connection errors and I can't find anything that has changed that can explain it. These servers have been running smoothly for years and suddenly two days ago these errors started showing up in my logs. The application successfully connects to and queries the database almost all the time but sporadically hangs when trying to connect after several queries in quick succession. The connection timeouts always happen at random intervals. It worked just fine before two days ago and the code hasn't changed in 4 years.

Things I've tried that didn't work:

  • Checked all InnoDB tables for corruption
  • Increased the back_log value in my.cnf from 50 to 1000 (and restarted MySQL)
  • Increased the max_connections value in my.cnf from 100 to 200 (and restarted MySQL)
  • Increased net.core.somaxconn from 128 to 1024
  • Increased net.ipv4.tcp_max_syn_backlog from 1024 to 8192
  • Verified there are no firewall rules blocking or throttling connections
  • Flushed the MySQL query cache
  • Cleared the MySQL query cache
  • Restarted the application server and database server (there are two separate servers)
  • Asked it nicely to simply start working again

My servers are EC2 instances in AWS and nothing about their configuration has been changed. There's no unusual traffic and the CPU, memory, and disk usage are well below 80%. Please help!

6
  • Please post results of SELECT @@version; Sep 22, 2021 at 21:30
  • @WilsonHauck the version is 5.5.20-log
    – aecend
    Sep 24, 2021 at 20:01
  • 1
    just wondering if you'd resolved this problem? If so, what were the root causes?
    – user240003
    Oct 7, 2021 at 0:34
  • 1
    Still unresolved.
    – aecend
    Oct 7, 2021 at 0:36
  • 1
    We've recently experienced the same symptoms as the problem you've described. We're still working through possible lines of enquiry. I would be interested to know if your problem gets resolved.
    – user240003
    Oct 7, 2021 at 2:11

1 Answer 1

0

G'day

Our issue was resolved by AWS Support informing us that there had been intermittent elevated error rates or timeouts on requests to Lambda functions connecting to VPC enabled resources in the AP-SOUTHEAST-2 Region.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.