For the last few days I've been trying to figure out why my MySQL read slave servers are capped at 1012+1 (+1 for the superuser) connections when I have the max_connections set to 5000. It seems that no matter what I change, the "Historic max_used_connections" never increases beyond 1013.
The mysql read slaves support a high traffic web site. I have a cluster of 3 read only slaves. Each read slave seems to be capped at 1013 max connections. I can take two out of the cluster so all connections go to one and still only 1013. No error messages show up in the mysql log file.
A mysqladmin status on one of the read slaves shows this:
Uptime: 1661 Threads: 128 Questions: 17634224 Slow queries: 0 Opens: 2441 Flush tables: 1 Open tables: 938 Queries per second avg: 10616.630
Same server, mtop shows this:
load average: 0.05, 0.15, 0.19 mysqld 5.1.66-log up 0 day(s), 0:28 hrs 106 threads: 4 running, 906 cached. Queries/slow: 1.6K/0 Cache Hit: 99.96% Opened tables: 0 RRN: 278 TLW: 1.0K SFJ: 0 SMP: 0 QPS: 1
Each server has 12 dual-core Intel CPU's (24 cores) with 140 GB of RAM running RHEL 6.3, Kernel 2.6. (x86_64)
Any idea what would be limiting/capping the connections at 1012+1 ?
EDIT Also wanted to add two things:
1) I am running the "free/community" edition of MySQL that you get from a Red Hat 6.3 YUM install. Could there be some limitation in that version?
2) I've changed the /etc/security/limits.conf file on the MySQL read slaves, increasing values for users mysql and root (did not help my situation):
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1161534 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 10240 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited