Quick summary. Wordpress community website, large userbase. Has caching for guests (WP Super Cache). Always operates around load of 1-2, or 5-6 at peak hours CPU.
As of 4pm EST on Feb 15, the CPU load shot to 10 and ever since then has had a "resting" load of 7-8 and is at extreme levels during the day usually 12.
What was tried:
- Many reboots, it goes right back up to 12.
- Blanked out the website for users, guests were served cache. Load stayed at 8 (weird?...)
- Slow query log is useless, logging many non-normally-slow queries because entire server is slow.
Here is a typical htop snapshot: https://i.stack.imgur.com/2o3II.png
The moment it happened: https://i.stack.imgur.com/QXdpV.png
No plugin/code changes...Why?
I sort of feel it's not just slow plugins/frontend queries and such, due to how it immediately jumps to 12 on reboot and how even with the site blanked out (not calling up posts, comments, sidebars, etc) it was still at a very high load.
How can I determine what's going on?