OK and now here are all things I've done so far to troubleshoot.
1) Cloned my server to rule out a hardware problem. No improvement.
2) Took an image of another server running a WordPress site with most of the same plugins and just a different theme and installed this problem site on it. No improvement. Among other things the purpose here was to rule out that I didn't screw something up fiddling around with Apache and MySQL tuning trying to fix the problem.
3) Tried managed WordPress hosting at Flywheel. They are working on it. They made the following suggestions which did not help at all.
A) Install "Delete Expired Transients" plugin and delete expired transients. (Note I did not do "all transients -- use with caution! because I don't really understand what this does)
B) Created an index over that table [wp_options I think] for the "autoload" data which will help speed up that query but doesn't fully solve the issue if transients aren't being removed properly.
C) Behind the scenes things that are totally opaque to me but they mentioned "bumping performance settings" and "tuning things".
The only difference with Flywheel is that the site seems to perform acceptably with CloudFlare enabled whereas CloudFlare could not save my server but I am not sure that would hold with a traffic spike. Without CloudFlare the Flywheel installation is terrible too. I get page load times as long as 8 or 9 seconds. The admin panel is almost unsuable. Today I have not been able to connect Jetpack to WordPress.com due to a 15,000 milisecond time out limitation by Flywheel.
4) I installed a query monitor and while the MySQL performance certainly could be better it does not appear to be the main bottleneck right now.
As I continue to test and tune on my own server configuration I get results like this in the admin panel:
Page generation time: 3.6608 12.2% of 30s limit
Peak memory usage: 104,416 kB 39.8% of 262,144 kB limit
Database Query time: 0.0179
Database Queries: SELECT: 23 SHOW: 1 DELETE: 2
And I see things like this on the front page:
Page generation time: 3.4225 11.4% of 30s
Memory usage 102,837 kB 19.6% of 524,288 kB limit
Query time: 0.1003
Database Query Time: SELECT: 55 SHOW: 1 UPDATE: 1
On the front end I see the page generation time going through the roof sometimes though. It ranges from 3-12 seconds. The rest of the stuff is relatively consistent whether I'm looking at the front end or the admin panel. And this is just me as a single client browsing on my test server - the live site is running off Flywheel now. Sometimes just reloading the home page once pushes the load up over 1! I'll see 3-4 apache processes popup taking a combined 90-100% of the CPU and they remain for a second or a two and then disappear. Of course with just a couple more clients this would bog down completely very fast.
The memory usage seems fine (although again this is with only one client testing) but the CPU is through the roof.
Do you have any other ideas for me? I would really like to solve this.
Thanks.
