We removed the trends box, because it used an unnecessary outer join that drew in most of the database. We need to replace that query, then put it back. Jeff mentioned Zamboni does inner joins by default for add-ons, so I can borrow code from there.
I have a patch for this that re-enables the box but does not perform a useless LEFT JOIN anymore: http://github.com/fwenzel/reporter/commit/904de4e Need to check how that performs on production, but I am guessing "slightly" better than the old one.
Let's sync the db's and do a load test (page views on various searches as well as the main dashboard frontpage) of this before going to production
I don't think we can reproduce the load that prod is getting on stage. But we can deploy it, then back it out shortly thereafter if the problems from yesterday show up again.
I think we're going to need to find a method to load test stage from here on out before pushing changes to production. I'll look into this.
This is back and working now. To find a load testing solution is going to take some time though and some consultation with stephen donner.