The generic cluster in PHX1 seems to be overloaded. generic102.webapp.phx1 alerted on high load and low swap. It was getting unresponsive so I had to restart Apache. Shortly after, generic1 hit maxClients, and later generic3 did as well.
we found that helpdesk.mozilla.org was missing a log directory and as a result there were a large number of apache processing stuck trying to write their logs. :jakem has applied this update to puppet and is running a puppetctl across the web nodes now.
these changes have resolved the issues we were seeing. nice work :jakem \o/