Closed Bug 877887 Opened 12 years ago Closed 12 years ago

Multiple websites are offline because of probable puppet change

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

All
Other
task
Not set
blocker

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: fox2mike, Assigned: fox2mike)

References

Details

Tracker. Timeline so far : [15:06:43] < nagios-phx1> | Thu 15:06:43 PDT [134] bouncer2.webapp.phx1.mozilla.com:httpd max clients is UNKNOWN: Unknown: Something weird happend or the server did not respond. [15:17:54] <@ fox2mike> | it's like apache isn't configured to run on port 81 [15:27:12] <% dumitru> | May 30 15:22:00 svn1 puppet-agent[595]: (/File[/etc/httpd/mozilla/defaults.conf]/ensure) removed This will affect anything using the webapp module.
Assignee: server-ops → shyam
Severity: minor → blocker
Component: Server Operations → Server Operations: Web Operations
QA Contact: shyam → nmaul
Group: infra → mozilla-corporation-confidential
Seems like this is only affecting sites in phx1.
Group: mozilla-corporation-confidential
Blocks: 877876
Blocks: 877905
We tracked this down to puppet shooting itself in the foot with an upgrade of mod_passenger. Most services have recovered and we're looking to make sure everything is back online. Apologies for the inconvenience caused.
No longer blocks: 877876
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Our 23+ site test-suites are all roses, now; verified FIXED (and checked http://status.mozilla.com/).
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.