887531 - wiki.m.o is slow

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Dumitru Gherman [:dumitru]]

Got a nagios and catchpoint alert:

[15:24] < nagios-phx1> | Wed 15:24:58 PDT [130] wiki.mozilla.org:HTTP - / is CRITICAL: CRITICAL - Socket timeout after 10 seconds
 Warning: Timing Page Response with Suspect 

newrelic also sees an increase time starting 3:19 PM Pacific time.

Comment 1

[Dumitru Gherman [:dumitru]]

Strange enough, both stage and production seems to have the same issue, as reported in newrelic, starting at the same time.
Other web properties are OK.

Comment 2

[Dumitru Gherman [:dumitru]]

Seeing this:

Error Allowed memory size of 134217728 bytes exhausted (tried to allocate 375923 bytes)

Comment 3

[Dumitru Gherman [:dumitru]]

This recovered while we were looking into it.
Keeping the bug open for further investigation.

Comment 4

[Chris Turra [:cturra]]

the root cause was the work :digi and i were doing here in bug 858512. we have added a CNAME back to the old pp-memcache-generic0{12}.phx nodes while we audit what nodes have hard coded these into their settings files.

will grab ownership of this.

Comment 5

[Chris Turra [:cturra]]

after completing the audit last night, we removed the CNAME records and have seen no further issues overnight. i am marking this bug as r/fixed.