Ryan noticed and i confirmed that we get ISE Error 503 from hg.mozilla.org - could someone take a look at this, thanks! Trees closed for this error
i don;t get the error 503 now but somehow hg operations are now super slow here. Keeping the trees closed for investigation
Nagios just recovered. hgweb8.dmz.scl3.mozilla.com:http - hgweb Port 80 is OK: HTTP OK: HTTP/1.1 200 Script output follows - 14339 bytes in 2.797 second response time Fubar is looking at this, hopefully he has more info.
reopen the trees 7:41 but leaving the tree open
(In reply to Carsten Book [:Tomcat] from comment #4) > reopen the trees 7:41 but leaving the tree open err that should be leaving the bug open for investigation :)
http://nigelbabu.github.io/hgstats/#hours/2 shows a spike in http requests, but not very far outside other spikes. the only errors logged are "Timeout when reading response headers from daemon process" from hgweb.wsgi, and resulting tracebacks for dropped connections. A bunch of build/tools and mozharness clones were taking a really long time - ~280 seconds, and it looks like a bunch of the above were some that took even longer. it looks like most of the build/ reqs were from AWS I don't see any obvious smoking gun.
updating to show this is RFO needed
If we haven't done it in a year, we're not doing it.