841519 - hgweb servers under heavy load

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Michael Burns [:mburns]]

<nagios-scl3> hgweb6.dmz.scl3.mozilla.com:Load is WARNING: WARNING - load average: 24.08, 19.29, 10.99
<nagios-scl3> hg-zlb.vips.scl3.mozilla.com:https - ssl hgweb string search is CRITICAL: CRITICAL - Socket timeout after 10 seconds

#IT <RyanVM> mburns: seeing a lot of timeouts around hg/tbpl/etc


hgweb/tbpl have been under heavy load. bkero has investigated and didn't see anything malicious about the activity,

Comment 1

[Ben Kero [:bkero]]

http://it.pastebin.mozilla.org/2139785 contains the outstanding requests (the ones preceded by W). I asked releng to comment.

They said that the annotate requests were done by humans and are expensive, although I doubt this would result in the same condition.

RyanVM said the trees are now open again.

Comment 2

[hwine]

Nothing unusual happening on the RelEng side to explain this.

Comment 3

[hwine]

I'll note 3 instances of request for 'server-status' (which is not enabled on our web heads). And 3 instances of request for mozilla-central bundle <= a bundle request takes 4 minutes of CPU time each (as seen by a HEAD request for it). It may be worth seeing if the bundle requests came from the build-vpn (legit) or externally (somewhat odd)

Comment 4

[Ben Kero [:bkero]]

Closing as nothing can be done for historical cases like this. If the issue happens again please file a new bug. We have better instrumentation for figuring out what is happening now.