628986 - load average on stage is 105, causing failed builds

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

We're losing a whole bunch of nightlies because of it.

Comment 1

[Chris AtLee [:catlee]]

it's mostly post_upload.py instances, which are called when builds finish and are uploading their results to stage.  did something happen to make the disks really slow here?

Comment 2

[[e-mail] (:bhearsum)]

Changes were made in the past couple of weeks to some of the mounts, in bug 614786. Dunno if it's related to this issue or not.

Comment 3

[Corey Shields [:cshields]]

vsftpd on dm-ftp01 is getting hammered, currently 800+ running..  Going to try and settle that down.

Comment 4

[Dave Miller [:justdave]]

killing apache on dm-download02 solved it.  the firefox tree is reverse-proxied on the back-end to dm-ftp01.  Reviewing the logs showed majority of the traffic was complete MARs for Firefox 2.0.0.20.  Added 2.0.0.20 back to the distribution list for mozilla-releases so it would sync out to mirrors, re-enabled apache on dm-download02, haven't had any additional significant load since...

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> killing apache on dm-download02 solved it.  the firefox tree is reverse-proxied
> on the back-end to dm-ftp01.  Reviewing the logs showed majority of the traffic
> was complete MARs for Firefox 2.0.0.20.  Added 2.0.0.20 back to the
> distribution list for mozilla-releases 

Was Firefox2.0.0.20 recently removed from mozilla-releases? Or is this a spike increase in demand for 2.0.0.20 ?


> so it would sync out to mirrors,
> re-enabled apache on dm-download02, haven't had any additional significant load
> since...
ok.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

We currently know that every week for the last many weeks, we get hit with about 1.25M download attempts for 2.0.0.20-complete.  I am working on getting a spreadsheet that details things better, but if this build was recently taken out of the mirror network then that could explain the problem you are seeing.

Comment 7

[Daniel Einspanjer [:dre] [:deinspanjer]]

Nelson, please attach the spreadsheet here when the query finishes.

Comment 8

[Nelson]

Attachment: Fx 2.0 downloads past 30 days

Here's the download info. Broken down by version and download type, all versions of Fx 2.0 with 200 or more avg daily downloads in the analysis period are included.

Comment 9

[[e-mail] (:bhearsum)]

Any chance we have IP addreses of the clients?

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

Yes, I can get that for the most recent data.  Need to move it to a secure bug though to protect user privacy

Comment 11

[[e-mail] (:bhearsum)]

We should spin this issue off into one anyways -- this bug was tracking the outage.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

We didn't remove Fx 2.0.0.20 from mozilla-releases any time recently. It would have been gone for many many months.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps the day has come where we disable all updates from 2.0.0.x -> 2.0.0.24, while leaving the major update from 2.0.0.24 to later branches. I've long held the suspicion that we have people which are stuck in a loop - they download an update, fail to apply it, download again, fail again. Can we prove/disprove that in metrics-land ?

Comment 14

[Daniel Einspanjer [:dre] [:deinspanjer]]

I have the IP data if someone can point me at a bug to post it in.  We are getting thousands of downloads per day from a handful of specific IPs.

Comment 15

[Dave Miller [:justdave]]

bug 628185 removed a few other more recent releases out of the -releases
module, making dm-download02 the only place to go to get those releases.  The
addition of those to dm-download02's load was probably just enough to push it
over.

Comment 16

[Dave Miller [:justdave]]

Did we have any determination of what was going on from metrics?  I don't want to hold this open in my queue, so if there's anything left to do here it needs to be assigned appropriately (or get a new bug filed for the postmortem that goes to the appropriate place so I can close this one).

Comment 17

[Daniel Einspanjer [:dre] [:deinspanjer]]

The attached chart definitely shows significant traffic for downloads of 2.0.0.20.  We never received a bug asking for the IP addresses responsible.

I don't think there is anything more to be done on this bug so I'd think it could be closed out.

Comment 18

[Dave Miller [:justdave]]

Guess so.  If you need anything else from me, reopen it.