829169 - ftp nagios alerts going off shortly after unthrottling Firefox 18.0

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Saw this from Nagios:
14:06 < nagios-scl3> Thu 11:06:19 PST [543] releases-zlb.vips.scl3.mozilla.com:http - service for ftp vips is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 
                     Internal Server Error - 630 bytes in 1.014 second response time (http://m.allizom.org/http+-+service+for+ftp+vips)
14:07 < nagios-scl3> Thu 11:07:09 PST [548] ftp3.dmz.scl3.mozilla.com:http - service for ftp vips is CRITICAL: Connection refused 
                     (http://m.allizom.org/http+-+service+for+ftp+vips)
14:07 < nagios-scl3> Thu 11:07:19 PST [550] ftp1.dmz.scl3.mozilla.com:http - service for ftp vips is CRITICAL: Connection refused 
                     (http://m.allizom.org/http+-+service+for+ftp+vips)
14:08 < nagios-scl3> Thu 11:08:28 PST [554] ftp6.dmz.scl3.mozilla.com:http - service for ftp vips is CRITICAL: Connection refused 
                     (http://m.allizom.org/http+-+service+for+ftp+vips)
14:09 < nagios-scl3> Thu 11:09:28 PST [559] ftp2.dmz.scl3.mozilla.com:http - service for ftp vips is CRITICAL: Connection refused 
                     (http://m.allizom.org/http+-+service+for+ftp+vips)
14:09 < nagios-scl3> Thu 11:09:38 PST [561] ftp1-zlb.vips.scl3.mozilla.com:https - service for ftp vips is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 
                     Internal Server Error - 630 bytes in 0.019 second response time (http://m.allizom.org/https+-+service+for+ftp+vips)
14:09 < nagios-scl3> Thu 11:09:39 PST [563] ftp4.dmz.scl3.mozilla.com:http - service for ftp vips is CRITICAL: Connection refused 
                     (http://m.allizom.org/http+-+service+for+ftp+vips)
14:11 < nagios-scl3> Thu 11:11:08 PST [567] releases-zlb.vips.scl3.mozilla.com:https - service for ftp vips is CRITICAL: CRITICAL - Socket timeout 
                     after 10 seconds (http://m.allizom.org/https+-+service+for+ftp+vips)


And I've seen complaints from some users too.

Comment 1

[Ed Morley [:emorley]]

All trees closed, since jobs are failing with variations on https://tbpl.mozilla.org/php/getParsedLog.php?id=18680520&tree=Mozilla-Inbound

Comment 2

[u279076]

Escalating this to blocker since it prevents QA from qualifying updates.

Comment 3

[Rick Bryce [:rbryce]]

We have noticed similar issues in recent weeks, when the apache config changes.  I had to restart 1 ftp server manually.  Still investigating, but this appears to be resolved.

Comment 4

[[e-mail] (:bhearsum)]

I think this is fixed now?

Comment 5

[Rick Bryce [:rbryce]]

ftp1-6 are in working order now.  We do have a bug to resolve what I think is the underlying issue here.  Bug 826405

Comment 6

[Mounir Lamouri (:mounir)]

Is it safe to reopen the trees?

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Rick Bryce [:rbryce] from comment #5)
> ftp1-6 are in working order now.  We do have a bug to resolve what I think
> is the underlying issue here.  Bug 826405

This is a legal bug, wrong bug #?

Comment 8

[Ed Morley [:emorley]]

(In reply to Mounir Lamouri (:mounir) from comment #6)
> Is it safe to reopen the trees?

Done

Comment 9

[Rick Bryce [:rbryce]]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> (In reply to Rick Bryce [:rbryce] from comment #5)
> > ftp1-6 are in working order now.  We do have a bug to resolve what I think
> > is the underlying issue here.  Bug 826405
> 
> This is a legal bug, wrong bug #?

Bug 826495

Comment 10

[u279076]

19b1 update testing seem fine on releasetest now.