939525 - Community servers inaccessible after scl3 maintenance (incl. updates.bugzilla.org:80/tcp)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: DC Other, task)

Description

[Teemu Mannermaa (:wicked)]

After scl3 network maintenance mentioned in https://status.mozilla.org/en-US/detail/22/ we (Bugzilla Project) have lost access to our community servers. They are also not servicing people as intended. Outage includes private backend LAN connections from our virtual host (landfill.bugzilla.org) so the physical servers are either offline or network is totally broken for them.

The servers in question are:

  bugzilla1.community.scl3.mozilla.com
  bugzilla2.community.scl3.mozilla.com
  bugzilla3.community.scl3.mozilla.com

Can we get them back up ASAP, since especially http://updates.bugzilla.org needs to be up or Bugzilla installations all over the world will be affected negatively.

Comment 1

[Rick Bryce [:rbryce]]

> Can we get them back up ASAP, since especially http://updates.bugzilla.org
> needs to be up or Bugzilla installations all over the world will be affected
> negatively.

Is this really the case?  I have no docs and no alerting of any kind on this.  I want to help but I don't even have access info for the out-of-band for theses servers.  

Any idea who handles these servers regularly, or who built them in the first place?

Comment 2

[Frédéric Buclin]

(In reply to Rick Bryce [:rbryce] from comment #1)
> Is this really the case?

All Bugzilla installations now get:

"The remote file http://updates.bugzilla.org/bugzilla-update.xml cannot be downloaded (reason: 500 Can't connect to updates.bugzilla.org:80 (timeout)).
Either the remote server is temporarily unavailable, or your web server cannot access the web."

This doesn't prevent Bugzilla from working, it's just an annoying warning that all admins get since last night. This also means Bugzilla is unable to look for new releases itself as it cannot access bugzilla-update.xml anymore.

Comment 3

[Frédéric Buclin]

Also, Tinderbox and bugbot are gone.

Comment 4

[Rick Bryce [:rbryce]]

I did get access to the oob.  Servers are powered on, and this appears to be a network issue.

Comment 5

[:Atoll]

Moving to Netops queue to raise awareness.

Comment 6

[Adam Newman [:adam]]

awareness raised.

Comment 7

[Justin Wood (:Callek)]

at first glance this does not negatively impact any SeaMonkey Systems.

I'm CC'ing ewong to mention if he notices any issues though.

Comment 8

[Adam Newman [:adam]]

I've changed the trunk configuration on switch1.r101-23 to the updated config to work with the cores after yesterday's maintenance. This switch was unfortunately missed in the move and we were not alerted to it. updates.bugzilla.org and bugzilla[1-3].community.scl3.mozilla.com are now back online.