778908 - tinderbox.m.o is not generating output

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

+++ This bug was initially created as a clone of Bug #728510 +++

(Apologies for anyone extraneous on the CC list; I don't know all the parties using/managing tinderbox and figured the last "tinderbox is failing" bug would have most of the requisite people.)

Sometime in the past 12-24 hours, tinderbox stopped displaying output for Camino, Bugzilla, NSS, SeaMonkey, etc.

Right now all that's displayed is the "dead tree" info: tree status, and the column headers for Build Time and Guilty; no builders, and no builds.

For "Previous 12 Hours" the header cells for the builders also appear, but no builds.  Going back 12 more hours (or "Previous 24 hours" from the main page), things look normal.

I can confirm on my end that mail is getting sent to tinderbox, so it seems like it's a problem on the tinderbox end.

This obviously makes it difficult for projects that rely on tinderbox to continue making checkins.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Suspiciously correlated with with the downtime between 12 and 3pm Sunday, going by http://tinderbox.mozilla.org/showbuilds.cgi?tree=Camino&maxdate=1343637757&hours=24&legend=0&norules=1

This affects l10n too. Did we lose a route for mail to get into tinderbox.m.o or something ?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Smokey, could you confirm where you are sending from (community build network??), the the destination address, and what MX host that turns into ?

Comment 3

[casey ransom [:casey]]

I see a long backlog of mail on mx1 attempting to contact scl3 hosts on internal IPs, which, currently broken due to work on 735395.

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

It sounds like comment 3 may have identified the problem, but just to answer Nick's question:

(In reply to Nick Thomas [:nthomas] from comment #2)
> Smokey, could you confirm where you are sending from (community build
> network??), the the destination address, and what MX host that turns into ?

From hosts on community.scl3.mozilla.com and comm-build.scl1.mozilla.com
To [e-mail]
With MX
tinderbox.mozilla.org mail is handled by 10 mx1.corp.phx1.mozilla.com.
tinderbox.mozilla.org mail is handled by 10 mx2.corp.phx1.mozilla.com.

(FWIW, flows for Camino builders in comm-build.scl1 to tinderbox.m.o were set up in bug 748973; there's another one for the boxes in community.scl3 that I can't find right now, and I think additional bugs for each other project X network.)

Comment 5

[Teemu Mannermaa (:wicked)]

What's bug 735395 and when do we expect this bug fixed? Having Tinderbox Server dead for us over two days is already bad enough and no comment for over 18h for this blocker is making me worry. :(

Comment 6

[Justin Wood (:Callek)]

(In reply to Teemu Mannermaa (:wicked) from comment #5)
> What's bug 735395 and when do we expect this bug fixed? 

Just to say from that bug, "there's a regression in the routing table affecting at least scl3 -> corp.phx1."

That was commented yesterday late-day. I am hoping it gets handled soon.

Comment 7

[Ravi Pina [:ravi]]

We will make the necessary route changes during our quiet period late this evening.

Comment 8

[Adrian J Fernandez [:Aj]]

The changes were done last night and the mail has been coming in.
There are still some stuck in the queue but should clear soon.

Please verify that everything is working as expected.

Comment 9

[Justin Wood (:Callek)]

Looking good for SeaMonkey.

Comment 10

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Everything looks good for Camino, too; thanks!