586179 - Builds not showing up on tinderbox

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

eg there was a 'Linux mozilla-central build' on the Firefox tree that built between 21:23 and 21:38, of which there is no sign on the waterfall. Revision was cbf493ff91ae.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I'm not seeing errors on our machines, so we are sending mail out through mail.build.mozilla.org (aka smtp.mozilla.org aka dm-mail03) OK. I seems to not be getting to dm-webtools02, or that is so buried it's not being processed.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

mail.build is the relay, tinderbox-daemon@tinderbox.m.o the destination.

Comment 3

[Shyam Mani [:fox2mike]]

well, we're doing a bunch of kernel upgrades across the board...but I'll look.

Comment 4

[Dave Miller [:justdave]]

dm-mail03 did get rebooted, a while ago.  It's up now though.  Linux boxes should have queued their mail.  I'd only suspect it lost if it was Windows.

Comment 5

[Dave Miller [:justdave]]

oh, I take that back, dm-mail03 hasn't been done yet, it was one of the ones with a broken package manager.  I'm doing it right now.

Comment 6

[Shyam Mani [:fox2mike]]

mkanat saw these :

The original message was received at Wed, 11 Aug 2010 03:25:32 -0700
from dm-webtools02.mozilla.org [10.2.74.14]

   ----- The following addresses had permanent fatal errors -----
"|handlemail.pl /var/www/webtools/tinderbox"
    (reason: 9)
    (expanded from: tinderbox-daemon-local)

   ----- Transcript of session follows -----
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 680.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 680.
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 680.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 684.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 687.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 690.
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 691.
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 692.
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 698.
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 706.
Global symbol "$form_ref" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 708.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 709.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 710.
Global symbol "$tree" requires explicit package name at /var/www/webtools/tinderbox/tbglobals.pl line 711.
Unmatched right curly bracket at /var/www/webtools/tinderbox/tbglobals.pl line 716, at end of line
/var/www/webtools/tinderbox/tbglobals.pl has too many errors.
Compilation failed in require at /etc/smrsh/handlemail.pl line 25.
554 5.3.0 unknown mailer error 9


Which could explain your missing emails. How do we fix?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps an accidental edit of the tinderbox code ? Can you diff against CVS ?

Comment 8

[Dave Miller [:justdave]]

the error in comment 6 is from stage, not production.  It's unrelated to this bug.

Comment 9

[Mike Taylor [:bear]]

it's possible if that tree was not present in tinderbox's current data that any email glitch would have prevented that tree entry from being created so it's missing from the waterfall

Comment 10

[Dave Miller [:justdave]]

if you know a general time when it was sent, we can check for it in the quarantine (it'll drop the mail into data/bad if it has parsing errors)

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

[in the meantime we've been using clean.pl to remove builds older than 45 days]

Builds which were slow to be processed are now not appearing at all. Last change was 19:24 on Firefox, 20:35 on MozillaTry; both have had builds since then. The trees are effectively closed, it's a big deal.

Comment 12

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

In case it's useful, last sign of any build activity on Camino tinderbox tree was 18:26 (though actual builds are ongoing).

Comment 13

[Dave Miller [:justdave]]

How are things looking now?  I kickstarted a new machine, mounted the tinderbox docroot on it over NFS, and shunted all the web traffic over to that (while leaving the mail processing on the original machine).  It looks like the load has leveled off, and the mail queue runner job that's supposed to run once a minute seems to actually be completing in less than a minute now (instead of 20 to 30 minutes each pass).

Comment 14

[Phil Ringnalda (:philor)]

Seems cooperative - I just starred a build and got tbpl to load the summary on the first try without a single timeout, which I know I used to do but I can't remember the last time it happened.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

I agree it's working much better now. The true test will be once Mountain View wakes up and starts the daily avalanche of builds on try and mozilla-central.

Comment 16

[Dave Miller [:justdave]]

Per the last several comments, this seems fixed.