386212 - fx-win32-tbox has lost the ability to report finishing nightly builds

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P1)

Description

[Phil Ringnalda (:philor)]

The 2007-06-26 http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1182966540.23912.gz&fulltext=1 failure to report the end of the nightly build seemed like just one of those things, since it had been having a really bad day, but the 2007-06-27 failure to report the end again makes it seem suspiciously like something's broken.

Comment 1

[:Gavin Sharp [email: [e-mail]]]

Maybe pushing the build to the nightly directory is somehow failing, and causing mail not to be sent out?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Builds are present in:

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2007-06-27-11-trunk/
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2007-06-28-05-trunk/

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Today's build finished about 07:33 (PDT). Reed looked at the mail server logs, and at that time there is 
 * a successful mail to build-announce@, notifying that list of the build completion
 * a quick connect/disconnect at 07:33
 * then a mail to tinderbox-daemon@

I'm suggesting the middle transaction is failed attempt to send the large log from the nightly build, and the last mail is start message for the hourly build at 07:34.

preed suggests that the this might be fallout from the Spam Filter changes. I'll dig up some info on message subject and so on.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Tinderbox messages appear to have no subject, are 
  To: [e-mail] 
(until that gets translated by mail handlers), and from cltbld.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Aravind did some more digging in the mail logs but found nothing amiss. I'll try to setup some logging in time for tomorrow morning's build. If all else fails we can turn on log compression.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Tinderbox changes to make blat log mail

Mail log setup, it's at /cygdrive/e/builds/mail.log. Probably the issue will go away now that the box is back on local storage, but this will be interesting anyway.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

What appears to be happening here is that a problem with the Breakpad symbol store blows the tinderbox message out to more than 30MB in size. Blat tries to send this in two parts and it doesn't work.

Over to Ted.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

More details - build log:

...
building symbol store
mkdir -p ./dist/crashreporter-symbols/2007062904
Failed to get CVS Revision for e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dist\public\nss\nssrwlkt.h
Failed to get CVS Revision for e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dist\public\nss\secomon.h
... thousands more lines like this


Mail log (nightly start message, then failed end message):

2007.06.29 04:04:14 (Fri)------------Start of Session-----------------
Blat v2.5.0 w/GSS encryption (build : Sep 14 2005 22:46:29)
<<<getline<<< 220 dm-mail01.mozilla.org ESMTP Postfix
>>>putline>>> EHLO fx-win32-tbox.fx-win32-tbox.build.mozilla.org
<<<getline<<< 250-dm-mail01.mozilla.org
<<<getline<<< 250-PIPELINING
<<<getline<<< 250-SIZE 31457280
<<<getline<<< 250-ETRN
<<<getline<<< 250-ENHANCEDSTATUSCODES
<<<getline<<< 250-8BITMIME
<<<getline<<< 250 DSN
Sending build_start_msg.tmp to [e-mail]
Login name is [e-mail]
Try number 1 of 5.
>>>putline>>> MAIL FROM:<[e-mail]>
<<<getline<<< 250 2.1.0 Ok
>>>putline>>> RCPT TO:<[e-mail]>
<<<getline<<< 250 2.1.5 Ok
>>>putline>>> DATA
<<<getline<<< 354 End data with <CR><LF>.<CR><LF>
<<<getline<<< 250 2.0.0 Ok: queued as 29D736A8132
>>>putline>>> QUIT
<<<getline<<< 221 2.0.0 Bye
2007.06.29 04:04:20 (Fri)-------------End of Session------------------

2007.06.29 05:05:36 (Fri)------------Start of Session-----------------
Blat v2.5.0 w/GSS encryption (build : Sep 14 2005 22:46:29)
<<<getline<<< 220 dm-mail01.mozilla.org ESMTP Postfix
>>>putline>>> EHLO fx-win32-tbox.fx-win32-tbox.build.mozilla.org
<<<getline<<< 250-dm-mail01.mozilla.org
<<<getline<<< 250-PIPELINING
<<<getline<<< 250-SIZE 31457280
<<<getline<<< 250-ETRN
<<<getline<<< 250-ENHANCEDSTATUSCODES
<<<getline<<< 250-8BITMIME
<<<getline<<< 250 DSN
Sending 2 parts for this message.
>>>putline>>> QUIT
<<<getline<<< 221 2.0.0 Bye
2007.06.29 05:05:37 (Fri)-------------End of Session------------------

Comment 9

[(not currently active) Ted Mielczarek]

Attachment: make getting cvs info an option, and only enable it on OS X currently

Ok, this will disable this feature on Windows currently.  It doesn't work properly under cygwin python, so we'll re-enable it when we get to a MozillaBuild tinderbox.  I've left it enabled on OS X, since it's not a problem there.  Linux has some other issues, so I'll re-enable it there once I've looked at those.

I don't know if bsmedberg is going to be around to review this, but I don't know anyone else who can currently.  In addition, I'm going to be gone all afternoon today, so if someone really wants this, you'll have to check it in without me.

Comment 10

[(not currently active) Ted Mielczarek]

Checked in.  The win32 tbox has already had one extra clobber today, so I'll leave it alone till tomorrow.

Comment 11

[(not currently active) Ted Mielczarek]

This seems to have done the trick.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Nice! Mail logging removed. Also noticed that Blat supports a -nomps to suppress multi-part messages (from v2.5, we have 2.5.0 on this box).

Comment 13

[(not currently active) Ted Mielczarek]

Attachment: re-enable --vcs-info on linux

Since it got switched to an objdir, and with this one extra check, --vcs-info should work fine on Linux.