Closed
Bug 386212
Opened 17 years ago
Closed 17 years ago
fx-win32-tbox has lost the ability to report finishing nightly builds
Categories
(Release Engineering :: General, defect, P1)
Release Engineering
General
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: philor, Assigned: ted)
Details
Attachments
(2 files, 1 obsolete file)
1.34 KB,
text/plain
|
Details | |
4.67 KB,
patch
|
benjamin
:
review+
|
Details | Diff | Splinter Review |
The 2007-06-26 http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1182966540.23912.gz&fulltext=1 failure to report the end of the nightly build seemed like just one of those things, since it had been having a really bad day, but the 2007-06-27 failure to report the end again makes it seem suspiciously like something's broken.
Comment 1•17 years ago
|
||
Maybe pushing the build to the nightly directory is somehow failing, and causing mail not to be sent out?
Comment 2•17 years ago
|
||
Builds are present in: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2007-06-27-11-trunk/ http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2007-06-28-05-trunk/
Comment 3•17 years ago
|
||
Today's build finished about 07:33 (PDT). Reed looked at the mail server logs, and at that time there is * a successful mail to build-announce@, notifying that list of the build completion * a quick connect/disconnect at 07:33 * then a mail to tinderbox-daemon@ I'm suggesting the middle transaction is failed attempt to send the large log from the nightly build, and the last mail is start message for the hourly build at 07:34. preed suggests that the this might be fallout from the Spam Filter changes. I'll dig up some info on message subject and so on.
Updated•17 years ago
|
Assignee: server-ops → aravind
Comment 4•17 years ago
|
||
Tinderbox messages appear to have no subject, are To: tinderbox-daemon@tinderbox.mozilla.org (until that gets translated by mail handlers), and from cltbld.
Comment 5•17 years ago
|
||
Aravind did some more digging in the mail logs but found nothing amiss. I'll try to setup some logging in time for tomorrow morning's build. If all else fails we can turn on log compression.
Assignee: aravind → nrthomas
Priority: -- → P1
Comment 6•17 years ago
|
||
Mail log setup, it's at /cygdrive/e/builds/mail.log. Probably the issue will go away now that the box is back on local storage, but this will be interesting anyway.
Updated•17 years ago
|
Attachment #270293 -
Attachment mime type: application/octet-stream → text/plain
Comment 7•17 years ago
|
||
What appears to be happening here is that a problem with the Breakpad symbol store blows the tinderbox message out to more than 30MB in size. Blat tries to send this in two parts and it doesn't work. Over to Ted.
Assignee: nrthomas → ted.mielczarek
Comment 8•17 years ago
|
||
More details - build log: ... building symbol store mkdir -p ./dist/crashreporter-symbols/2007062904 Failed to get CVS Revision for e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dist\public\nss\nssrwlkt.h Failed to get CVS Revision for e:\builds\tinderbox\fx-trunk\winnt_5.2_depend\mozilla\dist\public\nss\secomon.h ... thousands more lines like this Mail log (nightly start message, then failed end message): 2007.06.29 04:04:14 (Fri)------------Start of Session----------------- Blat v2.5.0 w/GSS encryption (build : Sep 14 2005 22:46:29) <<<getline<<< 220 dm-mail01.mozilla.org ESMTP Postfix >>>putline>>> EHLO fx-win32-tbox.fx-win32-tbox.build.mozilla.org <<<getline<<< 250-dm-mail01.mozilla.org <<<getline<<< 250-PIPELINING <<<getline<<< 250-SIZE 31457280 <<<getline<<< 250-ETRN <<<getline<<< 250-ENHANCEDSTATUSCODES <<<getline<<< 250-8BITMIME <<<getline<<< 250 DSN Sending build_start_msg.tmp to tinderbox-daemon@tinderbox.mozilla.org Login name is cltbld@fx-win32-tbox.build.mozilla.org Try number 1 of 5. >>>putline>>> MAIL FROM:<cltbld@fx-win32-tbox.build.mozilla.org> <<<getline<<< 250 2.1.0 Ok >>>putline>>> RCPT TO:<tinderbox-daemon@tinderbox.mozilla.org> <<<getline<<< 250 2.1.5 Ok >>>putline>>> DATA <<<getline<<< 354 End data with <CR><LF>.<CR><LF> <<<getline<<< 250 2.0.0 Ok: queued as 29D736A8132 >>>putline>>> QUIT <<<getline<<< 221 2.0.0 Bye 2007.06.29 04:04:20 (Fri)-------------End of Session------------------ 2007.06.29 05:05:36 (Fri)------------Start of Session----------------- Blat v2.5.0 w/GSS encryption (build : Sep 14 2005 22:46:29) <<<getline<<< 220 dm-mail01.mozilla.org ESMTP Postfix >>>putline>>> EHLO fx-win32-tbox.fx-win32-tbox.build.mozilla.org <<<getline<<< 250-dm-mail01.mozilla.org <<<getline<<< 250-PIPELINING <<<getline<<< 250-SIZE 31457280 <<<getline<<< 250-ETRN <<<getline<<< 250-ENHANCEDSTATUSCODES <<<getline<<< 250-8BITMIME <<<getline<<< 250 DSN Sending 2 parts for this message. >>>putline>>> QUIT <<<getline<<< 221 2.0.0 Bye 2007.06.29 05:05:37 (Fri)-------------End of Session------------------
Assignee | ||
Comment 9•17 years ago
|
||
Ok, this will disable this feature on Windows currently. It doesn't work properly under cygwin python, so we'll re-enable it when we get to a MozillaBuild tinderbox. I've left it enabled on OS X, since it's not a problem there. Linux has some other issues, so I'll re-enable it there once I've looked at those. I don't know if bsmedberg is going to be around to review this, but I don't know anyone else who can currently. In addition, I'm going to be gone all afternoon today, so if someone really wants this, you'll have to check it in without me.
Attachment #270322 -
Flags: review?(benjamin)
Updated•17 years ago
|
Component: Server Operations: Tinderbox Maintenance → Build & Release
QA Contact: justin → preed
Updated•17 years ago
|
Attachment #270322 -
Flags: review?(benjamin) → review+
Assignee | ||
Comment 10•17 years ago
|
||
Checked in. The win32 tbox has already had one extra clobber today, so I'll leave it alone till tomorrow.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 11•17 years ago
|
||
This seems to have done the trick.
Comment 12•17 years ago
|
||
Nice! Mail logging removed. Also noticed that Blat supports a -nomps to suppress multi-part messages (from v2.5, we have 2.5.0 on this box).
Status: RESOLVED → VERIFIED
Assignee | ||
Comment 13•17 years ago
|
||
Since it got switched to an objdir, and with this one extra check, --vcs-info should work fine on Linux.
Attachment #270730 -
Flags: review?(benjamin)
Assignee | ||
Updated•17 years ago
|
Attachment #270730 -
Attachment is obsolete: true
Attachment #270730 -
Flags: review?(benjamin)
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
You need to log in
before you can comment on or make changes to this bug.
Description
•