375437 - [Nightly Updates] partial update generation non-functional

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

The last partial I can find is dated 25-Mar-2007 08:02 PDT, and there are a bunch of red cells in the update status page (see URL).

Comment 1

[J. Paul Reed [:preed]]

Cause:

Subject: [build-announce] AUS2 Report - 2007/03/26 11:35
From: [e-mail]
Date: Mon, 26 Mar 2007 11:37:31 -0700
To: [e-mail]

Elapsed time:
Command exited with non-zero status 9
2:27.30

Program output:
Found stale PID file... overwriting it.
Running rsync -a  --delete-after -e "ssh -i /home/cltbld/.ssh/aus" [e-mail]:/opt/aus2/build /builds/nightly-partial-generation >/dev/null 2>&1...

Running cd /builds/nightly-partial-generation/build; find -depth -mtime +4 -type f -exec rm -f  \{\} \; ; find -depth -type d -empty -exec rmdir  \{\} \;...

Building patches for Sunbird/branch/WINNT_x86-msvc/en-US/2007032408
      => Updating to Sunbird/branch/WINNT_x86-msvc/en-US/2007032505
Touching /builds/nightly-partial-generation/aus/0/Sunbird/branch/WINNT_x86-msvc/2007032408/en-US/complete.txt... done
Touching /builds/nightly-partial-generation/aus/0/Sunbird/branch/WINNT_x86-msvc/2007032408/en-US/partial.txt... done
Downloading http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-24-08-mozilla1.8/sunbird-0.5pre.en-US.win32.complete.mar...
--11:37:30--  http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-24-08-mozilla1.8/sunbird-0.5pre.en-US.win32.complete.mar
           => `from.mar'
Resolving ftp.mozilla.org... 10.2.74.11
Connecting to ftp.mozilla.org|10.2.74.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,187,910 (5.9M) [application/octet-stream]

    0K ........ ........ ........ ........ ........ ........ 50%    4.09 MB/s
 3072K ........ ........ ........ ........ ........ ......  100%   55.16 MB/s

11:37:31 (7.51 MB/s) - `from.mar' saved [6187910/6187910]

done
Downloading http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-25-05-mozilla1.8/sunbird-0.5pre.en-US.win32.complete.mar...
--11:37:31--  http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-25-05-mozilla1.8/sunbird-0.5pre.en-US.win32.complete.mar
           => `to.mar'
Resolving ftp.mozilla.org... 10.2.74.11
Connecting to ftp.mozilla.org|10.2.74.11|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
11:37:31 ERROR 404: Not Found.

create_partial_patch: 'wget -O to.mar http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-25-05-mozilla1.8/sunbird-0.5pre.en-US.win32.complete.mar' returned non-zero! at /builds/nightly-partial-generation/app/patch-packager.pl line 537.

Rhelmer said he'd take a look as he's working on that machine today.

It'd be nice to know what caused this breakage.

Comment 2

[Robert Helmer [:rhelmer]]

Looks like a complete.txt got pushed to AUS but there is no complete MAR where it's pointing to:

http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-25-05-mozilla1.8/sunbird-0.5pre.en-US.win32.complete.mar

I've moved the complete.txt away, the nightly update script should be able to get past it now. I'll keep an eye on it and close the bug when we get a successful run.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

This problem has recurred. I've moved the complete.txt that points to 

http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/2007-03-29-05-trunk/sunbird-0.6a1.en-US.win32.complete.mar

out of the way, which should get us back in action.

From the Sunbird build log, there was a problem rsyncing the builds to the ftp staging server:
rsync -av -e ssh -2 /cygdrive/d/builds/tinderbox/Sunbird-Trunk/WINNT_5.2_Depend/2007032905/packages/ [e-mail]:/home/ftp/pub/calendar/sunbird/nightly/2007-03-29-05-trunk/
building file list ... done
./
rsync: failed to set times on "/home/ftp/pub/calendar/sunbird/nightly/2007-03-29-05-trunk/.": Operation not permitted (1)
sunbird-0.6a1.en-US.win32.complete.mar
sunbird-0.6a1.en-US.win32.installer.exe
sunbird-0.6a1.en-US.win32.zip
rsync: mkstemp "/home/ftp/pub/calendar/sunbird/nightly/2007-03-29-05-trunk/.sunbird-0.6a1.en-US.win32.complete.mar.FeNZgm" failed: Permission denied (13)
rsync: mkstemp "/home/ftp/pub/calendar/sunbird/nightly/2007-03-29-05-trunk/.sunbird-0.6a1.en-US.win32.installer.exe.YqIwfZ" failed: Permission denied (13)
rsync: mkstemp "/home/ftp/pub/calendar/sunbird/nightly/2007-03-29-05-trunk/.sunbird-0.6a1.en-US.win32.zip.A3EDVG" failed: Permission denied (13)
rsync: failed to set times on "/home/ftp/pub/calendar/sunbird/nightly/2007-03-29-05-trunk/.": Operation not permitted (1)

sent 20177250 bytes  received 92 bytes  3668607.64 bytes/sec
total size is 20174480  speedup is 1.00
rsync error: some files could not be transferred (code 23) at /home/lapo/packaging/tmp/rsync-2.6.6/main.c(791)
command failed!

That directory has owner/group of calbld/calbld, while many of the other ones in sunbird/nightly are cltbld/mozilla. This situation should get better once bug 373373 is fixed and the transition to community tinderboxes is completed.