307306 - [client] post-mozilla-rel.pl doesn't check for upload errors

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

post-mozilla-rel.pl should check for errors when uploading and go orange.  This
makes it easy to have missing nightlies, if, say, somebody messes up the
permissions on the FTP directory.  See bug 307162.

Right now the script just ignores errors and barges ahead:

make[1]: Leaving directory
`/cygdrive/c/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/xpinstall/packager'
make: Leaving directory
`/cygdrive/c/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/xpinstall/packager'
cp
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/dist/install/../*.zip
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/dist/install/2005-09-06-07-trunk/
ssh  -l cltbld stage.mozilla.org mkdir -p /home/ftp/pub/seamonkey/nightly
scp  -r
/cygdrive/c/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/dist/install/2005-09-06-07-trunk
[e-mail]:/home/ftp/pub/seamonkey/nightly
scp: /home/ftp/pub/seamonkey/nightly/2005-09-06-07-trunk: Permission denied
ssh  -l cltbld stage.mozilla.org chmod -R 775
/home/ftp/pub/seamonkey/nightly/2005-09-06-07-trunk
chmod: failed to get attributes of
`/home/ftp/pub/seamonkey/nightly/2005-09-06-07-trunk': No such file or directory
ssh  -l cltbld stage.mozilla.org cp -pf
/home/ftp/pub/seamonkey/nightly/2005-09-06-07-trunk/*
/home/ftp/pub/seamonkey/nightly/latest-trunk/
cp: cannot stat `/home/ftp/pub/seamonkey/nightly/2005-09-06-07-trunk/*': No such
file or directory

WINNT Mozilla Build available at: 
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2005-09-06-07-trunk/

Comment 1

[Robert Kaiser]

Along with that, it would be nice to have an option to create packages locally
but not upload them. With that, you can create tinderbox builds for testing even
if you don't have stage access (yet). I'm telling this here because it might be
something that can be solved in the same patch, and additionally, now one can
just not care about uploading and let it fail and still see a green tinderbox,
but after fixing this bug, it will go orange...

Comment 2

[Benjamin Smedberg]

Kairo, that's simple enough: just set $ssh_server to a local machine.

Comment 3

[Chris Cooper [:coop] (he/him)]

We also fail to check the status when we wget files for repackaging (l10n). See:

http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n/1141422691.11161.gz&fulltext=1

Comment 4

[Chris Cooper [:coop] (he/him)]

Our error checking has been somewhat improved by bug 336463, but we still don't check upload/rsync status very well.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Reassigning to default during triage.

Comment 6

[Robert Helmer [:rhelmer]]

There's a fairly general problem about Tinderbox not checking error codes here. Maybe we should just make sure everything uses an appropriate wrapper to call the shell, and check it there. 

Or we could whack-a-mole and fix just specific issues like uploading or wget (I know there are others), or WONTFIX it if we're just not going to do any of this (this bug has been open a long time considering the severity of the problem and the triviality of a fix like "check error codes when running shell commands").

Comment 7

[Chris Cooper [:coop] (he/him)]

The current iteration of the generated upload-packages.sh script inserts the following stanza after each command:

if [ $? != 0 ]; then
  echo "command failed!"
  exit 1
fi