383063 - SeaMonkey Mac nightlies not copied to latest-trunk/

Status: RESOLVED INVALID

(mozilla.org :: FTP: Staging, task, P3)

Description

[Robert Kaiser]

The SeaMonkey trunk Mac nightlies from cg-xserve02 are only uploaded to the dated directories, but not copied to latest-trunk.

Here's the relavent snippet from the latest nightly build log http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1180857780.23131.gz&fulltext=1
-----------------------------------------------------------

./upload-packages.sh
ssh -2 -l seabld stage.mozilla.org mkdir -p /home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk /home/ftp/pub/seamonkey/nightly/latest-trunk

rsync -av -e ssh -2 /builds/tinderbox/SeaMonkey-Trunk/Darwin_8.8.1_Depend/2007060301/packages/ [e-mail]:/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/
building file list ... done
rsync: failed to set times on "/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/.": Operation not permitted (1)
seamonkey-2.0a1pre.en-US.mac.dmg
rsync: failed to set times on "/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/.": Operation not permitted (1)

sent 23553013 bytes  received 40 bytes  9421221.20 bytes/sec
total size is 23549982  speedup is 1.00
rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-24/rsync/main.c(717)
command failed!

Comment 1

[Robert Kaiser]

Seems like the failure is actually happening already when uploading to the dated dir, and then the command for copying to latest-trunk isn't even called.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The linux build finished first, and used the cltbld key to create the dated directory:

ssh -2 -l cltbld stage.mozilla.org mkdir -p /home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk /home/ftp/pub/seamonkey/nightly/latest-trunk

rsync -av -e ssh -2 /builds/tinderbox/SeaMonkey-Trunk/Linux_2.6.9-42.ELsmp_Depend/2007060301/packages/ [e-mail]:/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/
building file list ... done
./
seamonkey-2.0a1pre.en-US.linux-i686.tar.bz2

which leaves the directory as
drwxrwxr-x     2 cltbld seamonkey  4096 Jun  3 03:19 2007-06-03-01-trunk

Comment 3

[Robert Kaiser]

With those permissions, I wonder why that failure happens, as seabld is a member of the seamonkey group, from what I see in /etc/group and that group has write permissions on the directory.

Interestingly, both sea-win32-tbox and cb-sea-linux-tbox, which upload as cltbld, have the same problem when copying over the files from the dated directory to latest-trunk which has
drwxrwsr-x  2     1038 seamonkey 4096 May 30 09:40 latest-trunk

In this case, exiting after the error is no problem, as copying to latest is the last thing supposed to happen there, and the coping itself succeeds. In the case of the Mac box, exiting after the error means that subsequent correction of permissions and copying to latest does not happen on those builds.

Apparently the group having write perm is not enough there to be able to set times if the directory is owned by a different user.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Zach, any ideas why this is happening ?

I've set the owner of latest-trunk to kairo (to match the other two latest dirs) but I don't expect that to help.

Comment 5

[Eddie Maddox]

It's now after 1 am, FRI 15 JUN 2007 Mozilla Local Time (PT).
Do you know where your SeaMonkey trunk Mac builds are?

I just checked:
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-trunk/
[   ] seamonkey-2.0a1pre.en-US.mac.dmg             12-Jun-2007 19:27   22M

http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2007-06-13-01-trunk/
[   ] seamonkey-2.0a1pre.en-US.mac.dmg            13-Jun-2007 03:16   22M

http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2007-06-14-01-trunk/
[   ] seamonkey-2.0a1pre.en-US.mac.dmg             14-Jun-2007 03:24   23M


That's progress, but it snagged on something again for the 13th & 14th.


Thanks much for the hard work,
Eddie

Comment 6

[Robert Kaiser]

The build from 12th is an unrelated build from a different machine. The failure on cg-xserve02 has not changed.

Comment 7

[Zach Lipton [:zach]]

Kairo: what user account on stage does cg-xserve02 upload as? Is it kairo or seabld? 

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to comment #6)
> The build from 12th is an unrelated build from a different machine. The failure
> on cg-xserve02 has not changed.

Yes, those other builds are my talkback experiments. My apologies, I should have shunted them to experimental/ or some such. 

Comment 9

[Robert Kaiser]

Bug 383775 'accidentally' fixed this by using scp instead of rsync for the upload.

zach: cg-xserve02 is using seabld.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

I don't think this is fixed, nagios is still complaining and the dir listing looks like:

[   ] seamonkey-2.0a1pre.en-US.linux-i686.tar.bz2  25-Jun-2007 02:45   14M  
[   ] seamonkey-2.0a1pre.en-US.mac.dmg             20-Jun-2007 07:54   22M  
[   ] seamonkey-2.0a1pre.en-US.win32.installer.exe 25-Jun-2007 03:13  8.8M  
[   ] seamonkey-2.0a1pre.en-US.win32.zip           25-Jun-2007 03:14   12M 

Over to the FTP maestro's

Comment 11

[Zach Lipton [:zach]]

Very interesting... Do we have log output to know where it's snagging now?

Assuming that it's failing on the "setting times" stage again, can we try using scp without the -p switch so that it won't try to do this. 

I just looked at the most recent dated dir with a mac trunk nightly: 

-rw-rw-rw-    1 seabld seabld    23639988 Jun 25 03:20 seamonkey-2.0a1pre.en-US.mac.dmg

666 permissions are obviously unsafe, and everything else is 775. Maybe we should try that in the hope that it makes the problem go away? Not entirely sure why the execute bit would be needed, but it's probably worth a shot.

Comment 12

[Zach Lipton [:zach]]

So I'm pretty stumped here. The command that is allegedly failing is:

scp -r -p -oProtocol=2 /builds/tinderbox/SeaMonkey-Trunk/Darwin_8.8.1_Depend/2007070201/packages/. [e-mail]:/home/ftp/pub/seamonkey/nightly/2007-07-02-01-trunk/

I don't have access to the build machine, but I can't see a reason why that's failing. The tinderbox client is unhelpful enough to not provide the actual error message (if there is one). Can someone with access to the build machine try manually running an scp to see if we can get a real error message?

Comment 13

[Robert Kaiser]

The error happens on the server side in any case, from what I see.
scp actually does copy the file up to the server, but apparently fails when setting permissions, times, etc. on it and comes back with a non-zero return code which triggers the "command failed!" warning on tinderbox and stops the script, prohibiting execution of the remote rsync command to copy from the dated to the latest dir.
Even the Linux box fails, but only when doing this remote rsync, with practically the same error messages as in comment #0 ("failed to set times ...") - because of this happening at the last command it should execute, it doesn't do any harm, but still the same error is seen.

Comment 14

[Robert Kaiser]

Invalid nowadays because we're not using those tinderboc client setups any more and we didn't see this problem with the buildbots ever.