The SeaMonkey trunk Mac nightlies from cg-xserve02 are only uploaded to the dated directories, but not copied to latest-trunk. Here's the relavent snippet from the latest nightly build log http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1180857780.23131.gz&fulltext=1 ----------------------------------------------------------- ./upload-packages.sh ssh -2 -l seabld stage.mozilla.org mkdir -p /home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk /home/ftp/pub/seamonkey/nightly/latest-trunk rsync -av -e ssh -2 /builds/tinderbox/SeaMonkey-Trunk/Darwin_8.8.1_Depend/2007060301/packages/ email@example.com:/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/ building file list ... done rsync: failed to set times on "/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/.": Operation not permitted (1) seamonkey-2.0a1pre.en-US.mac.dmg rsync: failed to set times on "/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/.": Operation not permitted (1) sent 23553013 bytes received 40 bytes 9421221.20 bytes/sec total size is 23549982 speedup is 1.00 rsync error: some files could not be transferred (code 23) at /SourceCache/rsync/rsync-24/rsync/main.c(717) command failed!
Seems like the failure is actually happening already when uploading to the dated dir, and then the command for copying to latest-trunk isn't even called.
The linux build finished first, and used the cltbld key to create the dated directory: ssh -2 -l cltbld stage.mozilla.org mkdir -p /home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk /home/ftp/pub/seamonkey/nightly/latest-trunk rsync -av -e ssh -2 /builds/tinderbox/SeaMonkey-Trunk/Linux_2.6.9-42.ELsmp_Depend/2007060301/packages/ firstname.lastname@example.org:/home/ftp/pub/seamonkey/nightly/2007-06-03-01-trunk/ building file list ... done ./ seamonkey-2.0a1pre.en-US.linux-i686.tar.bz2 which leaves the directory as drwxrwxr-x 2 cltbld seamonkey 4096 Jun 3 03:19 2007-06-03-01-trunk
With those permissions, I wonder why that failure happens, as seabld is a member of the seamonkey group, from what I see in /etc/group and that group has write permissions on the directory. Interestingly, both sea-win32-tbox and cb-sea-linux-tbox, which upload as cltbld, have the same problem when copying over the files from the dated directory to latest-trunk which has drwxrwsr-x 2 1038 seamonkey 4096 May 30 09:40 latest-trunk In this case, exiting after the error is no problem, as copying to latest is the last thing supposed to happen there, and the coping itself succeeds. In the case of the Mac box, exiting after the error means that subsequent correction of permissions and copying to latest does not happen on those builds. Apparently the group having write perm is not enough there to be able to set times if the directory is owned by a different user.
Zach, any ideas why this is happening ? I've set the owner of latest-trunk to kairo (to match the other two latest dirs) but I don't expect that to help.
It's now after 1 am, FRI 15 JUN 2007 Mozilla Local Time (PT). Do you know where your SeaMonkey trunk Mac builds are? I just checked: http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-trunk/ [ ] seamonkey-2.0a1pre.en-US.mac.dmg 12-Jun-2007 19:27 22M http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2007-06-13-01-trunk/ [ ] seamonkey-2.0a1pre.en-US.mac.dmg 13-Jun-2007 03:16 22M http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2007-06-14-01-trunk/ [ ] seamonkey-2.0a1pre.en-US.mac.dmg 14-Jun-2007 03:24 23M That's progress, but it snagged on something again for the 13th & 14th. Thanks much for the hard work, Eddie
The build from 12th is an unrelated build from a different machine. The failure on cg-xserve02 has not changed.
Kairo: what user account on stage does cg-xserve02 upload as? Is it kairo or seabld?
(In reply to comment #6) > The build from 12th is an unrelated build from a different machine. The failure > on cg-xserve02 has not changed. Yes, those other builds are my talkback experiments. My apologies, I should have shunted them to experimental/ or some such.
Bug 383775 'accidentally' fixed this by using scp instead of rsync for the upload. zach: cg-xserve02 is using seabld.
11 years ago
I don't think this is fixed, nagios is still complaining and the dir listing looks like: [ ] seamonkey-2.0a1pre.en-US.linux-i686.tar.bz2 25-Jun-2007 02:45 14M [ ] seamonkey-2.0a1pre.en-US.mac.dmg 20-Jun-2007 07:54 22M [ ] seamonkey-2.0a1pre.en-US.win32.installer.exe 25-Jun-2007 03:13 8.8M [ ] seamonkey-2.0a1pre.en-US.win32.zip 25-Jun-2007 03:14 12M Over to the FTP maestro's
Very interesting... Do we have log output to know where it's snagging now? Assuming that it's failing on the "setting times" stage again, can we try using scp without the -p switch so that it won't try to do this. I just looked at the most recent dated dir with a mac trunk nightly: -rw-rw-rw- 1 seabld seabld 23639988 Jun 25 03:20 seamonkey-2.0a1pre.en-US.mac.dmg 666 permissions are obviously unsafe, and everything else is 775. Maybe we should try that in the hope that it makes the problem go away? Not entirely sure why the execute bit would be needed, but it's probably worth a shot.
So I'm pretty stumped here. The command that is allegedly failing is: scp -r -p -oProtocol=2 /builds/tinderbox/SeaMonkey-Trunk/Darwin_8.8.1_Depend/2007070201/packages/. email@example.com:/home/ftp/pub/seamonkey/nightly/2007-07-02-01-trunk/ I don't have access to the build machine, but I can't see a reason why that's failing. The tinderbox client is unhelpful enough to not provide the actual error message (if there is one). Can someone with access to the build machine try manually running an scp to see if we can get a real error message?
The error happens on the server side in any case, from what I see. scp actually does copy the file up to the server, but apparently fails when setting permissions, times, etc. on it and comes back with a non-zero return code which triggers the "command failed!" warning on tinderbox and stops the script, prohibiting execution of the remote rsync command to copy from the dated to the latest dir. Even the Linux box fails, but only when doing this remote rsync, with practically the same error messages as in comment #0 ("failed to set times ...") - because of this happening at the last command it should execute, it doesn't do any harm, but still the same error is seen.
Invalid nowadays because we're not using those tinderboc client setups any more and we didn't see this problem with the buildbots ever.