382891 - Tinderbox client should upload files in a better way

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P3)

Description

[Reed Loden [:reed]]

Currently, when a tinderbox is uploading a new build to stage.m.o (surf), it uses scp (at least that's what rhelmer/zach tell me!) to send the file. This is causing a problem when ftp.m.o (manna) attempts to rsync from stage.m.o (surf) in the middle of a tinderbox uploading a build. Basically, rsync gets a "permission denied" error, as scp is using a temporary filename with bad permissions as it uploads the file. After the upload is complete, scp seems to rename the file and set the permissions correctly.

Example:

rsync: send_files failed to open "/seamonkey/tinderbox-builds/hoshi-mozilla1.8.0/.seamonkey-1.0.9.en-US.linux-i686.tar.gz.X78qBH" (in mozilla-all): Permission denied (13)

Note that the file begins with a period and ends with some random characters. This seems to be what scp uses during the file transfer. I'm unsure of what permission the file has during the rsync, but it is restrictive enough to not even allow root to open (and, therefore, copy/rsync) the file.

This issue has been going for a long time from what I can tell. It happens to pretty much all products from what I can tell, so it seems to be a general tinderbox client issue. Let me know if you need anything else.

Comment 1

[Chris Cooper [:coop] (he/him)]

What's the net effect here? Does the next rsync pass from stage to ftp correctly pick up the file? 

Remember that fixing this on the tinderbox side means finding a solution that works on (at least) 3 platforms. Is there a reason this needs to be fixed on the tinderbox side rather than adding some error checking to the stage->ftp rsync process?

Comment 2

[Robert Helmer [:rhelmer]]

There is no rsync for MSYS. Is this problem still happening? I think we need to find a different solution if so.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

I expect this spammy-cron message has gone away, as there's no longer a rsync from stage.m.o to ftp.m.o - cc'ing Server Ops for confirmation. I don't think it's an issue with the new staging setup either.

Comment 4

[Dave Miller [:justdave]]

We still get them constantly with the releases rsyncs.  I wasn't aware that scp used a temp file, I thought it just wrote overtop of the original.  I *know* rsync uses a temp file, and I also know it uses that naming convention.  There's also parameters you can pass to rsync that control the way it does those temp files, such as putting them in a different directory than the destination, etc.

Comment 5

[Dave Miller [:justdave]]

oh, and the rsync to the mirrors (which produces said error) runs on the rsync protocol, and the rsync daemon runs as the user "nobody".  "nobody" can't access said temp files.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Could you give some examples for the releases rsyncs ? Tinderboxes should not be uploading into a release space, so perhaps there are some release files with bad perms that can be fixed. Or maybe it's do with AMO populating mozilla.org/addons.