748633 - Thunderbird try logs failing to upload

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[John Hopkins (:jhopkins)]

Exception: Command ['ssh', '-l', 'tbirdtry', '-i', '/home/cltbld/.ssh/trybld_dsa', '-p', '22', 'stage.mozilla.org', 'post_upload.py --tinderbox-builds-dir try-comm-central-win64 -b try-comm-central -p thunderbird-test -i 20120424160341 --revision 1437e018e2e2 --release-to-tinderbox-dated-builds /tmp/tmp.LgHsc23952 /tmp/tmp.LgHsc23952/try-comm-central-win64-bm14-try1-build1.txt.gz'] returned non-zero exit code 1:
Traceback (most recent call last):
  File "/usr/local/bin/post_upload.py", line 441, in ?
    func(options, upload_dir, files)
  File "/usr/local/bin/post_upload.py", line 185, in ReleaseToTinderboxBuilds
    ReleaseToBuildDir(TINDERBOX_BUILDS_PATH, TINDERBOX_URL_PATH, options, upload_dir, files, dated)
  File "/usr/local/bin/post_upload.py", line 180, in ReleaseToBuildDir
    CopyFileToDir(f, upload_dir, tinderboxBuildsPath)
  File "/usr/local/bin/post_upload.py", line 53, in CopyFileToDir
    os.makedirs(full_dest_dir, 0755)
  File "/usr/lib/python2.4/os.py", line 156, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.4/os.py", line 159, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/ftp/pub/thunderbird-test/tinderbox-builds/try-comm-central-win64'

Comment 1

[Mike Taylor [:bear]]

the log upload command .json files have been "parked" into 
  
  /dev/shm/queue/commands/bug748633

on bm09 and bm14

Comment 2

[John Hopkins (:jhopkins)]

We're getting green try builds now, but I've seen a 'dead' file for (what appeared to be) a successful build so I'll keep this open for now.

Comment 3

[Chris AtLee [:catlee]]

This doesn't prevent builds from finishing successfully, it just means logs never get uploaded, which will also prevent the builds being marked as finished in statusdb - which means they'll never finish as far as tbpl is concerned.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Is this still an issue ? I see builds from the 26th and 27th where the upload is into thunderbird-test/try-builds instead of tinderbox-builds/, and the ownership of try-builds looks correct.

Comment 6

[Mike Taylor [:bear]]

we will find out if this is not fixed by the appearance of dead items in the command queue the next time a tbird build happens

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Apparently I spoke too soon, we hit 14 of these errors.

Comment 8

[John Hopkins (:jhopkins)]

Attachment: proposed patch

try-comm-central was not being recognized as a 'try' branch (thanks standard8).

Comment 9

[John Hopkins (:jhopkins)]

Comment on attachment 619578 [details] [diff] [review]
proposed patch

Review of attachment 619578 [details] [diff] [review]:
-----------------------------------------------------------------

Landed in http://hg.mozilla.org/build/buildbotcustom/rev/03c085f326cc