807313 - dead jobs in command queue relating to spidermonkey try uploads

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Found this on bm32. I'm surprised to see things uploading to try, as this master is a production build pool one, not try....

Possibly related to bug 794339?

INFO:util.commands:command: START
INFO:util.commands:command: /builds/buildbot/build1/bin/python /builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py -r 2 -t 10 --master-name bm32-build1 --try --product spidermonkey --platform linux64-debug --branch try --user trybld -i /home/cltbld/.ssh/trybld_dsa stage.mozilla.org /builds/buildbot/build1/master/try_linux64-debug_spidermonkey-rootanalysis 1
INFO:util.commands:command: stdin: <open file '/dev/null', mode 'r' at 0x1bf82540>
INFO:util.commands:command: cwd: /builds/buildbot/queue
INFO:util.commands:command: output:
Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 343, in <module>
    print ssh(user=options.user, identity=options.identity, host=host, remote_cmd=post_upload_cmd)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 41, in ssh
    return retry(do_cmd, attempts=retries+1, sleeptime=retry_sleep, args=(cmd,))
  File "/builds/buildbot/build1/tools/lib/python/util/retry.py", line 31, in retry
    return action(*args, **kwargs)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 33, in do_cmd
    cmd, retcode, output))
Exception: Command ['ssh', '-l', 'trybld', '-i', '/home/cltbld/.ssh/trybld_dsa', '-p', '22', 'stage.mozilla.org', u'post_upload.py -b try -p spidermonkey --revision 31114f78fe9f --who [e-mail] --builddir try-linux64-debug --release-to-try-builds /tmp/tmp.RdnM39ZUQk /tmp/tmp.RdnM39ZUQk/try_linux64-debug_spidermonkey-rootanalysis-bm32-build1-build1.txt.gz'] returned non-zero exit code 1:
sys.argv: ['/usr/local/bin/post_upload.py', '-b', 'try', '-p', 'spidermonkey', '--revision', '31114f78fe9f', '--who', '[e-mail]', '--builddir', 'try-linux64-debug', '--release-to-try-builds', '/tmp/tmp.RdnM39ZUQk', '/tmp/tmp.RdnM39ZUQk/try_linux64-debug_spidermonkey-rootanalysis-bm32-build1-build1.txt.gz']
Traceback (most recent call last):
  File "/usr/local/bin/post_upload.py", line 511, in <module>
    func(options, upload_dir, files)
  File "/usr/local/bin/post_upload.py", line 341, in ReleaseToTryBuilds
    CopyFileToDir(f, upload_dir, tryBuildsPath)
  File "/usr/local/bin/post_upload.py", line 70, in CopyFileToDir
    os.makedirs(full_dest_dir, 0755)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/ftp/pub/spidermonkey/try-builds'
INFO:util.commands:command: END (31.24 elapsed)

Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 347, in <module>
    main()
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 344, in main
    post_runner.processBuild(options, build_path, request_ids)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 282, in processBuild
    log_url = self.uploadLog(build)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 95, in uploadLog
    output = get_output(cmd, stdin=devnull)
  File "/builds/buildbot/build1/tools/lib/python/util/commands.py", line 92, in get_output
    raise e
subprocess.CalledProcessError: Command '['/builds/buildbot/build1/bin/python', '/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py', '-r', '2', '-t', '10', '--master-name', u'bm32-build1', '--try', '--product', 'spidermonkey', '--platform', 'linux64-debug', '--branch', 'try', '--user', u'trybld', '-i', u'/home/cltbld/.ssh/trybld_dsa', u'stage.mozilla.org', '/builds/buildbot/build1/master/try_linux64-debug_spidermonkey-rootanalysis', '1']' returned non-zero exit status 1

Result: 1

Comment 1

[Chris AtLee [:catlee]]

The directory just got created last night - https://bugzilla.mozilla.org/show_bug.cgi?id=807203

Comment 2

[[e-mail] (:bhearsum)]

Cool. I requeued these and they worked fine.