751878 - OSError: [Errno 13] Permission denied: '/home/ftp' for pvtbuilds2.dmz.scl3.mozilla.com

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

Now that bug 751715 got fixed we see this other issue.

INFO:util.commands:command: /builds/buildbot/build1/bin/python /builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py -r 2 -t 10 --master-name bm13-build1 --product fuzzing --platform win32 --branch idle --user ffxbld -i /home/cltbld/.ssh/ffxbld_dsa pvtbuilds2.dmz.scl3.mozilla.com /builds/buildbot/build1/master/fuzzer-win32 260
INFO:util.commands:command: stdin: <open file '/dev/null', mode 'r' at 0x5c928a0>
INFO:util.commands:command: cwd: /builds/buildbot/queue
INFO:util.commands:command: output:
Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 331, in <module>
    print ssh(user=options.user, identity=options.identity, host=host, remote_cmd=post_upload_cmd)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 40, in ssh
    return retry(do_cmd, attempts=retries+1, sleeptime=retry_sleep, args=(cmd,))
  File "/builds/buildbot/build1/tools/lib/python/util/retry.py", line 31, in retry
    return action(*args, **kwargs)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 32, in do_cmd
    cmd, retcode, output))
Exception: Command ['ssh', '-l', 'ffxbld', '-i', '/home/cltbld/.ssh/ffxbld_dsa', '-p', '22', 'pvtbuilds2.dmz.scl3.mozilla.com', 'post_upload.py --tinderbox-builds-dir idle-win32 -b idle -p fuzzing --release-to-tinderbox-builds /tmp/tmp.aUtgEvzDH8 /tmp/tmp.aUtgEvzDH8/fuzzer-win32-bm13-build1-build260.txt.gz'] returned non-zero exit code 1:
Traceback (most recent call last):
  File "/usr/local/bin/post_upload.py", line 441, in <module>
    func(options, upload_dir, files)
  File "/usr/local/bin/post_upload.py", line 192, in ReleaseToTinderboxBuildsOverwrite
    ReleaseToTinderboxBuilds(options, upload_dir, files, dated=False)
  File "/usr/local/bin/post_upload.py", line 185, in ReleaseToTinderboxBuilds
    ReleaseToBuildDir(TINDERBOX_BUILDS_PATH, TINDERBOX_URL_PATH, options, upload_dir, files, dated)
  File "/usr/local/bin/post_upload.py", line 180, in ReleaseToBuildDir
    CopyFileToDir(f, upload_dir, tinderboxBuildsPath)
  File "/usr/local/bin/post_upload.py", line 53, in CopyFileToDir
    os.makedirs(full_dest_dir, 0755)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/ftp'
INFO:util.commands:command: END (31.60 elapsed)

Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 327, in <module>
    main()
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 324, in main
    post_runner.processBuild(options, build_path, request_ids)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 265, in processBuild
    log_url = self.uploadLog(build)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 95, in uploadLog
    output = get_output(cmd, stdin=devnull)
  File "/builds/buildbot/build1/tools/lib/python/util/commands.py", line 89, in get_output
    raise e
subprocess.CalledProcessError: Command '['/builds/buildbot/build1/bin/python', '/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py', '-r', '2', '-t', '10', '--master-name', u'bm13-build1', '--product', 'fuzzing', '--platform', 'win32', '--branch', 'idle', '--user', u'ffxbld', '-i', u'/home/cltbld/.ssh/ffxbld_dsa', u'pvtbuilds2.dmz.scl3.mozilla.com', '/builds/buildbot/build1/master/fuzzer-win32', '260']' returned non-zero exit status 1

Result: 1

Comment 1

[Armen [:armenzg]]

Hi dustin, if you think you can get to this today please let me know.
Otherwise I will downtime the nagios check until whenever you tell you expect to get to them.

e.g. [11] buildbot-master08.build.sjc1:Command Queue is CRITICAL: 2 dead items

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I'll need to talk to someone who fully understands this project - I built this as described, and that seems to have not been a full description.  So I expect that fixing this problem will just reveal another.

Are you available later today?  Say, around 11:45 pacific?

Comment 3

[Armen [:armenzg]]

I am but I might enough know less than you already know.
We can fix and test until it works, maybe?

It is not a big issue it can wait until Monday if it works better.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

OK, let's wait until Monday.  I think that the right solution here is to merge this into the productdelivery module that digi built for the new stage.  The current form of that server doesn't bend at the right places, but since it's going into production this weekend, I don't want to mess with it right now.

I'll also need to find someone who understands how these uploads work, so I can abstract things properly.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

post_upload.py knows about two types of locations, public ones based at /home/ftp and private ones at /mnt/pvt_builds. The problem here is that we have fuzzing logs which are private, but we're not passing the right incantation to post_upload.py to tell it that. I think the simplest thing to do here is to teach the log upload that fuzzing logs are private, since that avoids re-arranging the server config and the fuzzing scripts themselves.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I thought this was as simple as doing this in buildbotcustom/bin/postrun.py
-        elif branch == 'shadow-central':
+        elif product == 'fuzzing':
             upload_args.append("--shadow")
but then you get errors from post_upload.py about the buildID not being set. 

I also don't see any code in postrun.py for suppressing Pulse messages for fuzzing, which seems desirable, so I'm tossing this back into the pool.

In the meantime I landed http://hg.mozilla.org/build/buildbotcustom/rev/2cc72cabf50f to disable doing anything for fuzzing, since we aren't any worse off than we were already, and nagios will be more helpful if it's not squawking about this.

Comment 7

[Chris AtLee [:catlee]]

fuzzing didn't use --shadow before, and I'd prefer to make that option go away instead of making fuzzing use it.

Comment 8

[Chris AtLee [:catlee]]

or maybe we fix bug 640943 and have post_upload.py on this host put stuff into a different directory.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #6)
> I thought this was as simple as doing this in buildbotcustom/bin/postrun.py
> -        elif branch == 'shadow-central':
> +        elif product == 'fuzzing':
>              upload_args.append("--shadow")
> but then you get errors from post_upload.py about the buildID not being set. 
> 
> I also don't see any code in postrun.py for suppressing Pulse messages for
> fuzzing, which seems desirable, so I'm tossing this back into the pool.
> 
> In the meantime I landed
> http://hg.mozilla.org/build/buildbotcustom/rev/2cc72cabf50f to disable doing
> anything for fuzzing, since we aren't any worse off than we were already,
> and nagios will be more helpful if it's not squawking about this.

made it to production today

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

I might admit to updating buildbotcustom on the build masters at comment #6 but not doing a reconfig - only under duress though because that sort of behaviour is frowned upon. In my defence, your Honour, there were zero other changes on the production branch.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Once we have per-host config of post_upload.py (bug 640943) we can deploy a specific config for pvtbuilds2 and this can be resolved.

Comment 13

[Chris AtLee [:catlee]]

Ok, so what are the correct values to use here for pvtbuilds2?

Comment 14

[Chris AtLee [:catlee]]

Dustin, please let me know where these logs should be going on this host.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

I don't know for sure, but per comment 5, I'm guessing /mnt/pvt_builds?

Comment 16

[Chris AtLee [:catlee]]

I un-backed out nick's patch here:
http://hg.mozilla.org/build/buildbotcustom/rev/c76a4d28491c

Comment 17

[Chris AtLee [:catlee]]

Logs are now available here:
https://pvtbuilds2.dmz.scl3.mozilla.com/pub/mozilla.org/fuzzing/tinderbox-builds

Comment 18

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Chris AtLee [:catlee] from comment #17)
> Logs are now available here:
> https://pvtbuilds2.dmz.scl3.mozilla.com/pub/mozilla.org/fuzzing/tinderbox-
> builds

I can't access this URL. Does this require some special permissions?

Comment 19

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Gary Kwong [:gkw, :nth10sd] from comment #18)
> (In reply to Chris AtLee [:catlee] from comment #17)
> > Logs are now available here:
> > https://pvtbuilds2.dmz.scl3.mozilla.com/pub/mozilla.org/fuzzing/tinderbox-
> > builds
> 
> I can't access this URL. Does this require some special permissions?

Filed bug 771717 for this.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Gary Kwong [:gkw, :nth10sd] from comment #19)
> Filed bug 771717 for this.

Which was closed 44 minutes before you posted this :)

Comment 21

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Dustin J. Mitchell [:dustin] from comment #20)
> (In reply to Gary Kwong [:gkw, :nth10sd] from comment #19)
> > Filed bug 771717 for this.
> 
> Which was closed 44 minutes before you posted this :)

yep, thanks for the quick response!