OSError: [Errno 13] Permission denied: '/home/ftp' for pvtbuilds2.dmz.scl3.mozilla.com

RESOLVED FIXED

Status

Release Engineering
General Automation
P2
normal
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: armenzg, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
Now that bug 751715 got fixed we see this other issue.

INFO:util.commands:command: /builds/buildbot/build1/bin/python /builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py -r 2 -t 10 --master-name bm13-build1 --product fuzzing --platform win32 --branch idle --user ffxbld -i /home/cltbld/.ssh/ffxbld_dsa pvtbuilds2.dmz.scl3.mozilla.com /builds/buildbot/build1/master/fuzzer-win32 260
INFO:util.commands:command: stdin: <open file '/dev/null', mode 'r' at 0x5c928a0>
INFO:util.commands:command: cwd: /builds/buildbot/queue
INFO:util.commands:command: output:
Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 331, in <module>
    print ssh(user=options.user, identity=options.identity, host=host, remote_cmd=post_upload_cmd)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 40, in ssh
    return retry(do_cmd, attempts=retries+1, sleeptime=retry_sleep, args=(cmd,))
  File "/builds/buildbot/build1/tools/lib/python/util/retry.py", line 31, in retry
    return action(*args, **kwargs)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py", line 32, in do_cmd
    cmd, retcode, output))
Exception: Command ['ssh', '-l', 'ffxbld', '-i', '/home/cltbld/.ssh/ffxbld_dsa', '-p', '22', 'pvtbuilds2.dmz.scl3.mozilla.com', 'post_upload.py --tinderbox-builds-dir idle-win32 -b idle -p fuzzing --release-to-tinderbox-builds /tmp/tmp.aUtgEvzDH8 /tmp/tmp.aUtgEvzDH8/fuzzer-win32-bm13-build1-build260.txt.gz'] returned non-zero exit code 1:
Traceback (most recent call last):
  File "/usr/local/bin/post_upload.py", line 441, in <module>
    func(options, upload_dir, files)
  File "/usr/local/bin/post_upload.py", line 192, in ReleaseToTinderboxBuildsOverwrite
    ReleaseToTinderboxBuilds(options, upload_dir, files, dated=False)
  File "/usr/local/bin/post_upload.py", line 185, in ReleaseToTinderboxBuilds
    ReleaseToBuildDir(TINDERBOX_BUILDS_PATH, TINDERBOX_URL_PATH, options, upload_dir, files, dated)
  File "/usr/local/bin/post_upload.py", line 180, in ReleaseToBuildDir
    CopyFileToDir(f, upload_dir, tinderboxBuildsPath)
  File "/usr/local/bin/post_upload.py", line 53, in CopyFileToDir
    os.makedirs(full_dest_dir, 0755)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/ftp'
INFO:util.commands:command: END (31.60 elapsed)

Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 327, in <module>
    main()
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 324, in main
    post_runner.processBuild(options, build_path, request_ids)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 265, in processBuild
    log_url = self.uploadLog(build)
  File "/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/postrun.py", line 95, in uploadLog
    output = get_output(cmd, stdin=devnull)
  File "/builds/buildbot/build1/tools/lib/python/util/commands.py", line 89, in get_output
    raise e
subprocess.CalledProcessError: Command '['/builds/buildbot/build1/bin/python', '/builds/buildbot/build1/lib/python2.6/site-packages/buildbotcustom/bin/log_uploader.py', '-r', '2', '-t', '10', '--master-name', u'bm13-build1', '--product', 'fuzzing', '--platform', 'win32', '--branch', 'idle', '--user', u'ffxbld', '-i', u'/home/cltbld/.ssh/ffxbld_dsa', u'pvtbuilds2.dmz.scl3.mozilla.com', '/builds/buildbot/build1/master/fuzzer-win32', '260']' returned non-zero exit status 1

Result: 1
Assignee: server-ops-releng → dustin
(Reporter)

Comment 1

6 years ago
Hi dustin, if you think you can get to this today please let me know.
Otherwise I will downtime the nagios check until whenever you tell you expect to get to them.

e.g. [11] buildbot-master08.build.sjc1:Command Queue is CRITICAL: 2 dead items
I'll need to talk to someone who fully understands this project - I built this as described, and that seems to have not been a full description.  So I expect that fixing this problem will just reveal another.

Are you available later today?  Say, around 11:45 pacific?
(Reporter)

Comment 3

6 years ago
I am but I might enough know less than you already know.
We can fix and test until it works, maybe?

It is not a big issue it can wait until Monday if it works better.
OK, let's wait until Monday.  I think that the right solution here is to merge this into the productdelivery module that digi built for the new stage.  The current form of that server doesn't bend at the right places, but since it's going into production this weekend, I don't want to mess with it right now.

I'll also need to find someone who understands how these uploads work, so I can abstract things properly.
post_upload.py knows about two types of locations, public ones based at /home/ftp and private ones at /mnt/pvt_builds. The problem here is that we have fuzzing logs which are private, but we're not passing the right incantation to post_upload.py to tell it that. I think the simplest thing to do here is to teach the log upload that fuzzing logs are private, since that avoids re-arranging the server config and the fuzzing scripts themselves.
Assignee: dustin → nrthomas
Component: Server Operations: RelEng → Release Engineering: Automation (General)
Priority: -- → P2
QA Contact: arich → catlee
I thought this was as simple as doing this in buildbotcustom/bin/postrun.py
-        elif branch == 'shadow-central':
+        elif product == 'fuzzing':
             upload_args.append("--shadow")
but then you get errors from post_upload.py about the buildID not being set. 

I also don't see any code in postrun.py for suppressing Pulse messages for fuzzing, which seems desirable, so I'm tossing this back into the pool.

In the meantime I landed http://hg.mozilla.org/build/buildbotcustom/rev/2cc72cabf50f to disable doing anything for fuzzing, since we aren't any worse off than we were already, and nagios will be more helpful if it's not squawking about this.
Assignee: nrthomas → nobody
(Assignee)

Comment 7

6 years ago
fuzzing didn't use --shadow before, and I'd prefer to make that option go away instead of making fuzzing use it.
(Assignee)

Comment 8

6 years ago
or maybe we fix bug 640943 and have post_upload.py on this host put stuff into a different directory.
(In reply to Nick Thomas [:nthomas] from comment #6)
> I thought this was as simple as doing this in buildbotcustom/bin/postrun.py
> -        elif branch == 'shadow-central':
> +        elif product == 'fuzzing':
>              upload_args.append("--shadow")
> but then you get errors from post_upload.py about the buildID not being set. 
> 
> I also don't see any code in postrun.py for suppressing Pulse messages for
> fuzzing, which seems desirable, so I'm tossing this back into the pool.
> 
> In the meantime I landed
> http://hg.mozilla.org/build/buildbotcustom/rev/2cc72cabf50f to disable doing
> anything for fuzzing, since we aren't any worse off than we were already,
> and nagios will be more helpful if it's not squawking about this.

made it to production today
I might admit to updating buildbotcustom on the build masters at comment #6 but not doing a reconfig - only under duress though because that sort of behaviour is frowned upon. In my defence, your Honour, there were zero other changes on the production branch.
Duplicate of this bug: 739484
Once we have per-host config of post_upload.py (bug 640943) we can deploy a specific config for pvtbuilds2 and this can be resolved.
Depends on: 640943
(Assignee)

Comment 13

6 years ago
Ok, so what are the correct values to use here for pvtbuilds2?
(Assignee)

Updated

6 years ago
Assignee: nobody → catlee
(Assignee)

Comment 14

6 years ago
Dustin, please let me know where these logs should be going on this host.
Assignee: catlee → dustin
I don't know for sure, but per comment 5, I'm guessing /mnt/pvt_builds?
Assignee: dustin → catlee
(Assignee)

Updated

6 years ago
Depends on: 770638
(Assignee)

Comment 16

6 years ago
I un-backed out nick's patch here:
http://hg.mozilla.org/build/buildbotcustom/rev/c76a4d28491c
(Assignee)

Comment 17

6 years ago
Logs are now available here:
https://pvtbuilds2.dmz.scl3.mozilla.com/pub/mozilla.org/fuzzing/tinderbox-builds
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(In reply to Chris AtLee [:catlee] from comment #17)
> Logs are now available here:
> https://pvtbuilds2.dmz.scl3.mozilla.com/pub/mozilla.org/fuzzing/tinderbox-
> builds

I can't access this URL. Does this require some special permissions?
(In reply to Gary Kwong [:gkw, :nth10sd] from comment #18)
> (In reply to Chris AtLee [:catlee] from comment #17)
> > Logs are now available here:
> > https://pvtbuilds2.dmz.scl3.mozilla.com/pub/mozilla.org/fuzzing/tinderbox-
> > builds
> 
> I can't access this URL. Does this require some special permissions?

Filed bug 771717 for this.
(In reply to Gary Kwong [:gkw, :nth10sd] from comment #19)
> Filed bug 771717 for this.

Which was closed 44 minutes before you posted this :)
(In reply to Dustin J. Mitchell [:dustin] from comment #20)
> (In reply to Gary Kwong [:gkw, :nth10sd] from comment #19)
> > Filed bug 771717 for this.
> 
> Which was closed 44 minutes before you posted this :)

yep, thanks for the quick response!
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.