770990 - make ssh keys available to mock environments

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[[e-mail] (:bhearsum)]

Tried to this in bug 759975 and it ended up causing burning and deleting /home/cltbld/.ssh on the host.

Comment 1

[[e-mail] (:bhearsum)]

John, we originally tried to do this through adding this to the mock configuration:
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/home/cltbld/.ssh',
                                                              '/home/mock_mozilla/.ssh'))

Doing so ended up deleting everything from /home/cltbld/.ssh during mock-init. I've been unable to find any useful documentation on Mock bind mounting and I'd appreciate any advice you could give here. Why does mock delete the host directory? How do we bind mount .ssh without that happening?

Comment 2

[[e-mail] (:bhearsum)]

I was just rereading the Traceback we got from mock --init:
Traceback (most recent call last):
  File "/usr/sbin/mock_mozilla", line 862, in <module>
    main(retParams)
  File "/usr/sbin/mock_mozilla", line 724, in main
    chroot.init()
  File "<peak.util.decorators.rewrap wrapping mock_mozilla.backend.init at 0x00BDCCF8>", line 3, in init
  File "/usr/lib/python2.6/site-packages/mock_mozilla/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mock_mozilla/backend.py", line 245, in init
    self._init()
  File "<peak.util.decorators.rewrap wrapping mock_mozilla.backend._init at 0x00BDF2A8>", line 3, in _init
  File "/usr/lib/python2.6/site-packages/mock_mozilla/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mock_mozilla/backend.py", line 365, in _init
    self._makeBuildUser()
  File "<peak.util.decorators.rewrap wrapping mock_mozilla.backend._makeBuildUser at 0x00BE6578>", line 3, in _makeBuildUser
  File "/usr/lib/python2.6/site-packages/mock_mozilla/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mock_mozilla/backend.py", line 872, in _makeBuildUser
    mock_mozilla.util.rmtree(self.makeChrootPath(self.homedir), selinux=self.selinux)
  File "<peak.util.decorators.rewrap wrapping mock_mozilla.util.rmtree at 0x00BD2C08>", line 3, in rmtree
  File "/usr/lib/python2.6/site-packages/mock_mozilla/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.6/site-packages/mock_mozilla/util.py", line 87, in rmtree
    shutil.rmtree(path, *args, **kargs)
  File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.6/shutil.py", line 221, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib64/python2.6/shutil.py", line 219, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/builds/mock_mozilla/mozilla-f16-i386/root/home/mock_mozilla/.ssh'


I _think_ what's happening here is that the bind mounting is happening correctly, but then some other part of the mock initialization process is cleaning out mock_mozilla's home directory. And because of the bind mount, it's deleting the contents of that directory and then trying to delete the directory itself.

If that's true, we may have to see if we can mount .ssh after the initialization, or perhaps mount it elsewhere.

John, does this seem plausible to you?

Comment 3

[[e-mail] (:bhearsum)]

I tried adding --no-clean to the mock init step - it didn't help at all.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Could you add a step to copy that info in?  It's not much data!  You could bind mount it elsewhere and then copy it after the homedir is cleaned.  Or if you can run steps outside the chroot, just copy it directly from ~cltbld into $chroot/home/mock_mozilla (and adjust ownership).

Comment 5

[[e-mail] (:bhearsum)]

Yeah, maybe...I wanted to avoid adding references to specific mock environments in Buildbot code, but it may be unavoidable. I need to look into it further. Thanks for the suggestions!

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> John, we originally tried to do this through adding this to the mock
> configuration:
> config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/home/cltbld/.
> ssh',
>                                                              
> '/home/mock_mozilla/.ssh'))
> 
> Doing so ended up deleting everything from /home/cltbld/.ssh during
> mock-init. I've been unable to find any useful documentation on Mock bind
> mounting and I'd appreciate any advice you could give here. Why does mock
> delete the host directory? How do we bind mount .ssh without that happening?

jhford: you've done this before - can you help advise here?

Comment 7

[[e-mail] (:bhearsum)]

Attachment: copy ssh keys into mock root

This is the best solution I could find to the problem, and I don't really understand how it works. AFAICT, $HOME and ~/ inside of the mock root are /builds. However, 'make upload' still looks at /home/mock_mozilla. This patch gets us copying the keys in after mock --init, and we have to chown them to the correct owner because there's no way to get them that way in the first place (--unpriv doesn't work with --copyin). I used mock_mozilla to do it to avoid putting the full paths to the mock dir in buildbotcustom. HOwever, we still have the mock_mozilla username and homedir path there, which sucks.

John, if you know of a better way to do this I'd love to hear it! I don't fully understand this.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Would it be cleaner to run mock_mozilla via a wrapper script that can read those values from the mock config?  Or, and this is probably a question for jhford, maybe it's an easy modification to mock so it does this itself?

Tell me to bug off if my ideas aren't helpful..

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Well, there is nothing that says that the keys have to be in the home directory, why not bind them to something like '/ssh-keys' in the mock environment?

As an interim hack to get things going, this looks fine, but I don't think this is the right long term solution.  My original goal was to have all of the actual uploading done outside of mock, or done in mock in a way that didn't require keys.

Comment 11

[[e-mail] (:bhearsum)]

Thanks for your input here, it's much appreciated!

(In reply to John Ford [:jhford] from comment #10)
> Well, there is nothing that says that the keys have to be in the home
> directory, why not bind them to something like '/ssh-keys' in the mock
> environment?

Yeah, this is certainly an option. We've got a lot of assumptions about ssh keys being in ~/, though. I don't think moving them is a great immediate fix to the problem.

> As an interim hack to get things going, this looks fine, but I don't think
> this is the right long term solution.

Agreed.

> My original goal was to have all of
> the actual uploading done outside of mock, or done in mock in a way that
> didn't require keys.

We're not at the keyless solution yet, but is calling 'make upload' in a ShellCommand instead of a MockCommand what you mean by 'uploading done outside of mock'? That might be trivially easy to do.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #11)
> > My original goal was to have all of
> > the actual uploading done outside of mock, or done in mock in a way that
> > didn't require keys.
> 
> We're not at the keyless solution yet, but is calling 'make upload' in a
> ShellCommand instead of a MockCommand what you mean by 'uploading done
> outside of mock'? That might be trivially easy to do.

Unfortunately, this doesn't work because our build system doesn't like the host OS being switched, evidently:
make[1]: Entering directory `/builds/slave/m-cen-linux32-gecko-ntly/build/obj-firefox/b2g/installer'
mkdir -p `dirname "../../dist//b2g-16.0a1.en-US.linux-i686.checksums"`
/bin/sh: /builds/slave/m-cen-linux32-gecko-ntly/build/obj-firefox/_virtualenv/bin/python: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Ben Hearsum [:bhearsum] from comment #11)
> Yeah, this is certainly an option. We've got a lot of assumptions about ssh
> keys being in ~/, though. I don't think moving them is a great immediate fix
> to the problem.

sigh, yah, a find+grep shows that it is baked into the factory.py script pretty hard :(

> We're not at the keyless solution yet, but is calling 'make upload' in a
> ShellCommand instead of a MockCommand what you mean by 'uploading done
> outside of mock'? That might be trivially easy to do.

As you saw in comment 12, that doesn't work.  What I meant was more of 'make upload' generating a data file that described the files and other needed build system metadata that need to be uploaded but not doing the actual upload.  After the mock stuff finishes, a script would parse that data file and do the upload.  The upload script would be the part that happens outside of mock.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to John Ford [:jhford] from comment #13)
> (In reply to Ben Hearsum [:bhearsum] from comment #11)
> > Yeah, this is certainly an option. We've got a lot of assumptions about ssh
> > keys being in ~/, though. I don't think moving them is a great immediate fix
> > to the problem.
> 
> sigh, yah, a find+grep shows that it is baked into the factory.py script
> pretty hard :(

I actually looked into this more and got most of the way there by moving UPLOAD_SSH_KEY definitions to config.py. However, putting the .ssh mount elsewhere means that ~/.ssh/known_hosts is inaccessible. To workaround that we'd need to pass sshDir or knownHostsFile to MercurialBuildFactory. At this point, this option has grown to such a degree that I don't think it's worthwhile pursuing.

I think the best option for now is the existing patch, and we'll fix it up properly when we have the ability to.

Comment 15

[Chris AtLee [:catlee]]

Comment on attachment 639735 [details] [diff] [review]
copy ssh keys into mock root

Review of attachment 639735 [details] [diff] [review]:
-----------------------------------------------------------------

makes me feel dirty

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 639735 [details] [diff] [review]
copy ssh keys into mock root

(In reply to Chris AtLee [:catlee] from comment #15)
> Comment on attachment 639735 [details] [diff] [review]
> copy ssh keys into mock root
> 
> Review of attachment 639735 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> makes me feel dirty

twss


Landed this on the default branch.

Comment 17

[[e-mail] (:bhearsum)]

In production, and working! http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-central/b2g-16.0a1.en-US.linux-i686.tar.bz2