987537 - Figure where to put sccache config for ceph

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Hommey [:glandium]]

While on AWS the S3 end point and access keys are extracted from the instance metadata, we don't have such information to access ceph on in-house build slaves.

We need a way to put at least the ceph access credentials somewhere that is not the firefox source tree. Essentially, this comes in the form of an ini file.

If the tooltool repository is never to be made public, this data could be put there. This would likely be more flexible than putting it on the buildbot side. But is the tooltool repository always going to stay private?

Comment 1

[Rail Aliiev [:rail]]

We can use Puppet to deploy the those, I believe. Just like we deploy Google API key:

http://hg.mozilla.org/build/puppet/file/7fe7ec1ff58c/modules/slave_secrets/manifests/init.pp#l41
http://hg.mozilla.org/build/puppet/file/7fe7ec1ff58c/modules/slave_secrets/manifests/google_api_key.pp

Will this work for you?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

FWIW, the only active tooltool repo is private to the releng network, although we have the capacity to create public repos as well.  But that level of privacy is not suitable for storing credentials.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

(Rail's absolutely right -- I'm just providing some "FYI" information)

Comment 4

[Michael Shal [:mshal]]

Attachment: ceph

I added the keys to eyaml and tested this out on a linux builder to verify that it creates the ~/.boto file correctly.

I'm not sure how this might interact with the ~/.boto file created for AWS instances (modules/aws_manager/manifests/secrets.pp), so if this isn't the right place to put it for in-house machines let me know.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8403531 [details] [diff] [review]
ceph

This looks good, but yes, I'm worried about a .boto conflict, too.  Is that other .boto not managed by puppet?

Comment 6

[Michael Shal [:mshal]]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> This looks good, but yes, I'm worried about a .boto conflict, too.  Is that
> other .boto not managed by puppet?

I think it is by the file I mentioned - in puppet/modules/aws_manager/manifests/secrets.pp there's a similar block and a slightly different dot_boto.erb file:

        "${users::buildduty::home}/.boto":
            mode      => 0600,
            owner     => "${users::buildduty::username}",
            group     => "${users::buildduty::group}",
            show_diff => false,
            content   => template("$module_name/dot_boto.erb");

I guess if I can run the puppet command on a aws instance against my environment successfully we'd be good to go?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, OK, so that's for aws_manager, which is not a class that's applied to slaves, and vice versa.  So you're fine.  We'll probably need to sort this out more generically later, but for now this will be fine.

Comment 8

[Michael Shal [:mshal]]

Comment on attachment 8403531 [details] [diff] [review]
ceph

https://hg.mozilla.org/build/puppet/rev/d649e2bf676a

Comment 9

[Michael Shal [:mshal]]

Attachment: ceph-ensure

Properly handle the 'ensure' attribute per Callek.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8403639 [details] [diff] [review]
ceph-ensure

Review of attachment 8403639 [details] [diff] [review]:
-----------------------------------------------------------------

Minor nits, but when fixed this looks good.

::: modules/slave_secrets/manifests/ceph_config.pp
@@ +4,5 @@
> +
> +class slave_secrets::ceph_config($ensure=present) {
> +    if ($ensure == 'present') {
> +        file {
> +            "${users::builder::home}/.boto":

For proper form, this class should include users::builder, lest these variables end up empty and installing secrets in /.boto.

::: modules/slave_secrets/manifests/init.pp
@@ +51,5 @@
>      # install ceph credentials on build slaves
>      if ($slave_type == 'build') {
> +        class {
> +            'slave_secrets::ceph_config':
> +                ensure => $ensure;

Indentation is wrong here

Comment 11

[Michael Shal [:mshal]]

Attachment: ceph-ensure

Updated with review feedback, r+ carried forward.

Comment 12

[Michael Shal [:mshal]]

Comment on attachment 8403995 [details] [diff] [review]
ceph-ensure

https://hg.mozilla.org/build/puppet/rev/7b2ec4361398

Comment 13

[Michael Shal [:mshal]]

Attachment: ceph-cfg

Now configured only for moco.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8404181 [details] [diff] [review]
ceph-cfg

Review of attachment 8404181 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/config/manifests/base.pp
@@ +146,5 @@
>      # the username under which all building and testing operations take place
>      $builder_username = 'cltbld'
>      # true if secret("google_api_key") should be installed at /builds/gapi.key
>      $install_google_api_key = false
> +    # true if ~/.boto should be installed with ceph keys

# true if ceph access keys should be installed on build slaves

Comment 15

[Michael Shal [:mshal]]

Attachment: ceph-cfg

Updated patch with review feedback, r+ carried forward.

Comment 16

[Michael Shal [:mshal]]

Comment on attachment 8404193 [details] [diff] [review]
ceph-cfg

https://hg.mozilla.org/build/puppet/rev/4a4d2e7da850

Comment 17

[Michael Shal [:mshal]]

Attachment: buildbot-configs-boto.patch

buildbot-configs changes to copy .boto into the mock environment.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8404356 [details] [diff] [review]
buildbot-configs-boto.patch

Looks like you collected all the set.

Comment 19

[Michael Shal [:mshal]]

Comment on attachment 8404356 [details] [diff] [review]
buildbot-configs-boto.patch

https://hg.mozilla.org/build/buildbot-configs/rev/bbece32f3105

Comment 20

[Michael Shal [:mshal]]

Seems to be deployed properly now.