Closed Bug 1020512 Opened 10 years ago Closed 9 years ago

give slaveapi aws_manager powers

Categories

(Infrastructure & Operations :: RelOps: Puppet, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jlund, Unassigned)

References

Details

Attachments

(3 files, 2 obsolete files)

the motivation for this stems from slaveapi needing to ssh into aws-manager1 and create instances (Bug 1019732)

currently aws-manager1 uses the 'buildduty' user. this password is not known and slaveapi doesn't appear to have an ssh key generated.

This leaves two options:

1)
   a) gen an ssh key on slaveapi1 and slaveapi-dev1
   b) add those to authorized_keys on aws-manager1
   c) give slaveapi the power to ssh via ssh_key.
      ** This will be added to impl of paramiko and SSHConsole - http://mxr.mozilla.org/build/source/slaveapi/slaveapi/clients/ssh.py#46

2)
   a) find or change 'builduty' pw
   b) add that pw to our credentials.json on slaveapi1 and slaveapi-dev1

It sounds like rail, hwine, and dustin are leaning away from giving slaveapi more pw's so option (1) might be best
dustin, do I do steps 1.a and 1.b via puppet? If so, would you mind giving me some pointers?

thanks for your time in advance.
Blocks: 1019732
Flags: needinfo?(dustin)
You would generated it by hand, and then add the private and public versions to the hosts using puppet.  The private key should be in hiera, while the public key should probably be in modules/ssh/manifests/keys.pp.

It'd be great if aws-manager1 limited connections with that key to the slaveapi hosts.
OK so I have generated some keys manually on slaveapi-dev1.

There are some unknowns about the next parts:

1) the 'user' that slaveapi uses is 'cltbld'. adding 'cltbld' to modules/ssh/manifests/keys.pp doesn't sound like the right thing to do here. modules/ssh/manifests/keys.pp just looks like a user list without any names that represent services (like slaveapi-dev1).

2) looking at https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Secrets#Using_EYAML I see:

eyaml encrypt --pkcs7-private-key /etc/hiera/keys/private_key.pem --pkcs7-public-key /etc/hiera/keys/public_key.pem \
    --output examples --password -l 'foo'

where I could replace foo with e.g. slaveapi-dev1_ssh_rsa

but that looking at that above cmd, it seems like it would generate a yaml prop. Not sure how to take the id_rsa priv key I generated by hand

3) should I be creating an environment file like: /etc/hiera/environments/jlund_secrets.eyaml and copy/paste there or use the '/etc/hiera/secrets.eyaml' file?



Would you mind giving me 5min just to go over how these things work and what I should do? Maybe over vidyo? I just want to make sure I do things right. I am going to keep plucking away at other parts of slaveapi in the mean time.

leaving needinfo open from comment 1
Assignee: nobody → relops
Component: Tools → RelOps: Puppet
Product: Release Engineering → Infrastructure & Operations
QA Contact: hwine → dustin
Summary: add ability for slaveapi to ssh into machines via ssh_key → give slaveapi aws_manager powers
met with dustin and rail. concluded that giving slaveapi local ability to create aws instances is the best course of action.

I had some time today and yesterday to start implementing this. patch incoming
Flags: needinfo?(dustin)
bear in mind this is my first puppet patch.

lot's of unknowns here but this should be a start
Attachment #8439561 - Flags: feedback?(rail)
adds users and basedir ref
Attachment #8439561 - Attachment is obsolete: true
Attachment #8439561 - Flags: feedback?(rail)
Attachment #8439565 - Flags: feedback?(rail)
Comment on attachment 8439565 [details] [diff] [review]
140612_bug_020512_give_slaveapi_aws_powers-DRAFT-2.patch

Review of attachment 8439565 [details] [diff] [review]:
-----------------------------------------------------------------

see inline comments for the unknowns.

also, WRT to testing, rail I think you mentioned having user enviroments. Should I create a jlund on a puppet master and test the patch when it's approved? This would be via: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment

::: modules/slaveapi/manifests/aws.pp
@@ +4,5 @@
> +
> +    # use the slaveapi user - cltbld instead of buildduty
> +    $user = $users::builder::username
> +    $group = $users::builder::group
> +    $home = $users::builder::home

So, rather than slaveapi having to switch users to do aws_manager stuff, I figured it could all go under cltbld. I am not sure if there are going to be issues with secrets.

@@ +10,5 @@
> +    $basedir = "${slaveapi::base::root}/${title}"
> +    $aws_dst = "${basedir}/aws"
> +    $cloud_tools_dst = "${aws_dst}/aws/cloud-tools"
> +    $secrets_dst = "${aws_dst}/aws/secrets"
> +    $aws_bin_dst = "${aws_dst}/aws/bin"

I figured we would keep this in /builds/slaveapi/{dev, prod}/ but encapsulate it in its own dir 'aws'

@@ +24,5 @@
> +            ensure => directory,
> +            mode => 0755,
> +            owner => $user,
> +            group => $group,
> +            require => File[$aws_dst];

There are many secret files but lot's of them appear to be related to stop_idle, watch_pending, etc.

We need aws_create_instances.py, free_ips.py, and aws_manage_instnances.py functionality. I don't think we need invtool as we have direct password access to inventory. I need to confirm with uberj if I can use api calls to do PUTs and POSTs.

So, IIUC, we only need the boto, and aws-ssh-key. Not 100% on cached_certs.

@@ +45,5 @@
> +            mode      => 0600,
> +            owner     => $user,
> +            group     => $group,
> +            show_diff => false,
> +            content   => secret("aws_manager_ssh_key");

can slaveapi->cltbld share aws_manager->buildduty's ssh key or do I need to generate a new one and add it via hiera?

@@ +66,5 @@
> +    file {
> +        "/etc/cron.d/aws-manager-update-hg-clone":
> +            content => "*/5 * * * * ${user} cd ${cloud_tools_dst} && ${packages::mozilla::py27_mercurial::mercurial} pull -u\n";
> +        "/etc/cron.d/aws-manager-delete-old-certs":
> +            content => "@daily find ${secrets_dst}/cached_certs -type f -mtime +30 -delete\n";

again not sure about cached_certs but we will for sure want the cron that pulls cloud-tools.

all the other crontasks I am not mentioning here I think we can leave out?

::: modules/slaveapi/manifests/base.pp
@@ +17,5 @@
> +    # XXX let's put the aws_manager stuff here first then extract it out
> +    include packages::mercurial
> +    $root = $::config::aws_manager_root
> +    $cloud_tools_dst = "${root}/cloud-tools"
> +    $secrets_dir = "${root}/secrets"

I think this whole chunk up to the XXX can be ripped out. it's old code

::: modules/slaveapi/manifests/instance.pp
@@ +12,5 @@
>      $config_file = "${basedir}/slaveapi.ini"
>      if (has_aspect("dev")) {
>          $bugzilla_url = $::config::slaveapi_bugzilla_dev_url
> +        # let's test the aws_manager stuff on dev first
> +        include slaveapi::aws

not sure if this is good practice but I'd like to be able to refine this patch until I get the expected behavior (i.e. create aws instances manually)

@@ +70,5 @@
> +                "boto==2.27.0",
> +                "iso8601==0.1.10",
> +                "repoze.lru==0.6",
> +                "ssh==1.8.0",
> +                "wsgiref==0.1.2",

rather than creating a new virtualenv, I thought it be easier (better?) to put it all under slaveapi. I don't want to pollute the packages here though so I didn't include all the ones from aws_manager:
    "MySQL-python==1.2.3",
    "SQLAlchemy==0.8.3",
    "ecdsa==0.10",
    "paramiko==1.12.0", # slaveapi already has
    "pycrypto==2.6.1", # slaveapi already has
    "requests==2.0.1",  # slaveapi has but much older
    "simplejson==3.3.1",


also note that I am not including the following from aws_manager either. Not sure though?:
    include packages::make
    include packages::mozilla::py27_virtualenv
    include packages::mozilla::py27_mercurial

::: modules/slaveapi/templates/dot_boto.erb
@@ +1,3 @@
> +[Credentials]
> +aws_access_key_id = <%= scope.function_secret(["aws_manager_access_key_id"]) %>
> +aws_secret_access_key = <%= scope.function_secret(["aws_manager_access_key"]) %>

again, not sure if I can use these secrets that are already generated in hiera.

rail, if I need to create any secrets, rail could you guide me from the AWS end through to puppet?
Comment on attachment 8439565 [details] [diff] [review]
140612_bug_020512_give_slaveapi_aws_powers-DRAFT-2.patch

Review of attachment 8439565 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/slaveapi/manifests/aws.pp
@@ +6,5 @@
> +    $user = $users::builder::username
> +    $group = $users::builder::group
> +    $home = $users::builder::home
> +
> +    $basedir = "${slaveapi::base::root}/${title}"

What does $title stand for here?

@@ +21,5 @@
> +            owner => $user,
> +            group => $group;
> +        "${secrets_dst}":
> +            ensure => directory,
> +            mode => 0755,

0700, please :)

@@ +24,5 @@
> +            ensure => directory,
> +            mode => 0755,
> +            owner => $user,
> +            group => $group,
> +            require => File[$aws_dst];

cached_certs is dead now, don't worry about it.
You can also drop redundant "require => File[$aws_dst]" - puppet handles that for you for dirs/files.

@@ +25,5 @@
> +            mode => 0755,
> +            owner => $user,
> +            group => $group,
> +            require => File[$aws_dst];
> +        "${secrets_dst}/cached_certs":

No need for this. Kill it.

@@ +45,5 @@
> +            mode      => 0600,
> +            owner     => $user,
> +            group     => $group,
> +            show_diff => false,
> +            content   => secret("aws_manager_ssh_key");

It's OK to reuse the same one. This key is used only when you create an instance (to ssh into it and run puppetize.sh) and it's wiped by puppet.

@@ +52,5 @@
> +    # cloud-tools repo
> +    mercurial::repo {
> +        "cloud-tools-${cloud_tools_dst}":
> +            hg_repo => "${config::cloud_tools_hg_repo}",
> +            dst_dir => cloud_tools_dst,

$cloud_tools_dst (you missed $)

@@ +63,5 @@
> +    }
> +
> +    # cron tasks
> +    file {
> +        "/etc/cron.d/aws-manager-update-hg-clone":

can you use another name for this file? just to avoid mental conflicts. :)

@@ +66,5 @@
> +    file {
> +        "/etc/cron.d/aws-manager-update-hg-clone":
> +            content => "*/5 * * * * ${user} cd ${cloud_tools_dst} && ${packages::mozilla::py27_mercurial::mercurial} pull -u\n";
> +        "/etc/cron.d/aws-manager-delete-old-certs":
> +            content => "@daily find ${secrets_dst}/cached_certs -type f -mtime +30 -delete\n";

Kill it, we don't use it anymore.

::: modules/slaveapi/manifests/base.pp
@@ +17,5 @@
> +    # XXX let's put the aws_manager stuff here first then extract it out
> +    include packages::mercurial
> +    $root = $::config::aws_manager_root
> +    $cloud_tools_dst = "${root}/cloud-tools"
> +    $secrets_dir = "${root}/secrets"

yes, this chunk can go away

::: modules/slaveapi/manifests/instance.pp
@@ +70,5 @@
> +                "boto==2.27.0",
> +                "iso8601==0.1.10",
> +                "repoze.lru==0.6",
> +                "ssh==1.8.0",
> +                "wsgiref==0.1.2",

Maybe it's worth to isolate the environment, esp if there are conflicting packages/versions.

You need make to install MySQL-python, IIRC.

::: modules/slaveapi/templates/dot_boto.erb
@@ +1,3 @@
> +[Credentials]
> +aws_access_key_id = <%= scope.function_secret(["aws_manager_access_key_id"]) %>
> +aws_secret_access_key = <%= scope.function_secret(["aws_manager_access_key"]) %>

I think it'd be better to use a separate key for slaveapi, so we can distinguish them in the AWS logs.
Attachment #8439565 - Flags: feedback?(rail) → feedback+
(In reply to Rail Aliiev [:rail] from comment #8)
> Comment on attachment 8439565 [details] [diff] [review]
> 140612_bug_020512_give_slaveapi_aws_powers-DRAFT-2.patch
> 
> Review of attachment 8439565 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/slaveapi/manifests/aws.pp
> @@ +6,5 @@
> > +    $user = $users::builder::username
> > +    $group = $users::builder::group
> > +    $home = $users::builder::home
> > +
> > +    $basedir = "${slaveapi::base::root}/${title}"
> 
> What does $title stand for here?

title is set on the slaveapi instance:
   this -> http://mxr.mozilla.org/build/source/puppet/modules/toplevel/manifests/server/slaveapi.pp#14
   instantiates -> http://mxr.mozilla.org/build/source/puppet/modules/slaveapi/manifests/instance.pp#5

I figured $title would be set as a global similarly like $module_name. If you're saying I can't reach $title from within aws.pp, what's the puppet idiom to pass that value?

> 
> @@ +21,5 @@
> > +            owner => $user,
> > +            group => $group;
> > +        "${secrets_dst}":
> > +            ensure => directory,
> > +            mode => 0755,
> 
> 0700, please :)

will fix


> You can also drop redundant "require => File[$aws_dst]" - puppet handles
> that for you for dirs/files.

oh, sorry, google was only able to tell me that puppet won't behave like 'mkdir -p' and will fail if parent dir is not present.

ahh, maybe puppet will look for the `File` declaration that matches a non existent parent dir of another `File` declaration.

> 
> @@ +25,5 @@
> > +            mode => 0755,
> > +            owner => $user,
> > +            group => $group,
> > +            require => File[$aws_dst];
> > +        "${secrets_dst}/cached_certs":
> 
> No need for this. Kill it.

/me sharpens axe

> 
> @@ +45,5 @@
> > +            mode      => 0600,
> > +            owner     => $user,
> > +            group     => $group,
> > +            show_diff => false,
> > +            content   => secret("aws_manager_ssh_key");
> 
> It's OK to reuse the same one. This key is used only when you create an
> instance (to ssh into it and run puppetize.sh) and it's wiped by puppet.

gotcha!

> 
> @@ +52,5 @@
> > +    # cloud-tools repo
> > +    mercurial::repo {
> > +        "cloud-tools-${cloud_tools_dst}":
> > +            hg_repo => "${config::cloud_tools_hg_repo}",
> > +            dst_dir => cloud_tools_dst,
> 
> $cloud_tools_dst (you missed $)

doh!

> @@ +63,5 @@
> > +    }
> > +
> > +    # cron tasks
> > +    file {
> > +        "/etc/cron.d/aws-manager-update-hg-clone":
> 
> can you use another name for this file? just to avoid mental conflicts. :)

will change to -> slaveapi-update-hg-cloud-tools


> ::: modules/slaveapi/manifests/instance.pp
> @@ +70,5 @@
> > +                "boto==2.27.0",
> > +                "iso8601==0.1.10",
> > +                "repoze.lru==0.6",
> > +                "ssh==1.8.0",
> > +                "wsgiref==0.1.2",
> 
> Maybe it's worth to isolate the environment, esp if there are conflicting
> packages/versions.

I was thinking that but I thought we could try one first. requests was the only thing that differed more than a x.y.Z version. it would be nice for slaveapi setup.py represent all the modules needed, and also not have to worry about certain slaveapi end-points needing to run in their own env. Might not be able to avoid that forever though.

> 
> You need make to install MySQL-python, IIRC.

ok so since I don't need mysql-python, I don't need make.

> 
> ::: modules/slaveapi/templates/dot_boto.erb
> @@ +1,3 @@
> > +[Credentials]
> > +aws_access_key_id = <%= scope.function_secret(["aws_manager_access_key_id"]) %>
> > +aws_secret_access_key = <%= scope.function_secret(["aws_manager_access_key"]) %>
> 
> I think it'd be better to use a separate key for slaveapi, so we can
> distinguish them in the AWS logs.

sounds good. so step 1) sync up with you and get a keys from AWS tools 2) add it via hiera.

rail - when would it suit you to gen keys with me? I'm assuming I need you or catlee to do it.
Flags: needinfo?(rail)
(In reply to Jordan Lund (:jlund) from comment #9)
> title is set on the slaveapi instance:
>    this ->
> http://mxr.mozilla.org/build/source/puppet/modules/toplevel/manifests/server/
> slaveapi.pp#14
>    instantiates ->
> http://mxr.mozilla.org/build/source/puppet/modules/slaveapi/manifests/
> instance.pp#5
> 
> I figured $title would be set as a global similarly like $module_name. If
> you're saying I can't reach $title from within aws.pp, what's the puppet
> idiom to pass that value?

That's $slaveapi_title, $toplevel::server::slaveapi::slaveapi_title rather. I'm not sure what would be the best way to access that or even use it. It doesn't sound right to include toplevel::server::slaveapi somewhere in your patch...

BTW, I'd avoid using $title variable, it's magic in puppet. See http://docs.puppetlabs.com/learning/definedtypes.html



> rail - when would it suit you to gen keys with me? I'm assuming I need you
> or catlee to do it.

passwords/slaveapi-aws.txt.gpg pushed to our secrets git repo
Flags: needinfo?(rail)
(In reply to Rail Aliiev [:rail] from comment #10)
> (In reply to Jordan Lund (:jlund) from comment #9)
> > title is set on the slaveapi instance:
> >    this ->
> > http://mxr.mozilla.org/build/source/puppet/modules/toplevel/manifests/server/
> > slaveapi.pp#14
> >    instantiates ->
> > http://mxr.mozilla.org/build/source/puppet/modules/slaveapi/manifests/
> > instance.pp#5
> > 
> > I figured $title would be set as a global similarly like $module_name. If
> > you're saying I can't reach $title from within aws.pp, what's the puppet
> > idiom to pass that value?
> 
> That's $slaveapi_title, $toplevel::server::slaveapi::slaveapi_title rather.
> I'm not sure what would be the best way to access that or even use it. It
> doesn't sound right to include toplevel::server::slaveapi somewhere in your
> patch...
> 
> BTW, I'd avoid using $title variable, it's magic in puppet. See
> http://docs.puppetlabs.com/learning/definedtypes.html
> 


hmm, so I meant, and this is how I understand it, $slaveapi_title is passed as the $title to the definedtype of the definition instance. You can see ben use it here: http://mxr.mozilla.org/build/source/puppet/modules/slaveapi/manifests/instance.pp#10

Is this logic false? If I'm right, I'm still not sure if classes I 'include' within instance.pp will also be passed the current $title
interdiff from first draft and rails feedback

complete diff incoming
On buildduty so progress on this is slower.

See previous comment for an interdiff of changes since last time.

Note, I added the boto pw's for slaveapi you created to hiera.

I'm guessing the idea here, once I get an r+, the 'safe' thing to do would be clone this patch into my puppet env and 'puppet agenst --test --environment=jlund' on the dev node of slaveapi?
Attachment #8439565 - Attachment is obsolete: true
Attachment #8441113 - Flags: review?(rail)
I'd encourage to test the manifest even before you submit them for review. I usually start with 

puppet agent --test --noop --server releng-puppet2.srv.releng.scl3.mozilla.com --environment $user

to see if everything goes as expected - you can catch syntax/logic errors.  

More on this at https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/HowTo/Set_up_a_user_environment
Comment on attachment 8441113 [details] [diff] [review]
140616_bug_020512_give_slaveapi_aws_powers.patch

Review of attachment 8441113 [details] [diff] [review]:
-----------------------------------------------------------------

In overall looks good, just one thing to test:

::: modules/slaveapi/manifests/instance.pp
@@ +16,5 @@
> +        class {
> +            "slaveapi::aws":
> +                slaveapi_title => $title,
> +        }
> +        include slaveapi::aws

I'm not sure if this will work... Can you test it?
(In reply to Rail Aliiev [:rail] from comment #15)
> Comment on attachment 8441113 [details] [diff] [review]
> 140616_bug_020512_give_slaveapi_aws_powers.patch
> 
> Review of attachment 8441113 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> In overall looks good, just one thing to test:
> 
> ::: modules/slaveapi/manifests/instance.pp
> @@ +16,5 @@
> > +        class {
> > +            "slaveapi::aws":
> > +                slaveapi_title => $title,
> > +        }
> > +        include slaveapi::aws
> 
> I'm not sure if this will work... Can you test it?

oops, I don't think I need the include line. I think the class call above acts as an instantiation. Either way, I'll test this out first. Testing this on a node before submitting for a review would make this a lot easier as I can catch the small mistakes!
Comment on attachment 8441113 [details] [diff] [review]
140616_bug_020512_give_slaveapi_aws_powers.patch

Resetting the "r" flag for now. Please reset it back once you get the patch tested.
Attachment #8441113 - Flags: review?(rail)
=== status update

my patch applied, added  a few tweaks and I am currently able to start the aws_create_instance script but I run into boto errors even though ~/.boto exists and crentials are present

error log: https://pastebin.mozilla.org/5466887

mtg with rail today to debug
I added 2 missing actions ("iam:AddRoleToInstanceProfile", "iam:PassRole") to slaveapi user permissions. Hopefully it works.
perfect. Those permissions must have been the issue. I was able to successfully create and puppetize tst-linux64-ec2-jlund.test.releng.use1.mozilla.com

patch incoming
hmm, so sanity check: I did get a prompt motd that IIRC, I don't normally see:

REBOOT REQUIRED: http://mxr.mozilla.org/build/source/puppet/modules/needs_reboot/manifests/motd.pp#6

aws_create_instance log:
889 [10.134.156.249] run: chmod 755 /root/puppetize.sh
890 [10.134.156.249] put: <file obj> -> /root/deploypass
891 [10.134.156.249] put: <file obj> -> /root/post-puppetize-hook.sh
892 [10.134.156.249] run: PUPPET_SERVER=releng-puppet2.srv.releng.use1.mozilla.com /root/puppetize.sh
893 [10.134.156.249] out: Puppetize output is in /root/puppetize.log
894 [10.134.156.249] out:
896 [10.134.156.249] run: sync
897 [10.134.156.249] run: sync
898 [10.134.156.249] run: reboot


I'm guessing that puppetize.sh did not work and a reboot is required so that the created instance will try to run puppet again.

rebooting now to see results



looking at /root/puppetize.log:
Contacting puppet server releng-puppet2.srv.releng.use1.mozilla.com
25 Jun 18:57:35 ntpdate[2807]: adjust time server 23.92.26.57 offset 0.061570 sec
Certificate request for tst-linux64-ec2-jlund.test.releng.use1.mozilla.com
securely removing deploypass
Running puppet agent against server 'releng-puppet2.srv.releng.use1.mozilla.com'
Puppet run failed; re-trying after 10m
Running puppet agent against server 'releng-puppet2.srv.releng.use1.mozilla.com'
Sourcing /root/post-puppetize-hook.sh
/var/log/boot.log reveals some hints:

puppet-agent[1052]: Could not request certificate: Error 400 on SERVER: this master is not a CA
Jun 25 12:29:24 ip-10-134-156-249 puppet-agent[1052]: Could not request certificate: Error 400 on SERVER: this master is not a CA
correction that was syslog
this seems to be a new issue that was happening in aws-manager1 as well.

1) rail applies patch: http://hg.mozilla.org/build/cloud-tools/rev/40038800c435
2) rails fixes all the things

Everything seems good, patch incoming again.
interdiff since last patch: https://pastebin.mozilla.org/5471787

this patch allowed me to successfully create/terminate an aws instance.

I have adjusted the patch to apply to both slaveapi-dev and slaveapi1 nodes at the same time as I went the route of testing on slaveapi-dev already with this patch via my puppet env.
Attachment #8446166 - Flags: review?(catlee)
Comment on attachment 8446166 [details] [diff] [review]
140625_bug_020512_give_slaveapi_aws_powers.patch

passing off to dustin since rail and catlee are on PTO. 

Dustin, you might have some context here since we spoke of doing this a few weeks ago and you also reviewed 'Bug 943134 - Puppetize a dedicated VM for AWS related management tools'.

thanks!
Attachment #8446166 - Flags: review?(catlee) → review?(dustin)
Comment on attachment 8446166 [details] [diff] [review]
140625_bug_020512_give_slaveapi_aws_powers.patch

Review of attachment 8446166 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/slaveapi/manifests/aws.pp
@@ +1,1 @@
> +class slaveapi::aws ($slaveapi_title='prod') {

Could you call this $environment?  The variable name confused me for a bit.
Attachment #8446166 - Flags: review?(dustin) → review+
Comment on attachment 8446166 [details] [diff] [review]
140625_bug_020512_give_slaveapi_aws_powers.patch

with: s/slaveapi_title/environment/

https://hg.mozilla.org/build/puppet/rev/d21fe100ff24
Attachment #8446166 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
this is broken for any cloud-tools rev after https://github.com/mozilla/build-cloud-tools/commit/8da5906002d13dbad7f9cbb37fb5874d1ca4afdf
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Depends on: 1128620
this was resolved on April 1st. somehow I missed to close this

http://hg.mozilla.org/build/puppet/rev/90668fc6e5b5
Status: REOPENED → RESOLVED
Closed: 10 years ago9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: