Closed Bug 1633077 Opened 6 years ago Closed 6 years ago

Request to Enable Single Sign On on AWS Accounts- Bill McCloskey (Mozilla) / searchfox and Account ID: 653057761566

Categories

(Webtools :: Searchfox, task)

Product:
Webtools
Component:
Searchfox
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: schu, Assigned: kats)

References

Details

Hi,

We're reaching out to Mozillians who are responsible for AWS accounts in an effort to standardize the authentication methods used for all of Mozilla's AWS accounts.

By standardizing your account and enabling Mozilla Single Sign On (SSO), you'll be able to use your existing Mozilla SSO web browser session to access AWS (no more typing your AWS password or AWS MFA token needed) and avoid storing static AWS API keys on your laptop. We will improve our ability to deprovision users when an employee leaves, gain enhanced visibility and auditability if your AWS account is compromised, reduce risk of exposure of AWS API keys and avoid manual processes.

We've created a set of instructions for AWS account owner(s) to enable federated AWS login with Single Sign On in your AWS account. The instructions can be found here. We'd like this to be completed within 2 weeks (5/8/20).

After enabling Single Sign on in your AWS account, you'll be able to access your account via the AWS Web Console, the CLI or in code using these instructions.

Once your team has used AWS with Single Sign On for a few weeks, you'll be ready to remove the existing IAM users of the Mozillians that use your AWS account. We will likely follow up with you in this ticket about the removal of those IAM users when the time comes.

If, in the process of enabling Single Sign On in your AWS account, you encounter any problems or have any questions, feel free to ask them here in this bug.

If you'd like to learn more about this project, more detail about the tech and more ways you can implement this single sign on system you can find that on this page.

Flags: needinfo?(kats)
Blocks: 1626082

Hi Sarah - I have one question (so far). Our project currently makes use of IAM roles such that when an EC2 instance is spawned, it has a role that allows it to create other EC2 instances. There is also an AWS Lambda task that (via an assigned IAM role) can spawn an EC2 instance. Will any changes be needed to that setup when migrating to SSO? Or does the SSO flow only affect (human) users?

Flags: needinfo?(kats) → needinfo?(schu)

I started going through the steps involved here:

  • Ran the identity provider cloudformation stack
  • Created a mozillians group (searchfox-aws) and added myself and a others to it as curators
  • Verified on people.mozilla.org that the mozillians group shows up in the access groups section
  • Ran the "Create IAM" role cloudformation stack with mozillians_searchfox-aws as the group
  • Did the rebuild-group-role-map thing
  • Browsed to the ?cache=false SSO url

However at this point it doesn't give me any role options. I just get a screen saying "Please select a role:" with nothing listed under it. Is there something else I need to do to make this work?

Assignee: nobody → kats

Adding in Gene and April to help answer ^^

Flags: needinfo?(gene)
Flags: needinfo?(april)

:kats I can confirm that you setup the role correctly, that you role is shown in the group role map and that you are a member of the group ( https://mozillians.org/en-US/group/searchfox-aws/ ).

I'll see if I can see in the logs why you're not getting the role offered to you in the role picker. In the meantime, do you want to try to bypass the role picker and just try a role directly?

Try visiting https://aws.sso.mozilla.com/?account=searchfox-mozilla&role=MAWS-Admin

and see if it works (that would help determine if this is a role picker problem or an access control problem)

Flags: needinfo?(kats)
Flags: needinfo?(gene)
Flags: needinfo?(april)

When I visit that link, it tries to do the federation and then comes back with Encountered error : Access denied

Flags: needinfo?(schu)
Flags: needinfo?(kats)
Flags: needinfo?(gene)

Oh, one thing that may be relevant: when I ran the CloudFormation stacks I did so using my individual account in the searchfox-mozilla AWS account, rather than using the "root credentials" for that account. I assumed that it would give me an error if it required root credentials but since everything seemed pass I figured it would be ok. Let me know if I need to use the root credentials instead.

When I visit that link, it tries to do the federation and then comes back with Encountered error : Access denied

Ok the problem is not with the role picker

Oh, one thing that may be relevant: when I ran the CloudFormation stacks I did so using my individual account in the searchfox-mozilla AWS account, rather than using the "root credentials" for that account.

Yes, that's fine, not a problem.

I'll dig into this further and look at the logs

Aha, I see the problem, you granted the group mozillians_searchfox-aws rights to the role, you meant to grant mozilliansorg_searchfox-aws

mozillians_searchfox-aws
mozilliansorg_searchfox-aws

I'd say, tear down the CloudFormation stack you deployed that provisioned the roles, and redo the steps from there (creating the roles and doing the rebuild and stuff) from that point using the correct group name.

It should work then.

Flags: needinfo?(gene) → needinfo?(kats)

Ah, I'll try that tomorrow. Note that the mana page linked in comment 0 says mozillians_ so that should be updated.

Ok, I deleted the CloudFormation stack and rebuilt it with mozilliansorg_searchfox-aws and now things work. Thanks!

Flags: needinfo?(kats)

Note that the mana page linked in comment 0 says mozillians_ so that should be updated.

:kats oh man! Thank you for catching that. I'm sorry we gave you flawed instructions (I feel bad for not catching this). Yes you did exactly what they said and the docs were wrong. Doh! I've fixed them now.

Thankfully it looks like you were the first group to encounter this problem.

Ok, I deleted the CloudFormation stack and rebuilt it with mozilliansorg_searchfox-aws and now things work. Thanks!

Excellent

No worries :)

The next problem I'm running into is this:

$ maws -o awscli
Usage: maws [OPTIONS]

Error: Invalid value for '-o' / '--output': AWS CLI is not detected on local system.

From a quick peek at the source code it looks like maws is looking for a aws executable. In my case I'm in a virtualenv with boto3 installed, so I don't have an aws executable but still need maws to dump its config into my ~/.aws folder. Is this something that needs a change in maws or should I be doing something differently?

Flags: needinfo?(gene)

I'll have April take this one

Flags: needinfo?(gene) → needinfo?(april)

maws doesn't directly touch the ~/.aws folder, simply because it relies on the aws cli to do so. If you want to output to file that looks like the aws config, it certainly can.

Alternatively, you can run pip install awscli inside your virtualenv, and things should work perfectly.

Flags: needinfo?(april)

(In reply to April King [:April] from comment #14)

If you want to output to file that looks like the aws config, it certainly can.

I'm not sure what you mean by this - are you saying there's a way for maws to output an aws config without having awscli installed?

Alternatively, you can run pip install awscli inside your virtualenv, and things should work perfectly.

I tried this, and then running $(maws -o awscli) does emit credentials to my ~/.aws/credentials file and enter the maws environment. However when I try to run python scripts that use boto3 they fail with botocore.exceptions.NoRegionError: You must specify a region.. This error happens even if I have a ~/.aws/config file with either

[default]
region = us-west-2

or

[searchfox-mozilla-MAWS-Admin]
region = us-west-2

Do you know why boto3 wouldn't be picking up the region from the config file, when I'm inside the "maws" environment? If I run maws-logout to exit the environment, and change the ~/.aws/credentials file to have [default] instead of [searchfox-mozilla-MAWS-Admin] at the top then I'm able to use my scripts with the maws-generated credentials. So the credentials themselves seem to be fine, but something in the maws environment seems to preventing boto3 from using the region specified in the ~/.aws/config.

Flags: needinfo?(april)

Hmmmm, I don't know why boto wouldn't be picking up the region set inside your credentials. I wonder if it's trying to grab something inside your virtualenv?

I have a few options:

  1. export AWS_REGION="us-west-2"
  2. use maws with --profile default, which will change the default profile.

I'm not sure what you mean by this - are you saying there's a way for maws to output an aws config without having awscli installed?

Yes, you can use -o shared, which will output to ~/.config/maws/config.ini, and you can point boto there with AWS_SHARED_CREDENTIALS_FILE as an environmental variable. That might also work inside virtualenv.

Flags: needinfo?(april)

(In reply to April King [:April] from comment #16)

  1. export AWS_REGION="us-west-2"

This didn't work. Although after some fiddling I noticed that if I just stuck region = us-west-2 in the credentials file (rather than the config file) it worked.

  1. use maws with --profile default, which will change the default profile.

This works as well. Although this approach doesn't give me the $PS1 prompt decorations that I get when I don't specify --profile. I'm waiting until tomorrow to see if the session expiry is handled reasonably; without the prompt decoration it may not be obvious from error messages that the session has expired.

I'm not sure what you mean by this - are you saying there's a way for maws to output an aws config without having awscli installed?

Yes, you can use -o shared, which will output to ~/.config/maws/config.ini, and you can point boto there with AWS_SHARED_CREDENTIALS_FILE as an environmental variable. That might also work inside virtualenv.

Ah, good to know. I might try this later but for now I think I can cobble together a reasonable workflow that's not too different from our existing workflow with the other options.

I had a couple of additional questions. One is in comment 1 above. The other is that I just want to confirm the future changes that will happen. It seems like the "User" accounts in AWS IAM will be deleted, along with any access keys attached to those user accounts. Is that correct? If so, will there be any other way of creating a long-lived access key via AWS (i.e. not through maws, but from the AWS console)? I'm imagining not, because it seems like the point of this procedure is to make it impossible to have a static long-lived access key, but I just wanted to confirm that.

Flags: needinfo?(april)

This works as well. Although this approach doesn't give me the $PS1 prompt decorations that I get when I don't specify --profile.

Good call, I've captured this bug here : https://github.com/mozilla-iam/mozilla-aws-cli/issues/215

I'm waiting until tomorrow to see if the session expiry is handled reasonably; without the prompt decoration it may not be obvious from error messages that the session has expired.

The session expiry in the prompt will work whether or not you're using a default profile name.

Kats, when choosing default, would you personally have any preference on what it displayed? default? The name of the role? Both, such as default (MAWS Admin)?

Thanks!

Flags: needinfo?(kats)

I think just the name of the role would be my preference. I don't expect to be mixing a default workflow with a non-default workflow so showing both wouldn't be useful to me personally.

Flags: needinfo?(kats)

Sounds good. Fixes for this have been submitted here: https://github.com/mozilla-iam/mozilla-aws-cli/pull/216

Flags: needinfo?(april)

Restoring needinfo for comment 17.

Flags: needinfo?(april)

I had a couple of additional questions. One is in comment 1 above. The other is that I just want to confirm the future changes that will happen. It seems like the "User" accounts in AWS IAM will be deleted, along with any access keys attached to those user accounts. Is that correct? If so, will there be any other way of creating a long-lived access key via AWS (i.e. not through maws, but from the AWS console)? I'm imagining not, because it seems like the point of this procedure is to make it impossible to have a static long-lived access key, but I just wanted to confirm that.

:gene would know the answer this a lot better than me, but I believe the answer that the long-term plan is to remove long-lived access keys for users, but leave them for services.

Flags: needinfo?(april)

Our project currently makes use of IAM roles such that when an EC2 instance is spawned, it has a role that allows it to create other EC2 instances. There is also an AWS Lambda task that (via an assigned IAM role) can spawn an EC2 instance. Will any changes be needed to that setup when migrating to SSO? Or does the SSO flow only affect (human) users?

The SSO flow only affects human users, no existing IAM Roles that are used by ec2 instances or lambda functions need be changed.

It seems like the "User" accounts in AWS IAM will be deleted, along with any access keys attached to those user accounts. Is that correct?

Yes. To be specific, the human IAM users. Any IAM users that have no password, only have API keys and are used for integrating services outside of AWS (e.g. some third party vendor doesn't integrate with AWS and so you have to create a dedicated IAM user to integrate with that service) would not be affected and should remain as they are.

If so, will there be any other way of creating a long-lived access key via AWS (i.e. not through maws, but from the AWS console)?

IAM users that have no password and exist as "service users" not human users should have long-lived access keys and would not use maws.

I'm imagining not, because it seems like the point of this procedure is to make it impossible to have a static long-lived access key, but I just wanted to confirm that.

No, static long-lived access keys are unfortunately necessary when integrating with non AWS services. After transitioning to maws there should be three types of access going on in your AWS account

  1. Human users should be using SSO login to access either the web console or to provision API keys with STS that last 12 hours
  2. AWS resources (ec2 instances, lambda functions, etc) should be using IAM Roles as (hopefully) they are already
  3. Third party services that have nothing to do with AWS and no understanding of AWS IAM Role assumption that need the ability to operate inside your AWS account should have IAM users provisioned for them that have no web console password and do have a long lived static API key pair associated with them. This long lived API key pair would then be provided to the third party company's integration.

Great, thanks for the detailed response!

kats,
Is this all setup and done, do you need anything else from us to enable it?

Flags: needinfo?(kats)

The setup is done in that searchfox maintainers/developers have working SSO-based access to the AWS web console. I was waiting on https://github.com/mozilla-iam/mozilla-aws-cli/pull/216 to be deployed so that I can update our documentation and workflow for SSH access to AWS instances. Until that happens we are still relying on a static key for SSH access.

Flags: needinfo?(kats)

Do you plan to push a new release of mozilla-aws-cli to PyPi in the near future?

Flags: needinfo?(gene)

I'll work on it tomorrow

:kats a new version the maws CLI tool has been released with the change to the prompt text

Flags: needinfo?(gene)

Confirmed, thanks!

Just a note to say that the new SSO process seems to be working fine for both web console access and our instance SSH access. I've deleted my own "static" access key from AWS IAM and everything still works.

Thanks Kartikaya

Let’s keep this bug open for now. We understand your team is now set up. Over the next 30 days, we would expect for there to be no cases of your team logging into the web console or interacting with the AWS API using their old IAM user API keys. We’ll check back in 30 days at which point the rest of the IAM users should be deleted. We’ve added Step 4 for how to delete and disable user roles here : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Federated+Login+Account+Setup#AWSFederatedLoginAccountSetup-Step4of4:Disablinganddeletinguserroles

:kats,
Now that you've been using SSO for a month, would you go in and delete the old IAM users?

Flags: needinfo?(kats)

Done. There's also an Administrators group in IAM which now has no more users. Presumably I should delete that too, can you confirm?

Flags: needinfo?(kats) → needinfo?(gene)

There's also an Administrators group in IAM which now has no more users. Presumably I should delete that too, can you confirm?

Yes, please do

Flags: needinfo?(gene)

Ok, all done. Should we close out this bug, or is there more work to be done later?

Nope, you're all set, thanks very much for taking care of this.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED

Thank you for the relatively painless migration!

See Also: → 1526029
You need to log in before you can comment on or make changes to this bug.