Closed Bug 1525127 Opened 6 years ago Closed 5 years ago

2019 Mozilla AWS Security Refresh : Update your CloudFormation stacks for nubis accounts

Categories

(Infrastructure & Operations :: SRE, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gene, Assigned: limed)

References

Details

Mozilla Enterprise Information Security (EIS) has an update to the security features we provide for your AWS account. You previously deployed the InfosecClientRoles into your account to enable EIS to perform security audits on your account and do security incident response in the case of a security breach.

We have a new CloudFormation template that we'd like to have you update to in order to

  • grant additional security auditing read permissions
  • change incident response to a role trusting a dedicated incident response AWS account
  • enable AWS GuardDuty threat detection service and wire it up to the Mozilla Defense Platform (MozDef)

There are many specifics if you're interested in the README

What we'd like you to do is update your existing CloudFormation stack

  • AWS account IDs
    • 019598067430
    • 058419420086
    • 093365119719
    • 095732026120
    • 177680776199
    • 178589013767
    • 222455217982
    • 246013728370
    • 314563910040
    • 359555865025
    • 412902012385
    • 417610946505
    • 503205249670
    • 516756624387
    • 517826968395
    • 558986605633
    • 586197220278
    • 589768463761
    • 602876482920
    • 633020870034
    • 645554471334
    • 711390850544
    • 723362035877
    • 921547910285
    • 324161760293
    • 932424332618
    • 333683252453
    • 329567179436
    • 598097830519
    • 674311274208
  • AWS region : Depends on the account
  • CloudFormation stack name : Probably opsec

with the new template. Here's how

Update your existing stack

You can either do the update in the AWS web console or on the command line with
the awscli tool. You'll be doing a CloudFormation stack update to a new template.

Update in the web console

  • Browse to the CloudFormation section

  • Select the InfosecClientRoles stack by checking the check
    circle next to it

  • In the Actions drop down in the upper right select Update Stack

  • Click the Next button

  • Enter an optional email address to receive notifications at of use of the incident
    response role

  • On the Specify stack details click the Next button

  • On the Configure stack options page click the Next button

  • On the Review page click the checkbox that says I acknowledge that AWS CloudFormation might create IAM resources.

  • Click the Update stack button

  • When the CloudFormation stack completes the creation process and the Status
    field changes from UPDATE_IN_PROGRESS to UPDATE_COMPLETE you're done.

Update on the command line

  • Set the EMAIL_ADDRESS that you'd like to receive notifications at if/when the
    incident response role is ever used. Note : EIS is always notified if the
    incident response role is ever used.
  • The STACK_NAME below is set to your existing InfosecClientRoles stack name
  • The REGION below is region in which your existing stack is deployed
EMAIL_ADDRESS=example@example.com
STACK_NAME=InfosecClientRoles
REGION=us-west-2
AWS_DEFAULT_REGION=${REGION} aws cloudformation update-stack \
  --stack-name ${STACK_NAME} \
  --template-url https://s3.amazonaws.com/public.us-west-2.infosec.mozilla.org/infosec-security-roles/cf/infosec-security-audit-incident-response-guardduty-roles-cloudformation.yml \
  --parameters ParameterKey=EmailAddress,ParameterValue=${EMAIL_ADDRESS} \
  --capabilities CAPABILITY_IAM

How do you like to be contacted?

Finally, if in the future you'd like to be contacted through a different channel
(GitHub issue, Bugzilla ticket, ServiceNow, email, etc) for this type of thing
or if there's a better person or place to make this request, do let us know.

I've added the 329567179436 consolidated billing account to the list above as it also needs an update.

I've added the 598097830519 IT Backups account

I've added the 674311274208 mozilla-0001-aws account

:limed how did this go last week? What's the current status

Flags: needinfo?(limed)

(In reply to Gene Wood [:gene] from comment #1)

I've added the 329567179436 consolidated billing account to the list above as it also needs an update.

This one should be done

Flags: needinfo?(limed)

Looking just now 329567179436 does not show up.

Flags: needinfo?(limed)

This one should be done

What process are you following to confirm that the stack update has succeeded?

:limed indicated he's targeting having the nubis code updated and rolled out to these accounts by March 12th

You can run this command with the API keys for a given account and the region set to the region in which the stack was deployed to see that stack status

aws cloudformation describe-stacks --stack-name opsec --query 'Stacks[*].[StackName, StackStatus, LastUpdatedTime]' --output text

I chatted with :limed about Comment 6. He redeployed the stack and it succeeded. He suspects he ran it previously in a different account accidentally.

I met with :limed today to follow up on Comment 8 to ensure that these would be deployed by next week (March 12th). He suggested :gozer he and I meet tomorrow to ensure this.

Ed Gozer and I met today

  • Nubis 2.4.0 release is cut
  • Gozer and Ed will deploy all the updates by March 12th
  • Ed will review the list of accounts that I've requested be updated in this ticket to verify that they are all accounts which either he or Gozer will be able to handle. Any that are not in their purview, Ed will call out in this ticket
Assignee: infra → nobody
Component: Infrastructure: Other → SRE
QA Contact: cshields → jhoward
Assignee: nobody → limed

Ed, Gozer, how did the deploys go over the last week. This was slated to be completed yesterday, can you update this ticket with the status?

Flags: needinfo?(gozer)

For the account ID's listed in comment 1 the stacks have been updated to the newest version

Flags: needinfo?(limed)
Flags: needinfo?(gozer)

Thank you

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED

Accounts

  • 093365119719
  • 417610946505
  • 674311274208

have been closed. I've removed records of the now missing IAM roles

Hey Ed,
I notice that these accounts which we previously had security roles in, continue to show as being children of consolidated billing but our roles don't work.

  • 019598067430
  • 711390850544
  • 579398555466
  • 602876482920

Any idea what happened?

Status: RESOLVED → REOPENED
Flags: needinfo?(limed)
Resolution: FIXED → ---

019598067430 - nubis-limed
711390850544 - mozilla-data-analytics
579398555466 - it-legacy-sandbox
602876482920 - nubis-gozer

All these accounts have been retired and are no longer in use

Flags: needinfo?(limed)

Got it. Would you remove these accounts from the consolidated billing AWS organization?

Flags: needinfo?(limed)

Thanks for removing these from the AWS organization

019598067430 - nubis-limed
579398555466 - it-legacy-sandbox
602876482920 - nubis-gozer

I'm assuming this account was deleted without tearing down the CloudFormation stacks in it (as the security audit and incident response role references remained)

711390850544 - mozilla-data-analytics

I've manually deleted the records of this accounts roles.

Work remaining : remove the now suspended AWS account 711390850544 - mozilla-data-analytics from the AWS organization

So the last account 711390850544 - mozilla-data-analytics we can't really remove. Most likely this is due some pending payments on the account, generally the steps to remove this account is to reopen the account (by calling aws support) and then accepting all the user agreement stuff and putting a CC on the account which seems like a lot of work for an account that is already suspended. I will also note that a suspend account will eventually get removed from the billing account after 3 months.

After discussion this with :gene he is good leaving the one account where it is.

Flags: needinfo?(limed)
Status: REOPENED → RESOLVED
Closed: 6 years ago5 years ago
Resolution: --- → FIXED
See Also: → 1627289
You need to log in before you can comment on or make changes to this bug.