Request for Infra to update Infosec security auditing IAM Role and enable CloudTrail

RESOLVED FIXED

Status

Infrastructure & Operations
Infrastructure: Other
RESOLVED FIXED
2 years ago
10 months ago

People

(Reporter: gene, Assigned: gozer)

Tracking

Details

(Reporter)

Description

2 years ago
First, thanks so much for granting Infosec (Previously called Opsec and our full name now is Enterprise Information Security) permissions to perform security audits and incident response on your AWS account earlier this year.

177680776199	Nubis Lab - 1400
230871113385	Plan B Bugzilla - 1400
248062938574	Plan B LDAP Master - 1400
329567179436	Consolidated Billing
330914478726	Plan B Okta LDAP Gateway - 1400
369987351092	IT CDN - 1400
589768463761	Nubis Market - 1400
598097830519	IT Backups - 1400
604263479206	Plan B Akamai eDNS Slave - 1400
647505682097	mozilla-sandbox - 1400

We'd like to do two things. First we'd like to have the IAM Roles that you created for us earlier this year in your account updated and secondly we'd like to have CloudTrail enabled or updated in your account and connected to our Mozilla wide secure CloudTrail storage account.

For the first part, the update of the security audit IAM role, we'd like to update the IAM role for a few reasons

* We've migrated all activities related to auditing and incident response of Mozilla accounts to a dedicated AWS account for better separation of concerns. This will improve the security around the entity that you're granting auditing and incident response permissions to.
* We've separated out security auditing and incident response permissions into two distinct roles. This will allow us to grant certain rights to certain systems but not others. Another example of improved security through separation of duties ( https://www.owasp.org/index.php/Separation_of_duties )

Here are the steps to upgrade the IAM Roles

* Delete the old CloudFormation stack that you deployed for us. This stack was probably called "opsec-security-audit-role". You may need to check a few different regions to see where you deployed it. By deleting this stack it will delete the old IAM Role which granted us permission to do security audits.
* Deploy the new CloudFormation stack by following these steps
  * Note : These steps are also outlined in the "Create a Trusting Account using CloudFormation" section here : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Security+Auditing+and+Incident+Response+Services#AWSSecurityAuditingandIncidentResponseServices-CreateaTrustingAccountusingCloudFormation
* Log into their AWS web console in in either the us-west-2 region or the us-east-1 region (the only regions that support AWS Lambda currently)
* Browse to the CloudFormation section
* Click the Create Stack button
*     In the Name field enter something like "InfosecClientRoles"
*     In the Source field select Specify an Amazon S3 template URL and type in
      https://s3.amazonaws.com/infosec-cloudformation-templates/infosec-security-audit-incident-response-roles-cloudformation.json
* Click the Next button
* Deploy the "infosec-security-audit-incident-response-roles-cloudformation.json" template
* On the Options page click the Next button
* On the Review page click the checkbox that says I acknowledge that this template might cause AWS CloudFormation to create IAM resources.
* Click the Create button
* When the CloudFormation stack completes the creation process and the Status field changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.
* Comment in this ticket letting us know the stack was created successfully.

For the second part, the secure CloudTrail storage, we'd like to have you either enable or if enabled re-configure CloudTrail to use our secure CloudTrail storage account. If the account is already using the secure CloudTrail storage, we'd still like to have it reconfigured with this new version as the old version didn't include SNS CloudTrail notifications.

CloudTrail is an AWS product which creates an audit log of all calls made to the AWS API by your account. For example, when you spin up a new ec2 instance, that call to AWS to RunInstances can be recorded with CloudTrail. This audit log is stored in S3. Here's why you should enable CloudTrail and use the AWS Secure CloudTrail Storage Account

* By enabling CloudTrail, in the event of a security incident, the audit trail created by CloudTrail will help Infosec determine what has been affected and how the account was compromised
* By using the AWS Secure CloudTrail Storage Account instead of storing the CloudTrail logs in your own account you get a few things
  * Firstly it easily allows Infosec to get access to the audit logs in the case of an incident
  * Far more importantly, by storing the logs in a separate, locked down account, in the event of an attacker gaining control of your AWS account, they will be unable to hide their tracks by deleting the CloudTrail logs after they finish. The worst they will be able to do is disable CloudTrail logging (which Infosec security auditing will detect immediately)
* By deploying this CloudFormation template, CloudTrail will be enabled for your account in *all* regions, avoiding the hassle of either manually configuring CloudTrail in every region, or deploying individual CloudTrail stacks in every region.

Here are the steps to begin using the AWS Secure CloudTrail Storage Account

Note these steps are outlined in the "Deploy the CloudFormation template" section of this page : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-DeploytheCloudFormationtemplate

1. Browse to AWS CloudFormation in either us-west-2 Oregon, or us-east-1 N. Virginia (the 2 regions that support AWS Lambda) : https://console.aws.amazon.com/cloudformation/home?region=us-west-2
2. Click "Create Stack"
3. Under "Choose a template" select "Specify an Amazon S3 template URL"
4. Enter this URL : https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage.json
5. In the "Stack name" field enter "DeployCloudTrailCloudFormationStacks" and click "Next"
6. On the "Options" screen click "Next"
7. On the "Review" screen in the "Capabilities" section, check the checkbox for "I acknowledge that this template might cause AWS CloudFormation to create IAM resources." and click "Create"

To learn how to fetch your CloudTrail logs from the secure storage account or to subscribe to notifications from CloudTrail, read about usage here :
https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-Usage
(Reporter)

Updated

2 years ago
Duplicate of this bug: 1227315
All of the above steps could be automated - given access to AWS tools and the appropriate IAM permissions - so that these changes ('remove X', 'add Y, Z, W') are deployed correctly and consistently.

Doing these individually by hand without such a tool seems like a great deal of manual labor that will need to be performed on at least 30 Mozilla AWS accounts, if not more.

Is such automation being developed?
Flags: needinfo?(gene)
(Reporter)

Comment 3

2 years ago
This is about as automated as I'm able to make it. Each service is deployed with a single cloudformation template (security auditing and cloudformation). So it's two api calls in each AWS account to enable the two services.

To the question of whether there's a way to automate this across many AWS accounts, that would depend upon a pre-existing user or role with rights to AssumeRole into each AWS account and have all the permissions needed existing across all the accounts which I don't think exists. If that would be a valuable automation tool for IT admins to have it may be something worth looking into. The security risk of having an account as powerful as that might be an issue, might not.

If you've got suggestions of ways in which something here could be more automated, I'd be happy to implement it.
Flags: needinfo?(gene)
(Reporter)

Updated

2 years ago
Duplicate of this bug: 1229903
(Reporter)

Comment 5

2 years ago
Richard, I'd like to add a clarification to these instructions which confused some other folks.

For the security auditing and incident response IAM Roles, you only need to deploy the CloudFormation template in a single region (either us-west-2 or us-east-1 but not both). This single CloudFormation stack will enable security auditing for your account in *all* regions.

The same goes for the CloudTrail CloudFormation template. It only needs to be deployed in a single region (either us-west-2 or us-east-1 but not both). The stack will in turn (on it's own) create a "sub-stack" in every region to enable CloudTrail.

So, in summary, you should be deploying in total 2 CloudFormation templates. The first to enable security auditing, the second to enable CloudTrail.
(Reporter)

Comment 6

2 years ago
Richard, do you know when this is slated to be done?
Flags: needinfo?(riweiss)
(Reporter)

Comment 7

2 years ago
I just talked to Richard and he indicated that this request should go to JD and Gozer instead of him. There are 2 accounts in the list that he owns and 1 that jake owns, so I'll open separate tickets on those. So here's the updated list of accounts that this ticket is tracking :

177680776199	Nubis Lab - 1400
230871113385	Plan B Bugzilla - 1400
589768463761	Nubis Market - 1400
604263479206	Plan B Akamai eDNS Slave - 1400
647505682097	mozilla-sandbox - 1400

JD or Gozer, would you or someone on your team be able to help with what's up in Comment 0?
Flags: needinfo?(riweiss) → needinfo?(jcrowe)

Comment 8

2 years ago
Gene,

I will put this on the list for roadmap planning we have going on next week. Can you indicate for me the level of criticality AKA does it need to be done in 60 days 90 days etc...

Thanks
Flags: needinfo?(jcrowe)
(Reporter)

Comment 9

2 years ago
Sure. The CloudTrail update should be done either by the end Q1 or by the time that you have a need to consume your CloudTrail logs and want the SNS notification to trigger that consumption. The driver for time on our side is the point at which we finish enabling MozDef to be triggered by SNS (instead of polling) for CloudTrail logs which will be this quarter.

For the IAM Role update, that's a bit more urgent as until we have that we can't both audit your accounts for security risks and emit events about those risks to MozDef for action. With the existing "v1" IAM roles that you have, we can either audit, or alert but not both (fundamental IAM limitation : http://serverfault.com/questions/696193/how-can-i-chain-aws-iam-assumerole-api-calls ). So ideally the new IAM Roles would get deployed in the next 30 days.
(Reporter)

Updated

2 years ago
Assignee: riweiss → jcrowe
(Reporter)

Comment 10

2 years ago
The "IT CDN - 1400" account looks good with the new security audit roles and CloudTrail config. 1 down.
(Reporter)

Comment 11

2 years ago
Oh, sorry, disregard Comment 10 , my records incorrectly associated this bug with the "IT CDN - 1400" account which is actually being tracked in Bug 1242001 by webops.
(Assignee)

Updated

2 years ago
Assignee: jcrowe → gozer
Status: NEW → ASSIGNED
(Assignee)

Comment 12

2 years ago
See https://github.com/nubisproject/nubis-stacks/pull/279
(Assignee)

Updated

2 years ago
Blocks: 1216784
(Assignee)

Comment 13

2 years ago
PR has been merged, will be picked up when we roll out v1.0.2
(Reporter)

Comment 14

2 years ago
Thanks so much for integrating the new roles in.

As it looks like from this ticket that the Secure CloudTrail Storage hasn't been integrated into nubis yet, I'd like to change (and simplify) the CloudTrail CloudFormation stack that you're integrating.

In this ticket originally (back in December) I'd asked you to deploy this CloudFormation template in the accounts (See Comment 0)

https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage.json

I'd like to change that to a new simpler method which takes advantage of new AWS functionality released after this ticket was created ( https://aws.amazon.com/blogs/aws/aws-cloudtrail-update-turn-on-in-all-regions-use-multiple-trails/ ).

Would you like me to request that in this ticket or should I open a new ticket?

The new template is

https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage_globally.json
Flags: needinfo?(gozer)
(Reporter)

Comment 15

2 years ago
In talking with :r2 and :jd yesterday, they confirmed that the CloudTrail config in play in nubis is still the one from September 2015.

Gozer or limed can you update it to use the new V2 cloudtrail config (in Comment 14 )?
Flags: needinfo?(limed)

Comment 16

2 years ago
Gene,

In talking to gozer, we are actually using this policy now:

https://github.com/nubisproject/nubis-deploy/tree/master/modules/opsec

Do we still need someone to update?
Flags: needinfo?(limed)
(Reporter)

Comment 17

2 years ago
:jd, That's the v2 security auditing roles which :gozer mentioned in Comment 13 . Those look great.

I'm talking about CloudTrail though (not security audit roles). The Nubis CloudTrail code that :limed made appears to live here : https://github.com/nubisproject/nubis-cloudtrail

This is the code from September 2015 (what I'm calling V1 cloudtrail).

What needs to happen is that it be replaced with the V2 cloudtrail logging that takes advantage of the new functionality AWS released in December 2015. The link to both the V1 and V2 (for comparison) CloudTrail templates is in Comment 14.

Comment 18

2 years ago
Gene,

Sorry for the confusion. I checked and we did not have that included. It has been added by:

https://github.com/nubisproject/nubis-deploy/pull/35

Can you please take a look and make sure that this is exactly what you need? We are including this in the most recent release so it will be deployed within a few days to all Nubis accounts.
Flags: needinfo?(gozer)
(Assignee)

Comment 19

2 years ago
Landed with https://github.com/nubisproject/nubis-deploy/pull/35
(Reporter)

Comment 20

2 years ago
:jd thanks for getting that change in. I'm not familiar with the syntax but we can just see if your accounts start logging to the secure storage account when it's deployed. When does this stack get deployed to the accounts in Comment 0 out of curiosity (not familiar with the release cycle)

Comment 21

2 years ago
Gene,

The answer is complicated but I will try to explain it here.

177680776199	Nubis Lab - 1400 (Next week)
230871113385	Plan B Bugzilla - 1400 (New account already created, migrating this week or early next week)
248062938574	Plan B LDAP Master - 1400 (In process of being shut down)
329567179436	Consolidated Billing (r2 is responsible for this one completely, I have no state)
330914478726	Plan B Okta LDAP Gateway - 1400 (Was shut down several weeks ago)
369987351092	IT CDN - 1400 (I don't think we maintain this one, if there is a different expectation, please let me know)
589768463761	Nubis Market - 1400 (Next week)
598097830519	IT Backups - 1400 (I don't think we maintain this one, if there is a different expectation, please let me know)
604263479206	Plan B Akamai eDNS Slave - 1400 (New account already created, migrating this week or early next week)
647505682097	mozilla-sandbox - 1400 (No upgrade scheduled, will decom in a week or two)
(Reporter)

Comment 22

2 years ago
Cool, ya I should have said the account list in Comment 7 , which looks good. 1 or 2 weeks for all of them is great, thanks.

177680776199	Nubis Lab - 1400 (Next week)
230871113385	Plan B Bugzilla - 1400 (New account already created, migrating this week or early next week)
589768463761	Nubis Market - 1400 (Next week)
604263479206	Plan B Akamai eDNS Slave - 1400 (New account already created, migrating this week or early next week)
647505682097	mozilla-sandbox - 1400 (No upgrade scheduled, will decom in a week or two)
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 23

2 years ago
:jd How's it lookin (here at the 2 week mark)? Are these 5 accounts further along towards adoption?
Flags: needinfo?(jcrowe)

Comment 24

2 years ago
Akami and Bugzilla have been provisioned in new accounts and should be working there. Akami should be cut over soon, it went to CAB this week. Bugzilla is working in the new account and the cutover is being planned.

We have been preempted by this killchain stuff and have not even touched the other three accounts, they are still on our list but I do not have an updated timeline at this point.
Flags: needinfo?(jcrowe)
(Reporter)

Comment 25

2 years ago
Got it, I hadn't put 2 and 2 together about the new accounts.

177680776199	Nubis Lab - 1400 (It's unknown when this account will have cloudtrail and the security audit roles enabled)
230871113385	Plan B Bugzilla - 1400 (314563910040 is the replacement for this account. The cutover to the new account is being planned)
589768463761	Nubis Market - 1400 (It's unknown when this account will have cloudtrail and the security audit roles enabled)
604263479206	Plan B Akamai eDNS Slave - 1400 (314563910040 is the replacement for this account. Akami should be cut over soon, it went to CAB this week.)
647505682097	mozilla-sandbox - 1400 (It's unknown when this account will be decommissioned )
(Reporter)

Comment 26

2 years ago
On May 6th, Nubis Lab (177680776199) had 2 stacks spun up with infosec security iam roles

arn:aws:cloudformation:us-west-2:177680776199:stack/opsec/344028e0-1409-11e6-b94d-50a68a201256
arn:aws:cloudformation:us-east-1:177680776199:stack/opsec/30ed0640-1409-11e6-891b-5044763e8bb3

Since this stack creates IAM Roles which are not scoped to a region, but instead are global, probably best to only create one stack in one region.

Would you tear one of them down and let me know which one remains?
(Reporter)

Updated

2 years ago
Flags: needinfo?(jcrowe)
(Assignee)

Updated

2 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 27

2 years ago
https://github.com/nubisproject/nubis-deploy/pull/65
https://github.com/nubisproject/nubis-deploy/issues/64
(Reporter)

Comment 28

2 years ago
Sweet, thanks :gozer

Here's the current status (as I know it)

177680776199	Nubis Lab - 1400
Cloudtrail is currently logging though it looks like most of the old trails were not removed when the new trail was created. There is a global trail called "opsec-cloudtrail" which is great. But there are also individual trails in each region other than us-west-2/us-east-1 that duplicate the functionality of the global cloudtrail. Those trails have names like "MozillaSecuredCloudTrail-CloudTrail-RYHL0G0I26B0"

Please delete all of the MozillaSecuredCloudTrail-CloudTrail-* Trails.

Once https://github.com/nubisproject/nubis-deploy/pull/65 is deployed we'll have a single set of security audit roles instead of a set for each region in ['us-west-2','us-east-1']


230871113385	Plan B Bugzilla - 1400
314563910040 is the replacement for this account. The cutover to the new account is being planned as of May 5th


589768463761	Nubis Market - 1400
Cloudtrail is currently logging  though it's unclear if it's enabled globally or not. Security audit roles have not yet been enabled


604263479206	Plan B Akamai eDNS Slave - 1400
314563910040 is the replacement for this account. Akami should be cut over soon, it went to CAB the week of May 5th. What was the outcome of it going to CAB week before last?


647505682097	mozilla-sandbox - 1400
It's unknown when this account will be decommissioned
(Assignee)

Comment 29

a year ago
(In reply to Gene Wood [:gene] from comment #28)
> Sweet, thanks :gozer
> 
> Here's the current status (as I know it)
> 
> 177680776199	Nubis Lab - 1400
> Cloudtrail is currently logging though it looks like most of the old trails
> were not removed when the new trail was created. There is a global trail
> called "opsec-cloudtrail" which is great. But there are also individual
> trails in each region other than us-west-2/us-east-1 that duplicate the
> functionality of the global cloudtrail. Those trails have names like
> "MozillaSecuredCloudTrail-CloudTrail-RYHL0G0I26B0"
> 
> Please delete all of the MozillaSecuredCloudTrail-CloudTrail-* Trails.

Yeah, I believe these were created with the first generation of Cloudtrail enabling cloud-formation
stuff and never got cleaned up. Working on it now.

> Once https://github.com/nubisproject/nubis-deploy/pull/65 is deployed we'll
> have a single set of security audit roles instead of a set for each region
> in ['us-west-2','us-east-1']
> 
> 
> 230871113385	Plan B Bugzilla - 1400
> 314563910040 is the replacement for this account. The cutover to the new
> account is being planned as of May 5th
> 
> 
> 589768463761	Nubis Market - 1400
> Cloudtrail is currently logging  though it's unclear if it's enabled
> globally or not. Security audit roles have not yet been enabled

That's because nubis-market is a shell (as in empty) account. We don't have EC2
instances normally deployed in there at all. So our normal account deployment stuff
hasn't been applied there yet. 
 
> 604263479206	Plan B Akamai eDNS Slave - 1400
> 314563910040 is the replacement for this account. Akami should be cut over
> soon, it went to CAB the week of May 5th. What was the outcome of it going
> to CAB week before last?

It got cutover.


> 647505682097	mozilla-sandbox - 1400
> It's unknown when this account will be decommissioned

:r2 ?
Flags: needinfo?(riweiss)
(Assignee)

Comment 30

a year ago
(In reply to Philippe M. Chiasson (:gozer) from comment #29)
> (In reply to Gene Wood [:gene] from comment #28)
> > Sweet, thanks :gozer
> > 
> > Here's the current status (as I know it)
> > 
> > 177680776199	Nubis Lab - 1400
> > Cloudtrail is currently logging though it looks like most of the old trails
> > were not removed when the new trail was created. There is a global trail
> > called "opsec-cloudtrail" which is great. But there are also individual
> > trails in each region other than us-west-2/us-east-1 that duplicate the
> > functionality of the global cloudtrail. Those trails have names like
> > "MozillaSecuredCloudTrail-CloudTrail-RYHL0G0I26B0"
> > 
> > Please delete all of the MozillaSecuredCloudTrail-CloudTrail-* Trails.

MozillaSecuredCloudTrail-CloudTrail-* Trails all deleted as of now.
(Reporter)

Comment 31

a year ago
Great, thanks :gozer!

> 589768463761	Nubis Market - 1400
> So our normal account deployment stuff hasn't been applied there yet. 

Is it planned to deploy the normal account stuff (including security auditing and cloudtrail) in this account?

> 604263479206	Plan B Akamai eDNS Slave - 1400

Is there an account decomm request in progress or is that planned?
(In reply to Gene Wood [:gene] from comment #31)
> > 604263479206	Plan B Akamai eDNS Slave - 1400
> 
> Is there an account decomm request in progress or is that planned?

Please see bug 1272487
(Reporter)

Comment 33

a year ago
Digi, perfect thank you.

Ok, current summary (as far as I can tell)

177680776199	Nubis Lab - 1400
Security audit and Cloudtrail complete

604263479206	Plan B Akamai eDNS Slave - 1400
This is being decommed in Bug 1272487

647505682097	mozilla-sandbox - 1400
Waiting to hear from :r2 when he's back on when this account is being decommed

230871113385	Plan B Bugzilla - 1400
314563910040 is the replacement for this account. The cutover to the new account is being planned as of May 5th

589768463761	Nubis Market - 1400
Account is missing normal nubis account deployment stuff including security auditing and cloudtrail
I'm working on shutting down the nubis-sandbox account. All of the users of the account, with one exception, have confirmed that their resources can be deleted. I need to open a new account for the Connected Devices team to migrate their work to before I can complete the shutdown.
Flags: needinfo?(riweiss)
(Reporter)

Comment 35

a year ago
:jd or :gozer

It looks like the Bugzilla (417610946505) and the Akamai (314563910040) accounts created infosec security audit roles back on April 19th and then again on April 20th. They were again created today at 11:47am and then at 12:30am. Are these accounts still in flux? I ask because I'd thought they were not back on the 20th so I added them to our systems but their role ARNs have changed 4 times now. Should I leave these accounts out until I hear that they're ready to avoid changing the ARNs multiple times?

Comment 36

a year ago
Hi, Sorry for the slow response, this tab got lost.

Akami should be stable. Bugzilla is not and may change again before we cut over.
Flags: needinfo?(jcrowe)
(Reporter)

Comment 37

10 months ago
Here's the updated status

604263479206	Plan B Akamai eDNS Slave - 1400
This account was closed in Bug 1272487

647505682097	mozilla-sandbox - 1400
This account was closed

230871113385	Plan B Bugzilla - 1400
Cutover to new account complete. This account was closed.

589768463761	Nubis Market - 1400
This account now has security audit roles established
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago10 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.