Closed Bug 1232086 Opened 9 years ago Closed 8 years ago

Request for Infra to update Infosec security auditing IAM Role and enable CloudTrail

Categories

(Infrastructure & Operations :: Infrastructure: Other, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gene, Assigned: gozer)

References

Details

First, thanks so much for granting Infosec (Previously called Opsec and our full name now is Enterprise Information Security) permissions to perform security audits and incident response on your AWS account earlier this year. 177680776199 Nubis Lab - 1400 230871113385 Plan B Bugzilla - 1400 248062938574 Plan B LDAP Master - 1400 329567179436 Consolidated Billing 330914478726 Plan B Okta LDAP Gateway - 1400 369987351092 IT CDN - 1400 589768463761 Nubis Market - 1400 598097830519 IT Backups - 1400 604263479206 Plan B Akamai eDNS Slave - 1400 647505682097 mozilla-sandbox - 1400 We'd like to do two things. First we'd like to have the IAM Roles that you created for us earlier this year in your account updated and secondly we'd like to have CloudTrail enabled or updated in your account and connected to our Mozilla wide secure CloudTrail storage account. For the first part, the update of the security audit IAM role, we'd like to update the IAM role for a few reasons * We've migrated all activities related to auditing and incident response of Mozilla accounts to a dedicated AWS account for better separation of concerns. This will improve the security around the entity that you're granting auditing and incident response permissions to. * We've separated out security auditing and incident response permissions into two distinct roles. This will allow us to grant certain rights to certain systems but not others. Another example of improved security through separation of duties ( https://www.owasp.org/index.php/Separation_of_duties ) Here are the steps to upgrade the IAM Roles * Delete the old CloudFormation stack that you deployed for us. This stack was probably called "opsec-security-audit-role". You may need to check a few different regions to see where you deployed it. By deleting this stack it will delete the old IAM Role which granted us permission to do security audits. * Deploy the new CloudFormation stack by following these steps * Note : These steps are also outlined in the "Create a Trusting Account using CloudFormation" section here : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Security+Auditing+and+Incident+Response+Services#AWSSecurityAuditingandIncidentResponseServices-CreateaTrustingAccountusingCloudFormation * Log into their AWS web console in in either the us-west-2 region or the us-east-1 region (the only regions that support AWS Lambda currently) * Browse to the CloudFormation section * Click the Create Stack button * In the Name field enter something like "InfosecClientRoles" * In the Source field select Specify an Amazon S3 template URL and type in https://s3.amazonaws.com/infosec-cloudformation-templates/infosec-security-audit-incident-response-roles-cloudformation.json * Click the Next button * Deploy the "infosec-security-audit-incident-response-roles-cloudformation.json" template * On the Options page click the Next button * On the Review page click the checkbox that says I acknowledge that this template might cause AWS CloudFormation to create IAM resources. * Click the Create button * When the CloudFormation stack completes the creation process and the Status field changes from CREATE_IN_PROGRESS to CREATE_COMPLETE. * Comment in this ticket letting us know the stack was created successfully. For the second part, the secure CloudTrail storage, we'd like to have you either enable or if enabled re-configure CloudTrail to use our secure CloudTrail storage account. If the account is already using the secure CloudTrail storage, we'd still like to have it reconfigured with this new version as the old version didn't include SNS CloudTrail notifications. CloudTrail is an AWS product which creates an audit log of all calls made to the AWS API by your account. For example, when you spin up a new ec2 instance, that call to AWS to RunInstances can be recorded with CloudTrail. This audit log is stored in S3. Here's why you should enable CloudTrail and use the AWS Secure CloudTrail Storage Account * By enabling CloudTrail, in the event of a security incident, the audit trail created by CloudTrail will help Infosec determine what has been affected and how the account was compromised * By using the AWS Secure CloudTrail Storage Account instead of storing the CloudTrail logs in your own account you get a few things * Firstly it easily allows Infosec to get access to the audit logs in the case of an incident * Far more importantly, by storing the logs in a separate, locked down account, in the event of an attacker gaining control of your AWS account, they will be unable to hide their tracks by deleting the CloudTrail logs after they finish. The worst they will be able to do is disable CloudTrail logging (which Infosec security auditing will detect immediately) * By deploying this CloudFormation template, CloudTrail will be enabled for your account in *all* regions, avoiding the hassle of either manually configuring CloudTrail in every region, or deploying individual CloudTrail stacks in every region. Here are the steps to begin using the AWS Secure CloudTrail Storage Account Note these steps are outlined in the "Deploy the CloudFormation template" section of this page : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-DeploytheCloudFormationtemplate 1. Browse to AWS CloudFormation in either us-west-2 Oregon, or us-east-1 N. Virginia (the 2 regions that support AWS Lambda) : https://console.aws.amazon.com/cloudformation/home?region=us-west-2 2. Click "Create Stack" 3. Under "Choose a template" select "Specify an Amazon S3 template URL" 4. Enter this URL : https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage.json 5. In the "Stack name" field enter "DeployCloudTrailCloudFormationStacks" and click "Next" 6. On the "Options" screen click "Next" 7. On the "Review" screen in the "Capabilities" section, check the checkbox for "I acknowledge that this template might cause AWS CloudFormation to create IAM resources." and click "Create" To learn how to fetch your CloudTrail logs from the secure storage account or to subscribe to notifications from CloudTrail, read about usage here : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-Usage
All of the above steps could be automated - given access to AWS tools and the appropriate IAM permissions - so that these changes ('remove X', 'add Y, Z, W') are deployed correctly and consistently. Doing these individually by hand without such a tool seems like a great deal of manual labor that will need to be performed on at least 30 Mozilla AWS accounts, if not more. Is such automation being developed?
Flags: needinfo?(gene)
This is about as automated as I'm able to make it. Each service is deployed with a single cloudformation template (security auditing and cloudformation). So it's two api calls in each AWS account to enable the two services. To the question of whether there's a way to automate this across many AWS accounts, that would depend upon a pre-existing user or role with rights to AssumeRole into each AWS account and have all the permissions needed existing across all the accounts which I don't think exists. If that would be a valuable automation tool for IT admins to have it may be something worth looking into. The security risk of having an account as powerful as that might be an issue, might not. If you've got suggestions of ways in which something here could be more automated, I'd be happy to implement it.
Flags: needinfo?(gene)
Richard, I'd like to add a clarification to these instructions which confused some other folks. For the security auditing and incident response IAM Roles, you only need to deploy the CloudFormation template in a single region (either us-west-2 or us-east-1 but not both). This single CloudFormation stack will enable security auditing for your account in *all* regions. The same goes for the CloudTrail CloudFormation template. It only needs to be deployed in a single region (either us-west-2 or us-east-1 but not both). The stack will in turn (on it's own) create a "sub-stack" in every region to enable CloudTrail. So, in summary, you should be deploying in total 2 CloudFormation templates. The first to enable security auditing, the second to enable CloudTrail.
Richard, do you know when this is slated to be done?
Flags: needinfo?(riweiss)
I just talked to Richard and he indicated that this request should go to JD and Gozer instead of him. There are 2 accounts in the list that he owns and 1 that jake owns, so I'll open separate tickets on those. So here's the updated list of accounts that this ticket is tracking : 177680776199 Nubis Lab - 1400 230871113385 Plan B Bugzilla - 1400 589768463761 Nubis Market - 1400 604263479206 Plan B Akamai eDNS Slave - 1400 647505682097 mozilla-sandbox - 1400 JD or Gozer, would you or someone on your team be able to help with what's up in Comment 0?
Flags: needinfo?(riweiss) → needinfo?(jcrowe)
Gene, I will put this on the list for roadmap planning we have going on next week. Can you indicate for me the level of criticality AKA does it need to be done in 60 days 90 days etc... Thanks
Flags: needinfo?(jcrowe)
Sure. The CloudTrail update should be done either by the end Q1 or by the time that you have a need to consume your CloudTrail logs and want the SNS notification to trigger that consumption. The driver for time on our side is the point at which we finish enabling MozDef to be triggered by SNS (instead of polling) for CloudTrail logs which will be this quarter. For the IAM Role update, that's a bit more urgent as until we have that we can't both audit your accounts for security risks and emit events about those risks to MozDef for action. With the existing "v1" IAM roles that you have, we can either audit, or alert but not both (fundamental IAM limitation : http://serverfault.com/questions/696193/how-can-i-chain-aws-iam-assumerole-api-calls ). So ideally the new IAM Roles would get deployed in the next 30 days.
Assignee: riweiss → jcrowe
The "IT CDN - 1400" account looks good with the new security audit roles and CloudTrail config. 1 down.
Oh, sorry, disregard Comment 10 , my records incorrectly associated this bug with the "IT CDN - 1400" account which is actually being tracked in Bug 1242001 by webops.
Assignee: jcrowe → gozer
Status: NEW → ASSIGNED
Blocks: 1216784
PR has been merged, will be picked up when we roll out v1.0.2
Thanks so much for integrating the new roles in. As it looks like from this ticket that the Secure CloudTrail Storage hasn't been integrated into nubis yet, I'd like to change (and simplify) the CloudTrail CloudFormation stack that you're integrating. In this ticket originally (back in December) I'd asked you to deploy this CloudFormation template in the accounts (See Comment 0) https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage.json I'd like to change that to a new simpler method which takes advantage of new AWS functionality released after this ticket was created ( https://aws.amazon.com/blogs/aws/aws-cloudtrail-update-turn-on-in-all-regions-use-multiple-trails/ ). Would you like me to request that in this ticket or should I open a new ticket? The new template is https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage_globally.json
Flags: needinfo?(gozer)
In talking with :r2 and :jd yesterday, they confirmed that the CloudTrail config in play in nubis is still the one from September 2015. Gozer or limed can you update it to use the new V2 cloudtrail config (in Comment 14 )?
Flags: needinfo?(limed)
Gene, In talking to gozer, we are actually using this policy now: https://github.com/nubisproject/nubis-deploy/tree/master/modules/opsec Do we still need someone to update?
Flags: needinfo?(limed)
:jd, That's the v2 security auditing roles which :gozer mentioned in Comment 13 . Those look great. I'm talking about CloudTrail though (not security audit roles). The Nubis CloudTrail code that :limed made appears to live here : https://github.com/nubisproject/nubis-cloudtrail This is the code from September 2015 (what I'm calling V1 cloudtrail). What needs to happen is that it be replaced with the V2 cloudtrail logging that takes advantage of the new functionality AWS released in December 2015. The link to both the V1 and V2 (for comparison) CloudTrail templates is in Comment 14.
Gene, Sorry for the confusion. I checked and we did not have that included. It has been added by: https://github.com/nubisproject/nubis-deploy/pull/35 Can you please take a look and make sure that this is exactly what you need? We are including this in the most recent release so it will be deployed within a few days to all Nubis accounts.
Flags: needinfo?(gozer)
:jd thanks for getting that change in. I'm not familiar with the syntax but we can just see if your accounts start logging to the secure storage account when it's deployed. When does this stack get deployed to the accounts in Comment 0 out of curiosity (not familiar with the release cycle)
Gene, The answer is complicated but I will try to explain it here. 177680776199 Nubis Lab - 1400 (Next week) 230871113385 Plan B Bugzilla - 1400 (New account already created, migrating this week or early next week) 248062938574 Plan B LDAP Master - 1400 (In process of being shut down) 329567179436 Consolidated Billing (r2 is responsible for this one completely, I have no state) 330914478726 Plan B Okta LDAP Gateway - 1400 (Was shut down several weeks ago) 369987351092 IT CDN - 1400 (I don't think we maintain this one, if there is a different expectation, please let me know) 589768463761 Nubis Market - 1400 (Next week) 598097830519 IT Backups - 1400 (I don't think we maintain this one, if there is a different expectation, please let me know) 604263479206 Plan B Akamai eDNS Slave - 1400 (New account already created, migrating this week or early next week) 647505682097 mozilla-sandbox - 1400 (No upgrade scheduled, will decom in a week or two)
Cool, ya I should have said the account list in Comment 7 , which looks good. 1 or 2 weeks for all of them is great, thanks. 177680776199 Nubis Lab - 1400 (Next week) 230871113385 Plan B Bugzilla - 1400 (New account already created, migrating this week or early next week) 589768463761 Nubis Market - 1400 (Next week) 604263479206 Plan B Akamai eDNS Slave - 1400 (New account already created, migrating this week or early next week) 647505682097 mozilla-sandbox - 1400 (No upgrade scheduled, will decom in a week or two)
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
:jd How's it lookin (here at the 2 week mark)? Are these 5 accounts further along towards adoption?
Flags: needinfo?(jcrowe)
Akami and Bugzilla have been provisioned in new accounts and should be working there. Akami should be cut over soon, it went to CAB this week. Bugzilla is working in the new account and the cutover is being planned. We have been preempted by this killchain stuff and have not even touched the other three accounts, they are still on our list but I do not have an updated timeline at this point.
Flags: needinfo?(jcrowe)
Got it, I hadn't put 2 and 2 together about the new accounts. 177680776199 Nubis Lab - 1400 (It's unknown when this account will have cloudtrail and the security audit roles enabled) 230871113385 Plan B Bugzilla - 1400 (314563910040 is the replacement for this account. The cutover to the new account is being planned) 589768463761 Nubis Market - 1400 (It's unknown when this account will have cloudtrail and the security audit roles enabled) 604263479206 Plan B Akamai eDNS Slave - 1400 (314563910040 is the replacement for this account. Akami should be cut over soon, it went to CAB this week.) 647505682097 mozilla-sandbox - 1400 (It's unknown when this account will be decommissioned )
On May 6th, Nubis Lab (177680776199) had 2 stacks spun up with infosec security iam roles arn:aws:cloudformation:us-west-2:177680776199:stack/opsec/344028e0-1409-11e6-b94d-50a68a201256 arn:aws:cloudformation:us-east-1:177680776199:stack/opsec/30ed0640-1409-11e6-891b-5044763e8bb3 Since this stack creates IAM Roles which are not scoped to a region, but instead are global, probably best to only create one stack in one region. Would you tear one of them down and let me know which one remains?
Flags: needinfo?(jcrowe)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Sweet, thanks :gozer Here's the current status (as I know it) 177680776199 Nubis Lab - 1400 Cloudtrail is currently logging though it looks like most of the old trails were not removed when the new trail was created. There is a global trail called "opsec-cloudtrail" which is great. But there are also individual trails in each region other than us-west-2/us-east-1 that duplicate the functionality of the global cloudtrail. Those trails have names like "MozillaSecuredCloudTrail-CloudTrail-RYHL0G0I26B0" Please delete all of the MozillaSecuredCloudTrail-CloudTrail-* Trails. Once https://github.com/nubisproject/nubis-deploy/pull/65 is deployed we'll have a single set of security audit roles instead of a set for each region in ['us-west-2','us-east-1'] 230871113385 Plan B Bugzilla - 1400 314563910040 is the replacement for this account. The cutover to the new account is being planned as of May 5th 589768463761 Nubis Market - 1400 Cloudtrail is currently logging though it's unclear if it's enabled globally or not. Security audit roles have not yet been enabled 604263479206 Plan B Akamai eDNS Slave - 1400 314563910040 is the replacement for this account. Akami should be cut over soon, it went to CAB the week of May 5th. What was the outcome of it going to CAB week before last? 647505682097 mozilla-sandbox - 1400 It's unknown when this account will be decommissioned
(In reply to Gene Wood [:gene] from comment #28) > Sweet, thanks :gozer > > Here's the current status (as I know it) > > 177680776199 Nubis Lab - 1400 > Cloudtrail is currently logging though it looks like most of the old trails > were not removed when the new trail was created. There is a global trail > called "opsec-cloudtrail" which is great. But there are also individual > trails in each region other than us-west-2/us-east-1 that duplicate the > functionality of the global cloudtrail. Those trails have names like > "MozillaSecuredCloudTrail-CloudTrail-RYHL0G0I26B0" > > Please delete all of the MozillaSecuredCloudTrail-CloudTrail-* Trails. Yeah, I believe these were created with the first generation of Cloudtrail enabling cloud-formation stuff and never got cleaned up. Working on it now. > Once https://github.com/nubisproject/nubis-deploy/pull/65 is deployed we'll > have a single set of security audit roles instead of a set for each region > in ['us-west-2','us-east-1'] > > > 230871113385 Plan B Bugzilla - 1400 > 314563910040 is the replacement for this account. The cutover to the new > account is being planned as of May 5th > > > 589768463761 Nubis Market - 1400 > Cloudtrail is currently logging though it's unclear if it's enabled > globally or not. Security audit roles have not yet been enabled That's because nubis-market is a shell (as in empty) account. We don't have EC2 instances normally deployed in there at all. So our normal account deployment stuff hasn't been applied there yet. > 604263479206 Plan B Akamai eDNS Slave - 1400 > 314563910040 is the replacement for this account. Akami should be cut over > soon, it went to CAB the week of May 5th. What was the outcome of it going > to CAB week before last? It got cutover. > 647505682097 mozilla-sandbox - 1400 > It's unknown when this account will be decommissioned :r2 ?
Flags: needinfo?(riweiss)
(In reply to Philippe M. Chiasson (:gozer) from comment #29) > (In reply to Gene Wood [:gene] from comment #28) > > Sweet, thanks :gozer > > > > Here's the current status (as I know it) > > > > 177680776199 Nubis Lab - 1400 > > Cloudtrail is currently logging though it looks like most of the old trails > > were not removed when the new trail was created. There is a global trail > > called "opsec-cloudtrail" which is great. But there are also individual > > trails in each region other than us-west-2/us-east-1 that duplicate the > > functionality of the global cloudtrail. Those trails have names like > > "MozillaSecuredCloudTrail-CloudTrail-RYHL0G0I26B0" > > > > Please delete all of the MozillaSecuredCloudTrail-CloudTrail-* Trails. MozillaSecuredCloudTrail-CloudTrail-* Trails all deleted as of now.
Great, thanks :gozer! > 589768463761 Nubis Market - 1400 > So our normal account deployment stuff hasn't been applied there yet. Is it planned to deploy the normal account stuff (including security auditing and cloudtrail) in this account? > 604263479206 Plan B Akamai eDNS Slave - 1400 Is there an account decomm request in progress or is that planned?
(In reply to Gene Wood [:gene] from comment #31) > > 604263479206 Plan B Akamai eDNS Slave - 1400 > > Is there an account decomm request in progress or is that planned? Please see bug 1272487
Digi, perfect thank you. Ok, current summary (as far as I can tell) 177680776199 Nubis Lab - 1400 Security audit and Cloudtrail complete 604263479206 Plan B Akamai eDNS Slave - 1400 This is being decommed in Bug 1272487 647505682097 mozilla-sandbox - 1400 Waiting to hear from :r2 when he's back on when this account is being decommed 230871113385 Plan B Bugzilla - 1400 314563910040 is the replacement for this account. The cutover to the new account is being planned as of May 5th 589768463761 Nubis Market - 1400 Account is missing normal nubis account deployment stuff including security auditing and cloudtrail
I'm working on shutting down the nubis-sandbox account. All of the users of the account, with one exception, have confirmed that their resources can be deleted. I need to open a new account for the Connected Devices team to migrate their work to before I can complete the shutdown.
Flags: needinfo?(riweiss)
:jd or :gozer It looks like the Bugzilla (417610946505) and the Akamai (314563910040) accounts created infosec security audit roles back on April 19th and then again on April 20th. They were again created today at 11:47am and then at 12:30am. Are these accounts still in flux? I ask because I'd thought they were not back on the 20th so I added them to our systems but their role ARNs have changed 4 times now. Should I leave these accounts out until I hear that they're ready to avoid changing the ARNs multiple times?
Hi, Sorry for the slow response, this tab got lost. Akami should be stable. Bugzilla is not and may change again before we cut over.
Flags: needinfo?(jcrowe)
Here's the updated status 604263479206 Plan B Akamai eDNS Slave - 1400 This account was closed in Bug 1272487 647505682097 mozilla-sandbox - 1400 This account was closed 230871113385 Plan B Bugzilla - 1400 Cutover to new account complete. This account was closed. 589768463761 Nubis Market - 1400 This account now has security audit roles established
Status: REOPENED → RESOLVED
Closed: 9 years ago8 years ago
Resolution: --- → FIXED
See Also: → 1476728
See Also: → 1525127
You need to log in before you can comment on or make changes to this bug.