1232128 - Request for Automation & Tools to update Infosec security auditing IAM Role and enable CloudTrail

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Gene Wood [:gene]]

First, thanks so much for granting Infosec (Previously called Opsec and our full name now is Enterprise Information Security) permissions to perform security audits and incident response on your AWS account earlier this year.

423163539776	Engineering Productivity - 8600

We'd like to do two things. First we'd like to have the IAM Roles that you created for us earlier this year in your account updated and secondly we'd like to have CloudTrail enabled in your account and connected to our Mozilla wide secure CloudTrail storage account.

For the first part, the update of the security audit IAM role, we'd like to update the IAM role for a few reasons

* We've migrated all activities related to auditing and incident response of Mozilla accounts to a dedicated AWS account for better separation of concerns. This will improve the security around the entity that you're granting auditing and incident response permissions to.
* We've separated out security auditing and incident response permissions into two distinct roles. This will allow us to grant certain rights to certain systems but not others. Another example of improved security through separation of duties ( https://www.owasp.org/index.php/Separation_of_duties )

Here are the steps to upgrade the IAM Roles

* Delete the old CloudFormation stack that you deployed for us. This stack was probably called "opsec-security-audit-role". You may need to check a few different regions to see where you deployed it. By deleting this stack it will delete the old IAM Role which granted us permission to do security audits.
* Deploy the new CloudFormation stack by following these steps
  * Note : These steps are also outlined in the "Create a Trusting Account using CloudFormation" section here : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Security+Auditing+and+Incident+Response+Services#AWSSecurityAuditingandIncidentResponseServices-CreateaTrustingAccountusingCloudFormation
* Log into their AWS web console in in either the us-west-2 region or the us-east-1 region (the only regions that support AWS Lambda currently)
* Browse to the CloudFormation section
* Click the Create Stack button
*     In the Name field enter something like "InfosecClientRoles"
*     In the Source field select Specify an Amazon S3 template URL and type in
      https://s3.amazonaws.com/infosec-cloudformation-templates/infosec-security-audit-incident-response-roles-cloudformation.json
* Click the Next button
* Deploy the "infosec-security-audit-incident-response-roles-cloudformation.json" template
* On the Options page click the Next button
* On the Review page click the checkbox that says I acknowledge that this template might cause AWS CloudFormation to create IAM resources.
* Click the Create button
* When the CloudFormation stack completes the creation process and the Status field changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.
* Comment in this ticket letting us know the stack was created successfully.

For the second part, the secure CloudTrail storage, we'd like to have you either enable or if enabled re-configure CloudTrail to use our secure CloudTrail storage account.

CloudTrail is an AWS product which creates an audit log of all calls made to the AWS API by your account. For example, when you spin up a new ec2 instance, that call to AWS to RunInstances can be recorded with CloudTrail. This audit log is stored in S3. Here's why you should enable CloudTrail and use the AWS Secure CloudTrail Storage Account

* By enabling CloudTrail, in the event of a security incident, the audit trail created by CloudTrail will help Infosec determine what has been affected and how the account was compromised
* By using the AWS Secure CloudTrail Storage Account instead of storing the CloudTrail logs in your own account you get a few things
  * Firstly it easily allows Infosec to get access to the audit logs in the case of an incident
  * Far more importantly, by storing the logs in a separate, locked down account, in the event of an attacker gaining control of your AWS account, they will be unable to hide their tracks by deleting the CloudTrail logs after they finish. The worst they will be able to do is disable CloudTrail logging (which Infosec security auditing will detect immediately)
* By deploying this CloudFormation template, CloudTrail will be enabled for your account in *all* regions, avoiding the hassle of either manually configuring CloudTrail in every region, or deploying individual CloudTrail stacks in every region.

Here are the steps to begin using the AWS Secure CloudTrail Storage Account

Note these steps are outlined in the "Deploy the CloudFormation template" section of this page : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-DeploytheCloudFormationtemplate

1. Browse to AWS CloudFormation in either us-west-2 Oregon, or us-east-1 N. Virginia (the 2 regions that support AWS Lambda) : https://console.aws.amazon.com/cloudformation/home?region=us-west-2
2. Click "Create Stack"
3. Under "Choose a template" select "Specify an Amazon S3 template URL"
4. Enter this URL : https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage.json
5. In the "Stack name" field enter "DeployCloudTrailCloudFormationStacks" and click "Next"
6. On the "Options" screen click "Next"
7. On the "Review" screen in the "Capabilities" section, check the checkbox for "I acknowledge that this template might cause AWS CloudFormation to create IAM resources." and click "Create"

To learn how to fetch your CloudTrail logs from the secure storage account or to subscribe to notifications from CloudTrail, read about usage here :
https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-Usage

Comment 1

[:glob ✱]

deployment of the CloudFormation stack failed:

CREATE_FAILED 	Custom::PublishToSNSInfo 	PublishToSNSInfo 	Failed to create resource. See the details in CloudWatch Log Stream: 2016/01/05/[$LATEST]5565a86f633847f48d78d1ba93d9ec03
Physical ID:InfosecClientRoles-PublishToSNSInfo-1N5H8OV0KP3E4

oddly enough that log stream doesn't exist; i only see 2016/01/05/[$LATEST]ecf0a6ceb4db47b6b0bb1b6df4885a3b, which doesn't contain an error.

Comment 2

[Gene Wood [:gene]]

Shoot.

The reason the log stream it mentions doesn't exist is that by default CloudFormation rolls back and deletes any stack that fails and in doing so it deletes the CloudWatch logs that were created.

You can disable this auto-rollback so the logs aren't deleted by changing an option when you create the CloudFormation stack.

First, delete the failed stack.

Next, create the stack again but this time when you get to the "Options" screen, click "Advanced" to expand the advanced options, then change the "Rollback on failure" value from "Yes" to "No"

Now when the stack creation fails the log stream won't be deleted and, if you're open to it, you could copy paste the error so I can determine the cause of the problem.

Also, sorry about this, I've tested this and it's worked for other accounts, I'm not sure off hand why it's failing but once I see the logs it should become apparent.

Comment 3

[Gene Wood [:gene]]

It looked like it worked that time (I got the SNS message with the information)!

Comment 4

[:glob ✱]

(In reply to Gene Wood [:gene] from comment #3)
> It looked like it worked that time (I got the SNS message with the
> information)!

yup!  i didn't do anything differently (aside from disabling rollback), so, uh, yay!

DeployCloudTrailCloudFormationStacks has also been created (which in turn appears to have created MozillaSecuredCloudTrail).

Comment 5

[Gene Wood [:gene]]

Hmm ok so that's very weird. That means that the problem, whatever it was, is ephemeral. Well, that's probably why I didn't find it in my testing. I'll go through and run the stack a bunch of times in my test environment to see if I can reproduce it.

Byron, thanks for deploying these two stacks, really appreciate it.