1232121 - Request for RelEng to update Infosec security auditing IAM Role and enable CloudTrail

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Gene Wood [:gene]]

First, thanks so much for granting Infosec (Previously called Opsec and our full name now is Enterprise Information Security) permissions to perform security audits and incident response on your AWS account earlier this year.

314336048151	Mozilla Release Engineering - 8100

We'd like to do two things. First we'd like to have the IAM Roles that you created for us earlier this year in your account updated and secondly we'd like to have CloudTrail enabled in your account and connected to our Mozilla wide secure CloudTrail storage account.

For the first part, the update of the security audit IAM role, we'd like to update the IAM role for a few reasons

* We've migrated all activities related to auditing and incident response of Mozilla accounts to a dedicated AWS account for better separation of concerns. This will improve the security around the entity that you're granting auditing and incident response permissions to.
* We've separated out security auditing and incident response permissions into two distinct roles. This will allow us to grant certain rights to certain systems but not others. Another example of improved security through separation of duties ( https://www.owasp.org/index.php/Separation_of_duties )

Here are the steps to upgrade the IAM Roles

* Delete the old CloudFormation stack that you deployed for us. This stack was probably called "opsec-security-audit-role". You may need to check a few different regions to see where you deployed it. By deleting this stack it will delete the old IAM Role which granted us permission to do security audits.
* Deploy the new CloudFormation stack by following these steps
  * Note : These steps are also outlined in the "Create a Trusting Account using CloudFormation" section here : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Security+Auditing+and+Incident+Response+Services#AWSSecurityAuditingandIncidentResponseServices-CreateaTrustingAccountusingCloudFormation
* Log into their AWS web console in in either the us-west-2 region or the us-east-1 region (the only regions that support AWS Lambda currently)
* Browse to the CloudFormation section
* Click the Create Stack button
*     In the Name field enter something like "InfosecClientRoles"
*     In the Source field select Specify an Amazon S3 template URL and type in
      https://s3.amazonaws.com/infosec-cloudformation-templates/infosec-security-audit-incident-response-roles-cloudformation.json
* Click the Next button
* Deploy the "infosec-security-audit-incident-response-roles-cloudformation.json" template
* On the Options page click the Next button
* On the Review page click the checkbox that says I acknowledge that this template might cause AWS CloudFormation to create IAM resources.
* Click the Create button
* When the CloudFormation stack completes the creation process and the Status field changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.
* Comment in this ticket letting us know the stack was created successfully.

For the second part, the secure CloudTrail storage, we'd like to have you either enable or if enabled re-configure CloudTrail to use our secure CloudTrail storage account.

CloudTrail is an AWS product which creates an audit log of all calls made to the AWS API by your account. For example, when you spin up a new ec2 instance, that call to AWS to RunInstances can be recorded with CloudTrail. This audit log is stored in S3. Here's why you should enable CloudTrail and use the AWS Secure CloudTrail Storage Account

* By enabling CloudTrail, in the event of a security incident, the audit trail created by CloudTrail will help Infosec determine what has been affected and how the account was compromised
* By using the AWS Secure CloudTrail Storage Account instead of storing the CloudTrail logs in your own account you get a few things
  * Firstly it easily allows Infosec to get access to the audit logs in the case of an incident
  * Far more importantly, by storing the logs in a separate, locked down account, in the event of an attacker gaining control of your AWS account, they will be unable to hide their tracks by deleting the CloudTrail logs after they finish. The worst they will be able to do is disable CloudTrail logging (which Infosec security auditing will detect immediately)
* By deploying this CloudFormation template, CloudTrail will be enabled for your account in *all* regions, avoiding the hassle of either manually configuring CloudTrail in every region, or deploying individual CloudTrail stacks in every region.

Here are the steps to begin using the AWS Secure CloudTrail Storage Account

Note these steps are outlined in the "Deploy the CloudFormation template" section of this page : https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-DeploytheCloudFormationtemplate

1. Browse to AWS CloudFormation in either us-west-2 Oregon, or us-east-1 N. Virginia (the 2 regions that support AWS Lambda) : https://console.aws.amazon.com/cloudformation/home?region=us-west-2
2. Click "Create Stack"
3. Under "Choose a template" select "Specify an Amazon S3 template URL"
4. Enter this URL : https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage.json
5. In the "Stack name" field enter "DeployCloudTrailCloudFormationStacks" and click "Next"
6. On the "Options" screen click "Next"
7. On the "Review" screen in the "Capabilities" section, check the checkbox for "I acknowledge that this template might cause AWS CloudFormation to create IAM resources." and click "Create"

To learn how to fetch your CloudTrail logs from the secure storage account or to subscribe to notifications from CloudTrail, read about usage here :
https://mana.mozilla.org/wiki/display/SECURITY/AWS+Secure+CloudTrail+Storage+System#AWSSecureCloudTrailStorageSystem-Usage

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Gene Wood [:gene] from comment #0)
> * Delete the old CloudFormation stack that you deployed for us. This stack
> was probably called "opsec-security-audit-role". You may need to check a few
> different regions to see where you deployed it. By deleting this stack it
> will delete the old IAM Role which granted us permission to do security
> audits.

DONE

> * Deploy the new CloudFormation stack by following these steps
> * Log into their AWS web console in in either the us-west-2 region or the
> us-east-1 region (the only regions that support AWS Lambda currently)
> * Browse to the CloudFormation section
> * Click the Create Stack button
> *     In the Name field enter something like "InfosecClientRoles"
> *     In the Source field select Specify an Amazon S3 template URL and type
> in
> https://s3.amazonaws.com/infosec-cloudformation-templates/infosec-security-
> audit-incident-response-roles-cloudformation.json
> * Click the Next button
> * Deploy the
> "infosec-security-audit-incident-response-roles-cloudformation.json" template
> * On the Options page click the Next button
> * On the Review page click the checkbox that says I acknowledge that this
> template might cause AWS CloudFormation to create IAM resources.
> * Click the Create button
> * When the CloudFormation stack completes the creation process and the
> Status field changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.
> * Comment in this ticket letting us know the stack was created successfully.

I used us-east-1.  This failed, sadly:

0:56:13 UTC-0500 	CREATE_FAILED 	Custom::PublishToSNSInfo 	PublishToSNSInfo 	Failed to create resource. See the details in CloudWatch Log Stream: 2015/12/17/[$LATEST]377670b9e94245b48f16504709299b59

Those details are:

START RequestId: ba95954a-a4d6-11e5-a08c-e97f07891690 Version: $LATEST
[INFO]	2015-12-17T15:56:26.575Z	ba95954a-a4d6-11e5-a08c-e97f07891690	REQUEST RECEIVED: 
{
    "StackId": "arn:aws:cloudformation:us-east-1:314336048151:stack/InfosecClientRoles/916b40b0-a4d6-11e5-ae67-50d5cd23c4c6",
    "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A314336048151%3Astack/InfosecClientRoles/916b40b0-a4d6-11e5-ae67-50d5cd23c4c6%7CPublishToSNSInfo%7C27e1954f-0f38-42dc-aae4-e909be7ba2c0?AWSAccessKeyId=AKIAJNXHFR7P7YGKLDPQ&Expires=1450374986&Signature=WXIOprPer%2BV8oz5iVyNSH6hr6kY%3D",
    "ResourceProperties": {
        "Message": "{\"Incident Response Role ARN\": \"arn:aws:iam::314336048151:role/InfosecClientRoles-InfosecIncidentResponseRole-1IP4M0AK2ZXGK\", \"Security Audit Role ARN\": \"arn:aws:iam::314336048151:role/InfosecClientRoles-InfosecSecurityAuditRole-CTRHCRCV4MY9\"}",
        "ServiceToken": "arn:aws:lambda:us-east-1:314336048151:function:InfosecClientRoles-PublishToSNS-16ATN67RQ3GOF",
        "TopicArn": "arn:aws:sns:us-west-2:371522382791:AccountUpdates"
    },
    "RequestType": "Delete",
    "ServiceToken": "arn:aws:lambda:us-east-1:314336048151:function:InfosecClientRoles-PublishToSNS-16ATN67RQ3GOF",
    "ResourceType": "Custom::PublishToSNSInfo",
    "PhysicalResourceId": "InfosecClientRoles-PublishToSNSInfo-1DGCMDPW4GQAW",
    "RequestId": "27e1954f-0f38-42dc-aae4-e909be7ba2c0",
    "LogicalResourceId": "PublishToSNSInfo"
}
[INFO]	2015-12-17T15:56:26.575Z	ba95954a-a4d6-11e5-a08c-e97f07891690	LambdaContext: 
{
    "aws_request_id": "ba95954a-a4d6-11e5-a08c-e97f07891690",
    "log_stream_name": "2015/12/17/[$LATEST]377670b9e94245b48f16504709299b59",
    "invoked_function_arn": "arn:aws:lambda:us-east-1:314336048151:function:InfosecClientRoles-PublishToSNS-16ATN67RQ3GOF",
    "client_context": null,
    "log_group_name": "/aws/lambda/InfosecClientRoles-PublishToSNS-16ATN67RQ3GOF",
    "function_name": "InfosecClientRoles-PublishToSNS-16ATN67RQ3GOF",
    "function_version": "$LATEST",
    "identity": "<__main__.CognitoIdentity object at 0x7f4e76e779d0>",
    "memory_limit_in_mb": "128"
}
[INFO]	2015-12-17T15:56:26.578Z	ba95954a-a4d6-11e5-a08c-e97f07891690	arguments are {u'Message': u'{"Incident Response Role ARN": "arn:aws:iam::314336048151:role/InfosecClientRoles-InfosecIncidentResponseRole-1IP4M0AK2ZXGK", "Security Audit Role ARN": "arn:aws:iam::314336048151:role/InfosecClientRoles-InfosecSecurityAuditRole-CTRHCRCV4MY9"}', u'TopicArn': u'arn:aws:sns:us-west-2:371522382791:AccountUpdates'}
END RequestId: ba95954a-a4d6-11e5-a08c-e97f07891690
REPORT RequestId: ba95954a-a4d6-11e5-a08c-e97f07891690	Duration: 708.75 ms	Billed Duration: 800 ms 	Memory Size: 128 MB	Max Memory Used: 42 MB	

I don't see anything obvious to fix, so back to you :)

Comment 2

[Gene Wood [:gene]]

Attachment: Screenshot+from+2015-12-21+16-49-30.png

I worked with Dustin and got logs for the Create Stack (the logs in Comment 1 are from the Delete Stack).

I'll look into what's causing this exception

Comment 3

[Gene Wood [:gene]]

Dustin, thanks for catching this bug.

https://github.com/mozilla/security/issues/8

I've fixed this and deployed the fix.

Please delete the failed stack and create it anew. It should work this time.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

GREAT SUCCESS!

Thanks :)

Comment 5

[Gene Wood [:gene]]

It looks like you missed the second half of the ticket that enables use of the secure CloudTrail storage system. I'll give you a simplified instruction here that overrides what's in Comment 0 related to CloudTrail

Please deploy this CloudFormation template once in one region : 

https://s3.amazonaws.com/infosec-cloudformation-templates/configure_cloudtrail_to_use_mozilla_secure_storage_globally.json

Comment 6

[Gene Wood [:gene]]

Looking good, thanks!