1259399 - CloudTools intermittently copies wrong AMI from use1 to usw2 during golden ami creation process

Status: RESOLVED DUPLICATE of bug 1152624

(Release Engineering :: General, defect)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: examples of aws-manager logs showing incorrect AMI selection

Frequently during the golden AMI creation process, the wrong AMI is selected for copying from us-east-1 to us-west-2.

This rarely causes problems because the selection filter usually picks the previous day's AMI (instead of the current day's). Which means that the damage is limited to creating builders with day-old puppet configuration.

Two examples of the aws-manager logs showing incorrect AMI selection are attached.

The filter is here: https://github.com/mozilla/build-cloud-tools/blob/master/cloudtools/scripts/aws_create_instance.py#L311

-- Where the selector is looking for the latest image.

Comment 1

[Rail Aliiev [:rail]]

Looks like https://github.com/mozilla/build-cloud-tools/blob/600dcc1aa87a3c00921373c4f503e7cc44f05043/cloudtools/aws/ami.py#L75 is not enough to make sure AMI is ready to be copied, even though we use the same state filter when we select AMIs to copy: https://github.com/mozilla/build-cloud-tools/blob/600dcc1aa87a3c00921373c4f503e7cc44f05043/cloudtools/aws/ami.py#L102 and we check the status of the underlying snapshot in https://github.com/mozilla/build-cloud-tools/blob/600dcc1aa87a3c00921373c4f503e7cc44f05043/cloudtools/aws/ami.py#L44

Maybe we should get rid of multiprocessing in https://github.com/mozilla/build-cloud-tools/blob/master/cloudtools/scripts/aws_create_instance.py#L229, return AMI ID from make_instance() and explicitly use it (retrying on errors)?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Dupe of bug 1152624 ?