1253435 - Change iops for w732 golden AMI for spot

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Q]

Right now disks are capping at 360 provisioned iops and we should be at 2000+. Assuming that  this lack of iops speed is causing command timeouts in tests. I am looking at the cloudtools config in the device-map ebs block but I am unsure if the setting would there or in the base AMI.

Comment 1

[Q]

Rail and/or Grenade,

  Would either of you have advice on this one. We need faster disk access on our spot instances as mentioned above. It looks like the golden capture process captures the daily AMI volume as "standard" in the t-w732 config file I see a device map that makes a "gp2" general purpose disk. Is the t-w732 config used for disks on  spot instances or is it somewhere else? Any advice on where to make changes?

Comment 2

[Rail Aliiev [:rail]]

I'd change "gp2" in https://github.com/mozilla/build-cloud-tools/blob/master/configs/t-w732#L41 to "io1" per http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html

Comment 3

[Q]

Should aws_create_instance.py respect these values to test with? I ran a test before with a temp version of the configuration with a type of io1 and a value for "iops" defined I also tried settings ebs optimized and did not get the desired effect.

Comment 4

[Rail Aliiev [:rail]]

Hmm, looks like we are ignoring that field in aws_create_instance.py: https://github.com/mozilla/build-cloud-tools/blob/master/cloudtools/scripts/aws_create_instance.py#L86-L102, but we use use it in aws_watch_pending.py via https://github.com/mozilla/build-cloud-tools/blob/53e9d57a7e281289399f9c92723e1895c2a060b7/cloudtools/aws/instance.py#L311-L312

We can either unify them or fix the usage in aws_create_instance.py.

Comment 5

[Q]

I see the results of the above as the golden image ( uses create_instance) is being spun up and caped as "standard" magnetic media which has an impact down the line. I believe the "bdm" blob that is being passed for the device should be respected with all AWS directives. I can take a swing at it but it would be much faster if there was someone we could tap with less of a cloud tools learning curve to fix aws_create_instance.py so I can test the IOPS changes.

Comment 6

[Q]

It may also be beneficial to respect the "ebsoptimized" instance flag in both tools if we are not already.

Comment 7

[Q]

Add in the need info request. Catlee, Arr who would be good to tap for the cloudtools changes that should be quick here ?

Comment 8

[Amy Rich [:arr] [:arich]]

rail or grenade are probably good bets for cloudtool changes.

Comment 9

[Rail Aliiev [:rail]]

There is a PR to review: https://github.com/mozilla/build-cloud-tools/pull/191

Comment 10

[Rob Thijssen [:grenade (EET/UTC+0300)]]

r+

Comment 11

[Rail Aliiev [:rail]]

merged

Comment 12

[Rob Thijssen [:grenade (EET/UTC+0300)]]

and another: https://github.com/mozilla/build-cloud-tools/pull/192

Comment 13

[Q]

Awesome thanks for the quick response guys!

Unfortunately this now fails because (at least in aws_create_instance.py) because the bdm blob creation does not pass through the iops parameter so creation will fail  with the error:

<Error><Code>InvalidParameterCombination</Code><Message>The parameter iops must be specified for io1 volumes.</Message></Error></Errors>

Comment 14

[Rob Thijssen [:grenade (EET/UTC+0300)]]

https://github.com/mozilla/build-cloud-tools/pull/195

Comment 15

[Rob Thijssen [:grenade (EET/UTC+0300)]]

merged.

Q, hopefully iops is set to 3000 now...

Comment 16

[Q]

Let me take a look thanks Rob

Comment 17

[Q]

Uh oh python failure in create instance on cli run:


  File "aws_create_instance.py", line 104
    and device_info.get("iops")
                              ^
SyntaxError: invalid syntax

this also breaks golden generation (which uses aws_create_instance.py under the hood)

Comment 18

[Q]

It looks like this change broke watch pending since it cannot process the iops paramater: 

   9263 Traceback (most recent call last):
   9264   File "aws_watch_pending.py", line 266, in do_request_spot_instances
   9265     is_spot=True, dryrun=dryrun, all_instances=all_instances)
   9266   File "aws_watch_pending.py", line 332, in do_request_instance
   9267     moz_instance_type, name, fqdn)
   9268   File "aws_watch_pending.py", line 367, in do_request_spot_instance
   9269     instance_profile_name=profile,
   9270   File "/builds/aws_manager/lib/python2.7/site-packages/boto/ec2/connection.py", line 1597, in request_spot_instances
   9271     verb='POST')
   9272   File "/builds/aws_manager/lib/python2.7/site-packages/boto/connection.py", line 1157, in get_list
   9273     raise self.ResponseError(response.status, response.reason, body)
   9274 EC2ResponseError: EC2ResponseError: 400 Bad Request
   9275 <?xml version="1.0" encoding="UTF-8"?>
   9276 <Response><Errors><Error><Code>InvalidParameterCombination</Code><Message>The parameter iops must be specified for io1 volumes.</Message></Error></Errors><RequestID>e76b8f75-60a2-4f0c-a179-5e6f98400f16</Reque        stID></Response>

Comment 19

[Rail Aliiev [:rail]]

https://github.com/mozilla/build-cloud-tools/pull/198 should fix this