1192898 - 1hr+ AWS test backlog

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

AWS test jobs are over an hour backlogged on inbound. Trees closed.

Comment 1

[Justin Wood (:Callek)]

Our old friend iz back:

 Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: 2015-08-10 09:16:35,714 - Cannot start
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: Traceback (most recent call last):
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 268, in do_request_spot_instances
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     is_spot=True, dryrun=dryrun, all_instances=all_instances)
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 331, in do_request_instance
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     moz_instance_type, name, fqdn)
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 368, in do_request_spot_instance
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     instance_profile_name=profile,
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/builds/aws_manager/lib/python2.7/site-packages/boto/ec2/connection.py", line 1597, in request_spot_instances
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     verb='POST')
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/builds/aws_manager/lib/python2.7/site-packages/boto/connection.py", line 1157, in get_list
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     raise self.ResponseError(response.status, response.reason, body)
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: EC2ResponseError: EC2ResponseError: 400 Bad Request
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: <?xml version="1.0" encoding="UTF-8"?>
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: <Response><Errors><Error><Code>InsufficientFreeAddressesInSubnet</Code><Message>There are not enough free addresses in subnet 'subnet-851471e0' to satisfy the requested number of instances.</Message></Error></Errors><RequestID>a0b8f79e-69a5-456e-88c0-8e40830daf3a</RequestID></Response>
Aug 10 09:18:02 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:28:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:43:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:48:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:58:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"