Status

Infrastructure & Operations
Buildduty
--
blocker
RESOLVED FIXED
3 years ago
13 days ago

People

(Reporter: RyanVM, Unassigned)

Tracking

Details

(Reporter)

Description

3 years ago
AWS test jobs are over an hour backlogged on inbound. Trees closed.
Our old friend iz back:

 Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: 2015-08-10 09:16:35,714 - Cannot start
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: Traceback (most recent call last):
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 268, in do_request_spot_instances
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     is_spot=True, dryrun=dryrun, all_instances=all_instances)
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 331, in do_request_instance
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     moz_instance_type, name, fqdn)
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 368, in do_request_spot_instance
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     instance_profile_name=profile,
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/builds/aws_manager/lib/python2.7/site-packages/boto/ec2/connection.py", line 1597, in request_spot_instances
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     verb='POST')
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/builds/aws_manager/lib/python2.7/site-packages/boto/connection.py", line 1157, in get_list
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     raise self.ResponseError(response.status, response.reason, body)
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: EC2ResponseError: EC2ResponseError: 400 Bad Request
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: <?xml version="1.0" encoding="UTF-8"?>
Aug 10 09:16:35 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: <Response><Errors><Error><Code>InsufficientFreeAddressesInSubnet</Code><Message>There are not enough free addresses in subnet 'subnet-851471e0' to satisfy the requested number of instances.</Message></Error></Errors><RequestID>a0b8f79e-69a5-456e-88c0-8e40830daf3a</RequestID></Response>
Aug 10 09:18:02 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:28:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:43:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:48:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"
Aug 10 09:58:01 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: lockfile: Sorry, giving up on "/builds/aws_manager/aws_watch_pending.py.lock"

Updated

3 years ago
Blocks: 1109871

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED

Updated

13 days ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.