1166088 - AWS machines not starting

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Most trees closed.

Comment 2

[Justin Wood (:Callek)]

Due to https://github.com/mozilla/build-cloud-tools/pull/73 (and /71 as well)

 May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: Traceback (most recent call last):
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 505, in <module>
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     main()
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 481, in main
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     ondemand_config=config.get("ondemand"),
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 397, in aws_watch_pending
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     spot_config=spot_config, dryrun=dryrun, slaveset=slaveset)
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "aws_watch_pending.py", line 85, in request_spot_instances
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     instance_config = load_instance_config(moz_instance_type)
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/builds/aws_manager/lib/python2.7/site-packages/repoze/lru/__init__.py", line 287, in lru_cached
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     val = f(*arg)
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/builds/aws_manager/cloud-tools/cloudtools/aws/__init__.py", line 248, in load_instance_config
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     moz_instance_type)))
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/tools/python27/lib/python2.7/json/__init__.py", line 278, in load
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     **kw)
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/tools/python27/lib/python2.7/json/__init__.py", line 326, in loads
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     return _default_decoder.decode(s)
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/tools/python27/lib/python2.7/json/decoder.py", line 366, in decode
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:   File "/tools/python27/lib/python2.7/json/decoder.py", line 384, in raw_decode
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py:     raise ValueError("No JSON object could be decoded")
May 18 15:55:31 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: ValueError: No JSON object could be decoded

Specifically I used ' instead of " for the quote markers....

Comment 3

[Justin Wood (:Callek)]

Attachment: Make valid json

Comment 4

[Justin Wood (:Callek)]

[19:32:27]	Callek	KWierso|sheriffduty: touchpoint, I believe things should be starting up soon, I merged the change live and things look sane... I'm monitoring

Comment 5

[Justin Wood (:Callek)]

[21:18:22]	Callek	KWierso|afk: RyanVM|afk: Ok imo trees no longer need to stay closed due to any ongoing releng issue...
[21:18:39]	Callek	despite the script not completely cycling through its attempts to start things yet, I think we're good on the code/infra side on my end...
[21:19:04]	Callek	I don't preclude the desire to keep it closed to verify that things that were pending are not hiding unkown bustage though
[21:19:10]	Callek	https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=ee7ff30474fd&filter-resultStatus=pending even has most of its stuff started

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I'm reopening and running away for the night.