989159 - Gaia try jobs unable to load gaia.json

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

A recent gaia try job is failing to load gaia.json:

11:37:33     INFO - Running pre-action listener: _resource_record_pre_action
11:37:33     INFO - Running main action method: pull
11:37:33     INFO - retry: Calling <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x10567d0>> with args: ('https://hg.mozilla.org/try/raw-file/66101b2e4fbe/b2g/config/gaia.json',), kwargs: {}, attempt #1
11:37:34     INFO - retry: Failed, sleeping 60 seconds before retrying
11:38:35     INFO - retry: Calling <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x10567d0>> with args: ('https://hg.mozilla.org/try/raw-file/66101b2e4fbe/b2g/config/gaia.json',), kwargs: {}, attempt #2
11:38:38     INFO - retry: Failed, sleeping 120 seconds before retrying
11:40:38     INFO - retry: Calling <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x10567d0>> with args: ('https://hg.mozilla.org/try/raw-file/66101b2e4fbe/b2g/config/gaia.json',), kwargs: {}, attempt #3
11:40:39     INFO - retry: Failed, sleeping 240 seconds before retrying
11:44:39     INFO - retry: Calling <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x10567d0>> with args: ('https://hg.mozilla.org/try/raw-file/66101b2e4fbe/b2g/config/gaia.json',), kwargs: {}, attempt #4
11:44:40     INFO - retry: Failed, sleeping 300 seconds before retrying
11:49:40     INFO - retry: Calling <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x10567d0>> with args: ('https://hg.mozilla.org/try/raw-file/66101b2e4fbe/b2g/config/gaia.json',), kwargs: {}, attempt #5
11:49:41    ERROR - <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x10567d0>> failed after 5 tries!
11:49:41    ERROR - Caught exception: Expecting property name: line 6 column 5 (char 160)
11:49:41    ERROR - Caught exception: Expecting property name: line 6 column 5 (char 160)
11:49:41    ERROR - Caught exception: Expecting property name: line 6 column 5 (char 160)
11:49:41    ERROR - Caught exception: Expecting property name: line 6 column 5 (char 160)
11:49:41    ERROR - Caught exception: Expecting property name: line 6 column 5 (char 160)

It wasn't clear from a cursory glance what's wrong.  Similar code on trunk branches works fine.

Has there been some change to network rules that prevent this fetch from succeeding on try machines?

full log: https://tbpl.mozilla.org/php/getParsedLog.php?id=36821752&tree=Try&full=1#error0

Comment 1

[Aki Sasaki (not active)]

[[e-mail] ~]$ curl https://hg.mozilla.org/try/raw-file/66101b2e4fbe/b2g/config/gaia.json
{
    "git": {
        "git_revision": "",
        "remote": "https://github.com/6a68/gaia.git",
        "branch": "bug-987417-fxa-settings-copy-update",
    },
    "revision": "57c407d3e94ba322688b4bd121eda317806ab44e",
    "repo_path": "/integration/gaia-central"
}

Either this was intermittent or it's been recently fixed.

Comment 2

[Jonathan Griffin (:jgriffin)]

It's not fixed yet.  :(  See https://tbpl.mozilla.org/?rev=66101b2e4fbe&tree=Try, where I just retriggered a "G".

The url is very reachable locally, but doesn't seem to be fetchable from the test slave.  Any ideas what might be happening?

Comment 3

[Aki Sasaki (not active)]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> It's not fixed yet.  :(  See
> https://tbpl.mozilla.org/?rev=66101b2e4fbe&tree=Try, where I just
> retriggered a "G".
> 
> The url is very reachable locally, but doesn't seem to be fetchable from the
> test slave.  Any ideas what might be happening?

Try http instead of https?
I wish that method were more verbose...

Comment 4

[Aki Sasaki (not active)]

Attachment: verbose load_json_from_url

With some pep8 fixes.

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 8399587 [details] [diff] [review]
verbose load_json_from_url

http://hg.mozilla.org/build/mozharness/rev/cef2d547ce35

Comment 6

[John Hopkins (:jhopkins)]

in production.

Comment 7

[Jonathan Griffin (:jgriffin)]

I've scheduled a retrigger on https://tbpl.mozilla.org/?rev=66101b2e4fbe&tree=Try, so hopefully we can see what the problem is.

Comment 8

[Jonathan Griffin (:jgriffin)]

Ok,the results are in.  This looks like a JSON-decoding error!

6a86, can you push this to try again with valid json?

Specifically, without a comma after the branch value:  https://hg.mozilla.org/try/rev/66101b2e4fbe

Comment 9

[Jonathan Griffin (:jgriffin)]

s/6a86/6a68/

Comment 10

[Aki Sasaki (not active)]

I think this bug is fixed?

Comment 11

[Jonathan Griffin (:jgriffin)]

Actually INVALID, I think.