1456357 - Intermittent [taskcluster:error] Get http://schemas.taskcluster.net/generic-worker/v1/payload.json: dial tcp 52.84.128.102:80: i/o timeout

Status: RESOLVED FIXED

(Taskcluster :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=175234664&repo=mozilla-release

https://queue.taskcluster.net/v1/task/IUPLmYSJQGmFt9vLhGH_ww/runs/1/artifacts/public/logs/live_backing.log

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'm curious why this particular file is being downloaded, and why it's over http..

Comment 2

[Pete Moore [:pmoore][:pete]]

This is indeed extremely strange.

The reference to the schema location is in the "id" property of the schema.

Here we see the lines where the task payload is validated:

https://github.com/taskcluster/generic-worker/blob/v10.7.8/main.go#L839-L841

> 	schemaLoader := gojsonschema.NewStringLoader(taskPayloadSchema())
> 	docLoader := gojsonschema.NewStringLoader(string(jsonPayload))
> 	result, err := gojsonschema.Validate(schemaLoader, docLoader)

The schema loader takes a string as input, which is the full payload schema, which is burned into the release here:

https://github.com/taskcluster/generic-worker/blob/v10.7.8/generated_windows.go#L311-L609

Looking at the spec, I see that the property should be named "$id" rather that "id". I therefore have no idea why gojsonschema, that validates the payload, would fetch content from the url. Even if it had been named correctly ("$id" instead of "id") then it is just a URI for resolving references, rather than a URL to actively fetch content from.

I think the safest thing to do will just be to delete the property entirely.

I suspect there is some magic happening in https://github.com/xeipuuv/gojsonschema that looks for strings that appear to be URLs (possibly checking for the '#' at the end) and then preemptively downloads them. This is just a hunch, because I have no other explanation for how something would find a reference to this url, and actually try to query it. Not good.

Comment 3

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for generic-worker

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/generic-worker

https://github.com/taskcluster/generic-worker/commit/81ac9f3f40e7ef454484c80d41c051daaab6964f
Bug 1456357 - remove references to http://schemas.taskcluster.net/generic-worker/v1/payload.json

https://github.com/taskcluster/generic-worker/commit/6c9eec9baee55f8f9842e6d01bb4a7011214e2b4
Merge pull request #85 from taskcluster/bug1456357

Bug 1456357 - remove references to http://schemas.taskcluster.net/generic-worker/v1/payload.json

Comment 5

[Pete Moore [:pmoore][:pete]]

Released in generic-worker 10.7.11.

Comment 6

[Intermittent Failures Robot]

6 failures in 780 pushes (0.008 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 4
* mozilla-release: 2

Platform breakdown:
* windows10-64-qr: 2
* windows10-64: 2
* windows2012-64: 1
* windows2012-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1456357&startday=2018-04-23&endday=2018-04-29&tree=trunk

Comment 7

[Intermittent Failures Robot]

1 failures in 639 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1456357&startday=2018-05-14&endday=2018-05-20&tree=trunk