Closed Bug 1402233 Opened 2 years ago Closed 2 years ago

Intermittent gecko decision task failure to process testing/web-platform/moz.build

Categories

(Firefox Build System :: General, defect, major)

defect
Not set
major

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox58 --- fixed

People

(Reporter: aryx, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

https://treeherder.mozilla.org/logviewer.html#?job_id=132636367&repo=mozilla-inbound

Retrigger fails, but it passes on the next push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b2feeea8d65dba956ba15b1c6cb80053ad1c6337&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=c4e009f8a48645dabc09713e9547d9740eb8f3ad

James, can you take a look at this, please? There had been an error with a different moz.build file yesterday after tooling changes, so if it's not caused by wpt, please needinfo gps. Thank you.

[task 2017-09-22T05:20:07.705119Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2017-09-22T05:20:07.840394Z] "GET /integration/mozilla-inbound/json-automationrelevance/9fb93fcf2b8eece6b7b23d12e29cbacc16e90b0c HTTP/1.1" 200 None
[task 2017-09-22T05:20:07.846223Z] 2 commits influencing task scheduling:
[task 2017-09-22T05:20:07.846363Z]  cfb739b281f8 Bug 1397145 - Remove the support for moz-blob type in XHR, r=smaug
[task 2017-09-22T05:20:07.846493Z]  9fb93fcf2b8e Bug 1397151 - Remove the support for moz-chunk-text type in XHR, r=smaug
[task 2017-09-22T05:20:14.508177Z] Reading file: /builds/worker/checkouts/gecko/moz.build
[task 2017-09-22T05:20:14.555000Z] Reading file: /builds/worker/checkouts/gecko/dom/moz.build
[task 2017-09-22T05:20:14.564201Z] Reading file: /builds/worker/checkouts/gecko/dom/base/moz.build
[task 2017-09-22T05:20:14.588638Z] Reading file: /builds/worker/checkouts/gecko/dom/webidl/moz.build
[task 2017-09-22T05:20:14.605659Z] Reading file: /builds/worker/checkouts/gecko/dom/xhr/moz.build
[task 2017-09-22T05:20:14.625454Z] Reading file: /builds/worker/checkouts/gecko/testing/moz.build
[task 2017-09-22T05:20:14.633856Z] Reading file: /builds/worker/checkouts/gecko/testing/web-platform/moz.build
[task 2017-09-22T05:20:14.782912Z] Traceback (most recent call last):
[task 2017-09-22T05:20:14.783020Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 154, in taskgraph_decision
[task 2017-09-22T05:20:14.783079Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-09-22T05:20:14.783148Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 135, in taskgraph_decision
[task 2017-09-22T05:20:14.783200Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-09-22T05:20:14.783291Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 178, in morphed_task_graph
[task 2017-09-22T05:20:14.783344Z]     return self._run_until('morphed_task_graph')
[task 2017-09-22T05:20:14.783413Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 283, in _run_until
[task 2017-09-22T05:20:14.783448Z]     k, v = self._run.next()
[task 2017-09-22T05:20:14.783499Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 270, in _run
[task 2017-09-22T05:20:14.783534Z]     do_not_optimize)
[task 2017-09-22T05:20:14.783594Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 55, in optimize_task_graph
[task 2017-09-22T05:20:14.783633Z]     do_not_optimize=do_not_optimize)
[task 2017-09-22T05:20:14.783691Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 126, in remove_tasks
[task 2017-09-22T05:20:14.783735Z]     if opt.should_remove_task(task, params, arg):
[task 2017-09-22T05:20:14.783796Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 376, in should_remove_task
[task 2017-09-22T05:20:14.783854Z]     scheduled = self.scheduled_by_push(params['head_repository'], params['head_rev'])
[task 2017-09-22T05:20:14.783916Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 944, in method_call
[task 2017-09-22T05:20:14.783958Z]     cache[args] = self.func(instance, *args)
[task 2017-09-22T05:20:14.784016Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 367, in scheduled_by_push
[task 2017-09-22T05:20:14.784063Z]     for p, m in rdr.files_info(changed_files).items():
[task 2017-09-22T05:20:14.784133Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/frontend/reader.py", line 1342, in files_info
[task 2017-09-22T05:20:14.784182Z]     paths, _ = self.read_relevant_mozbuilds(paths)
[task 2017-09-22T05:20:14.784258Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/frontend/reader.py", line 1306, in read_relevant_mozbuilds
[task 2017-09-22T05:20:14.784308Z]     self.config, metadata=metadata):
[task 2017-09-22T05:20:14.784391Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/frontend/reader.py", line 1061, in read_mozbuild
[task 2017-09-22T05:20:14.784434Z]     raise bre
[task 2017-09-22T05:20:14.785111Z] BuildReaderError: 
[task 2017-09-22T05:20:14.785155Z] ==============================
[task 2017-09-22T05:20:14.785188Z] ERROR PROCESSING MOZBUILD FILE
[task 2017-09-22T05:20:14.785220Z] ==============================
[task 2017-09-22T05:20:14.785244Z] 
[task 2017-09-22T05:20:14.785290Z] The error occurred while processing the following file:
[task 2017-09-22T05:20:14.785316Z] 
[task 2017-09-22T05:20:14.785361Z]     /builds/worker/checkouts/gecko/testing/web-platform/moz.build
[task 2017-09-22T05:20:14.785393Z] 
[task 2017-09-22T05:20:14.785435Z] The error was triggered on line 9 of this file:
[task 2017-09-22T05:20:14.785460Z] 
[task 2017-09-22T05:20:14.785498Z]     ('mozilla/meta/MANIFEST.json', 'mozilla/tests/')
[task 2017-09-22T05:20:14.785523Z] 
[task 2017-09-22T05:20:14.785578Z] An error was encountered as part of executing the file itself. The error appears to be the fault of the script.
[task 2017-09-22T05:20:14.785609Z] 
[task 2017-09-22T05:20:14.785642Z] The error as reported by Python is:
[task 2017-09-22T05:20:14.785663Z] 
[task 2017-09-22T05:20:14.785710Z]     ["UnboundLocalError: local variable 'wptmanifest' referenced before assignment\n"]
[task 2017-09-22T05:20:14.785731Z] 
[taskcluster 2017-09-22 05:20:15.199Z] === Task Finished ===
[taskcluster 2017-09-22 05:20:23.795Z] Unsuccessful task run with exit code: 1 completed in 45.948 seconds
Flags: needinfo?(james)
This has hit several times, e.g. the merges, and also fails if retriggered. Does it fail for every wpt push?
Severity: normal → major
Flags: needinfo?(gps)
I am also still seeing this in >50% of my recent try pushes and it's eating into my productivity.
Now I'm just doing all my try pushes with the sparse-profile change backed out and that seems to produce much better results.
Component: web-platform-tests → Build Config
Flags: needinfo?(james)
Product: Testing → Core
Version: Version 3 → unspecified
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #3)
> Now I'm just doing all my try pushes with the sparse-profile change backed
> out and that seems to produce much better results.

To be clear the change I backed out is https://hg.mozilla.org/integration/autoland/rev/bffb6b9d72128317e400b0bc35dcde97c7068250
So what I know is that the code that's failing tries to execfile testing/web-platform/tests/tools/localpaths.py so that the paths necessary to run wpt bits are in sys.path. That seems to be failing intermittently? At least I assume that's why wptmanifest is referenced before assignment. But the file it's running hasn't changed since August, so I don't think that can directly  be the problem.

Based on kats' observation above, I guess we are missing this file in the sparse checkout, or something?
See Also: → 1402142
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #4)
> (In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #3)
> > Now I'm just doing all my try pushes with the sparse-profile change backed
> > out and that seems to produce much better results.
> 
> To be clear the change I backed out is
> https://hg.mozilla.org/integration/autoland/rev/
> bffb6b9d72128317e400b0bc35dcde97c7068250

That commit should have nothing to do with the decision task.

And this isn't intermittent per se. Intermittent means a supposedly deterministic behavior isn't.

Different decision tasks against different changesets fundamentally change how the decision task operates. So this is a bug that manifests when specific conditions are met.

Anyway, I think I know what the problem is.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Flags: needinfo?(gps)
Duplicate of this bug: 1402409
Blocks: 1402010
Comment on attachment 8911276 [details]
Bug 1402233 - Add web platform tools files to taskgraph sparse profile

https://reviewboard.mozilla.org/r/182764/#review188014
Attachment #8911276 - Flags: review+
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/13ec36bee625
Add web platform tools files to taskgraph sparse profile r=mshal
https://hg.mozilla.org/mozilla-central/rev/13ec36bee625
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.