1402233 - Intermittent gecko decision task failure to process testing/web-platform/moz.build

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=132636367&repo=mozilla-inbound

Retrigger fails, but it passes on the next push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b2feeea8d65dba956ba15b1c6cb80053ad1c6337&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=c4e009f8a48645dabc09713e9547d9740eb8f3ad

James, can you take a look at this, please? There had been an error with a different moz.build file yesterday after tooling changes, so if it's not caused by wpt, please needinfo gps. Thank you.

[task 2017-09-22T05:20:07.705119Z] Starting new HTTPS connection (1): hg.mozilla.org
[task 2017-09-22T05:20:07.840394Z] "GET /integration/mozilla-inbound/json-automationrelevance/9fb93fcf2b8eece6b7b23d12e29cbacc16e90b0c HTTP/1.1" 200 None
[task 2017-09-22T05:20:07.846223Z] 2 commits influencing task scheduling:
[task 2017-09-22T05:20:07.846363Z]  cfb739b281f8 Bug 1397145 - Remove the support for moz-blob type in XHR, r=smaug
[task 2017-09-22T05:20:07.846493Z]  9fb93fcf2b8e Bug 1397151 - Remove the support for moz-chunk-text type in XHR, r=smaug
[task 2017-09-22T05:20:14.508177Z] Reading file: /builds/worker/checkouts/gecko/moz.build
[task 2017-09-22T05:20:14.555000Z] Reading file: /builds/worker/checkouts/gecko/dom/moz.build
[task 2017-09-22T05:20:14.564201Z] Reading file: /builds/worker/checkouts/gecko/dom/base/moz.build
[task 2017-09-22T05:20:14.588638Z] Reading file: /builds/worker/checkouts/gecko/dom/webidl/moz.build
[task 2017-09-22T05:20:14.605659Z] Reading file: /builds/worker/checkouts/gecko/dom/xhr/moz.build
[task 2017-09-22T05:20:14.625454Z] Reading file: /builds/worker/checkouts/gecko/testing/moz.build
[task 2017-09-22T05:20:14.633856Z] Reading file: /builds/worker/checkouts/gecko/testing/web-platform/moz.build
[task 2017-09-22T05:20:14.782912Z] Traceback (most recent call last):
[task 2017-09-22T05:20:14.783020Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 154, in taskgraph_decision
[task 2017-09-22T05:20:14.783079Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-09-22T05:20:14.783148Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 135, in taskgraph_decision
[task 2017-09-22T05:20:14.783200Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-09-22T05:20:14.783291Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 178, in morphed_task_graph
[task 2017-09-22T05:20:14.783344Z]     return self._run_until('morphed_task_graph')
[task 2017-09-22T05:20:14.783413Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 283, in _run_until
[task 2017-09-22T05:20:14.783448Z]     k, v = self._run.next()
[task 2017-09-22T05:20:14.783499Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 270, in _run
[task 2017-09-22T05:20:14.783534Z]     do_not_optimize)
[task 2017-09-22T05:20:14.783594Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 55, in optimize_task_graph
[task 2017-09-22T05:20:14.783633Z]     do_not_optimize=do_not_optimize)
[task 2017-09-22T05:20:14.783691Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 126, in remove_tasks
[task 2017-09-22T05:20:14.783735Z]     if opt.should_remove_task(task, params, arg):
[task 2017-09-22T05:20:14.783796Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 376, in should_remove_task
[task 2017-09-22T05:20:14.783854Z]     scheduled = self.scheduled_by_push(params['head_repository'], params['head_rev'])
[task 2017-09-22T05:20:14.783916Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/util.py", line 944, in method_call
[task 2017-09-22T05:20:14.783958Z]     cache[args] = self.func(instance, *args)
[task 2017-09-22T05:20:14.784016Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 367, in scheduled_by_push
[task 2017-09-22T05:20:14.784063Z]     for p, m in rdr.files_info(changed_files).items():
[task 2017-09-22T05:20:14.784133Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/frontend/reader.py", line 1342, in files_info
[task 2017-09-22T05:20:14.784182Z]     paths, _ = self.read_relevant_mozbuilds(paths)
[task 2017-09-22T05:20:14.784258Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/frontend/reader.py", line 1306, in read_relevant_mozbuilds
[task 2017-09-22T05:20:14.784308Z]     self.config, metadata=metadata):
[task 2017-09-22T05:20:14.784391Z]   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/frontend/reader.py", line 1061, in read_mozbuild
[task 2017-09-22T05:20:14.784434Z]     raise bre
[task 2017-09-22T05:20:14.785111Z] BuildReaderError: 
[task 2017-09-22T05:20:14.785155Z] ==============================
[task 2017-09-22T05:20:14.785188Z] ERROR PROCESSING MOZBUILD FILE
[task 2017-09-22T05:20:14.785220Z] ==============================
[task 2017-09-22T05:20:14.785244Z] 
[task 2017-09-22T05:20:14.785290Z] The error occurred while processing the following file:
[task 2017-09-22T05:20:14.785316Z] 
[task 2017-09-22T05:20:14.785361Z]     /builds/worker/checkouts/gecko/testing/web-platform/moz.build
[task 2017-09-22T05:20:14.785393Z] 
[task 2017-09-22T05:20:14.785435Z] The error was triggered on line 9 of this file:
[task 2017-09-22T05:20:14.785460Z] 
[task 2017-09-22T05:20:14.785498Z]     ('mozilla/meta/MANIFEST.json', 'mozilla/tests/')
[task 2017-09-22T05:20:14.785523Z] 
[task 2017-09-22T05:20:14.785578Z] An error was encountered as part of executing the file itself. The error appears to be the fault of the script.
[task 2017-09-22T05:20:14.785609Z] 
[task 2017-09-22T05:20:14.785642Z] The error as reported by Python is:
[task 2017-09-22T05:20:14.785663Z] 
[task 2017-09-22T05:20:14.785710Z]     ["UnboundLocalError: local variable 'wptmanifest' referenced before assignment\n"]
[task 2017-09-22T05:20:14.785731Z] 
[taskcluster 2017-09-22 05:20:15.199Z] === Task Finished ===
[taskcluster 2017-09-22 05:20:23.795Z] Unsuccessful task run with exit code: 1 completed in 45.948 seconds

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This has hit several times, e.g. the merges, and also fails if retriggered. Does it fail for every wpt push?

Comment 2

[Kartikaya Gupta (email:[e-mail])]

I am also still seeing this in >50% of my recent try pushes and it's eating into my productivity.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Now I'm just doing all my try pushes with the sparse-profile change backed out and that seems to produce much better results.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #3)
> Now I'm just doing all my try pushes with the sparse-profile change backed
> out and that seems to produce much better results.

To be clear the change I backed out is https://hg.mozilla.org/integration/autoland/rev/bffb6b9d72128317e400b0bc35dcde97c7068250

Comment 5

[James Graham [:jgraham]]

So what I know is that the code that's failing tries to execfile testing/web-platform/tests/tools/localpaths.py so that the paths necessary to run wpt bits are in sys.path. That seems to be failing intermittently? At least I assume that's why wptmanifest is referenced before assignment. But the file it's running hasn't changed since August, so I don't think that can directly  be the problem.

Based on kats' observation above, I guess we are missing this file in the sparse checkout, or something?

Comment 6

[Gregory Szorc [:gps]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #4)
> (In reply to Kartikaya Gupta (email:[e-mail]) from comment #3)
> > Now I'm just doing all my try pushes with the sparse-profile change backed
> > out and that seems to produce much better results.
> 
> To be clear the change I backed out is
> https://hg.mozilla.org/integration/autoland/rev/
> bffb6b9d72128317e400b0bc35dcde97c7068250

That commit should have nothing to do with the decision task.

And this isn't intermittent per se. Intermittent means a supposedly deterministic behavior isn't.

Different decision tasks against different changesets fundamentally change how the decision task operates. So this is a bug that manifests when specific conditions are met.

Anyway, I think I know what the problem is.

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bug 1402233 - Add web platform tools files to taskgraph sparse profile

This is needed so we can import the WPT manifest parser.

Like for the reftest parser, this should only be temporary until we no
longer parse test manifests as part of Files() evaluation.

Review commit: https://reviewboard.mozilla.org/r/182764/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/182764/

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8911276 [details]
Bug 1402233 - Add web platform tools files to taskgraph sparse profile

https://reviewboard.mozilla.org/r/182764/#review188014

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/13ec36bee625
Add web platform tools files to taskgraph sparse profile r=mshal

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/13ec36bee625

Comment 12

[Intermittent Failures Robot]

13 failures in 943 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 5
* mozilla-inbound: 5
* autoland: 3

Platform breakdown:
* gecko-decision: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1402233&startday=2017-09-18&endday=2017-09-24&tree=all