1543663 - Perma cram try Error running mach | NameError: global name 'errno' is not defined

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Andreea Pavel [:apavel]]

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=8428f4ba4a51304584ade9bcbbcb2d555807b406&selectedJob=239614623

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239614623&repo=autoland&lineNumber=518

[task 2019-04-11T10:06:17.615Z] + Error running mach:
[task 2019-04-11T10:06:17.615Z]
[task 2019-04-11T10:06:17.615Z] - Pushed via mach try syntax

[task 2019-04-11T10:06:17.618Z] + The details of the failure are as follows:
[task 2019-04-11T10:06:17.618Z] +
[task 2019-04-11T10:06:17.619Z] + NameError: global name 'errno' is not defined
[task 2019-04-11T10:06:17.619Z] +
[task 2019-04-11T10:06:17.619Z] + File "/builds/worker/checkouts/gecko/tools/tryselect/mach_commands.py", line 373, in try_syntax
[task 2019-04-11T10:06:17.619Z] + return self.run(**kwargs)
[task 2019-04-11T10:06:17.620Z] + File "/builds/worker/checkouts/gecko/tools/tryselect/mach_commands.py", line 180, in run
[task 2019-04-11T10:06:17.620Z] + return mod.run(**kwargs)
[task 2019-04-11T10:06:17.620Z] + File "/builds/worker/checkouts/gecko/tools/tryselect/selectors/syntax.py", line 612, in run
[task 2019-04-11T10:06:17.620Z] + return at.run(**kwargs)
[task 2019-04-11T10:06:17.621Z] + File "/builds/worker/checkouts/gecko/tools/tryselect/selectors/syntax.py", line 561, in run
[task 2019-04-11T10:06:17.621Z] + paths_by_flavor = self.paths_by_flavor(paths=paths, tags=tags)
[task 2019-04-11T10:06:17.621Z] + File "/builds/worker/checkouts/gecko/tools/tryselect/selectors/syntax.py", line 333, in paths_by_flavor
[task 2019-04-11T10:06:17.621Z] + tests = list(self.resolver.resolve_tests(paths=paths,
[task 2019-04-11T10:06:17.622Z] + File "/builds/worker/checkouts/gecko/tools/tryselect/selectors/syntax.py", line 320, in resolver
[task 2019-04-11T10:06:17.622Z] + self._resolver = TestResolver.from_environment(cwd=here)
[task 2019-04-11T10:06:17.622Z] + File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 184, in from_environment
[task 2019-04-11T10:06:17.622Z] + mozconfig=mozconfig)
[task 2019-04-11T10:06:17.623Z] + File "/builds/worker/checkouts/gecko/testing/mozbase/moztest/moztest/resolve.py", line 548, in init
[task 2019-04-11T10:06:17.623Z] + 'backend.TestManifestBackend'
[task 2019-04-11T10:06:17.623Z] + File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 241, in backend_out_of_date
[task 2019-04-11T10:06:17.624Z] + return self.build_out_of_date(backend_file, dep_file)
[task 2019-04-11T10:06:17.624Z] + File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 217, in build_out_of_date
[task 2019-04-11T10:06:17.624Z] + if e.errno == errno.ENOENT:
[task 2019-04-11T10:06:17.625Z] + [1]
[task 2019-04-11T10:06:17.625Z] $ ls $MOZBUILD_STATE_PATH/autotry.ini
[task 2019-04-11T10:06:17.625Z] */mozbuild/autotry.ini': No such file or directory (glob)
[task 2019-04-11T10:06:17.625Z] [2]
[task 2019-04-11T10:06:17.625Z]
[task 2019-04-11T10:06:17.625Z] # Ran 5 tests, 0 skipped, 2 failed.
[task 2019-04-11T10:06:17.731Z] Test configuration changed. Regenerating backend.
[task 2019-04-11T10:06:17.731Z] No build detected, test metadata may be incomplete.
[taskcluster 2019-04-11 10:06:18.163Z] === Task Finished ===
[taskcluster 2019-04-11 10:06:18.164Z] Unsuccessful task run with exit code: 1 completed in 341.399 seconds

Comment 1

[Andrew Halberstadt [:ahal]]

The "tryselect" python-tests are also affected by this:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239680791&repo=try&lineNumber=336

So it seems to be all tests under tools/tryselect no matter the suite.

Comment 2

[Michael Shal [:mshal]]

Attachment: Bug 1543663 - Add missing 'import errno' to fix some python-tests; r?ahal

Bug 1537574 moved some code that uses errno from building.py to base.py,
but the latter didn't already import errno.

Comment 3

[Michael Shal [:mshal]]

I'm kinda surprised the linter didn't pick that up. In any case, just adding the missing import seems to fix './mach python-test tools/tryselect/test' locally for me. Is there a certain try job I should run to more completely verify this?

Comment 4

[Andrew Halberstadt [:ahal]]

Nobody has enabled the linter on mozbuild yet:
https://searchfox.org/mozilla-central/source/.flake8#28

If that fixes it locally, I'd be pretty confident of it fixing it in CI too. But you can run:

./mach try fuzzy -q "'tryselect"

if you want to be safe.

Comment 5

[Michael Shal [:mshal]]

Looks like some of the tests are still failing, though I can't tell if it's related to bug 1537574. The errno import issue is fixed, at least: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7627ca250582c6e93c483c9f97dfd0b9d82145df

Comment 6

[Andrew Halberstadt [:ahal]]

I'd strongly suspect bug 1537574 since the TestResolver (which is what was using the TestManifestBackend) is in the tracebacks. Also I haven't landed any changes to that module since bug 1537574 landed.

Comment 7

[Andrew Halberstadt [:ahal]]

(In reply to Andrew Halberstadt [:ahal] from comment #6)

I'd strongly suspect bug 1537574 since the TestResolver (which is what was using the TestManifestBackend) is in the tracebacks. Also I haven't landed any changes to that module since bug 1537574 landed.

Also the "cram" task isn't a real failure. But I think your changes caused some output that will either need to be suppressed, or we'll need to update the tests.

Comment 8

[Michael Shal [:mshal]]

For cram it looks like mozbuild's out-of-date logic differs from make's logic. Make was calling $(wildcard) on all inputs, so if an input is missing, it would not cause the backend to regenerate. I've verified this locally and am double-checking on try.

Comment 9

[Michael Shal [:mshal]]

Attachment: Bug 1543663 - Add $HOME/.mozbuild/fzf/bin to PATH for tryselect cramtests; r?ahal

This helps run the tests locally if fzf is normally installed in
$HOME/.mozbuild. Since the tests set MOZBUILD_STATE_PATH to a temporary
directory, fzf_bootstrap() can't find fzf in the HOME location unless it
is added to PATH.

Depends on D27125

Comment 10

[Michael Shal [:mshal]]

Attachment: Bug 1543663 - Create an empty config.status for TestManifestBackend; r?chmanchester

Bug 1537574 moved the backend-out-of-date logic from make to mozbuild.
Make had been using $(wildcard) on the list of inputs, meaning
non-existent inputs were pruned and would not cause the backend to be
out of date. Mozbuild should preserve this behavior, otherwise a backend
may be continually out-of-date in some cases.

For example, the TestManifestBackend depends on config.status, since a
configure run may change the set of supported tests. However, if the
TestManifestBackend is created without a configure run, config.status
doesn't exist, and the backend is perpetually out-of-date unless we
preserve make's $(wildcard) semantics. To do so we simply need to ignore
missing inputs.

Depends on D27194

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bff1edac1f4b
Add missing 'import errno' to fix some python-tests; r=ahal
https://hg.mozilla.org/integration/autoland/rev/051bf3692bd6
Add $HOME/.mozbuild/fzf/bin to PATH for tryselect cramtests; r=ahal
https://hg.mozilla.org/integration/autoland/rev/c6d76bbda8fe
Create an empty config.status for TestManifestBackend; r=chmanchester

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/bff1edac1f4b
https://hg.mozilla.org/mozilla-central/rev/051bf3692bd6
https://hg.mozilla.org/mozilla-central/rev/c6d76bbda8fe

Comment 13

[Intermittent Failures Robot]

19 failures in 3025 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 3
• mozilla-inbound: 7

Platform breakdown:

• linux64: 16
• windows10-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543663&startday=2019-04-08&endday=2019-04-14&tree=all