1304593 - Frequent taskcluster-only Windows build Intermittent 'make -k check' did not run successfully. Please check log for errors. after recipe for target 'mozlint/test/test_types.py-run' failed

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=3845693&repo=autoland

https://queue.taskcluster.net/v1/task/dMMESqHnT-miRpNNFYfjng/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Intermittent Failures Robot]

16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 7
* autoland: 6
* try: 1
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* windowsxp: 10
* windows8-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1304593&startday=2016-09-22&endday=2016-09-22&tree=all

Comment 2

[Intermittent Failures Robot]

38 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 16
* autoland: 16
* try: 3
* mozilla-central: 1
* mozilla-aurora: 1
* fx-team: 1

Platform breakdown:
* windowsxp: 25
* windows8-64: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1304593&startday=2016-09-19&endday=2016-09-25&tree=all

Comment 3

[Andrew Halberstadt [:ahal]]

Weird, so test_types.py runs and passes but then make check tries to run "test_types.py-run".. I have no idea what this file is or why make check is trying to run it. I'm investigating.

Comment 4

[Andrew Halberstadt [:ahal]]

I found where -run is getting added:
https://dxr.mozilla.org/mozilla-central/source/config/rules.mk#103

However it looks like my original assessment that the py-run file was being mistaken for a test is wrong. For some reason make check thinks the pytest test failed, even though all tests say PASSED.

I'm very tempted to ignore this bug and pull the python unittests out of make check instead. With the vcs checkout work gps has done recently, it might actually be fairly trivial to do this.

Comment 5

[Andrew Halberstadt [:ahal]]

So I wrote a patch [1] that changes make check to call |mach python-test| instead of doing the weird '-run' hack, but that resulted in a much more verbose log [2]. And for some reason the mozbase tests dump structured jsonl messages to the log which is probably not acceptable. Sigh, I guess I should just figure out why make check thinks that test is failing. I'm still at a loss, and not sure the best way to test this out.

[1] https://hg.mozilla.org/try/rev/11f9065b7f33bfce8880604dfa316b4721463082
[2] https://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-fb02ce373e19254df5546a8d1911111553a745b5/try-linux64/try-linux64-bm78-try1-build16722.txt.gz

Comment 6

[Andrew Halberstadt [:ahal]]

Actually, I just wasn't paying close attention. Looks like the output with and without my patch in [1] is nearly the same. I can add --log-no-times to the mach invocation to make it even closer (though not quite identical). I'll get a patch up.

Comment 7

[Andrew Halberstadt [:ahal]]

So my patch still caused problems on osx.. but it seems like this hasn't happened in a couple of days? The last occurrence seems to have been Sept. 25th, so I think I'm going to sit on my patch for bit and keep an eye out for this again.

Comment 8

[Phil Ringnalda (:philor)]

Busted and hidden by the hg cert change, just got visible again this morning.

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1304593 - Fix broken test_entry_point.py in mach,

Review commit: https://reviewboard.mozilla.org/r/82182/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/82182/

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1304593 - Use |mach python-test| to run python unittests from make check,

Review commit: https://reviewboard.mozilla.org/r/82184/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/82184/

Comment 11

[(not currently active) Ted Mielczarek]

Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review80940

::: config/rules.mk:107
(Diff revision 1)
> +.PHONY: run-python-unittests
>  
> -RUN_PYTHON_UNIT_TESTS := $(addsuffix -run,$(PYTHON_UNIT_TESTS))
> +check:: run-python-unittests
>  
> -.PHONY: $(RUN_PYTHON_UNIT_TESTS)
> -
> +run-python-unittests::
> +	@$(MOZILLA_DIR)/mach --log-no-times python-test $(PYTHON_UNIT_TESTS)

I think we'd be better off just removing the rules here, and sticking a top-level `check::` target, either in $topsrcdir/Makefile.in or testsuite-targets.mk that invokes `mach python-test` to run all the Python unittests in one go. (Then maybe we could figure out if there's anything else useful hanging off of `make check` and maybe just stop calling it?)

You should also sanity check that the `python-test` target exits with a non-zero return code when tests fail.

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1304593 - Take universal binaries into account when detecting objdir mismatches,

When attempting to run mach from a make target, there were failures on OSX due to an objdir
mismatch:
https://dxr.mozilla.org/mozilla-central/rev/7c576fe3279d87543f0a03b844eba7bc215e17f1/python/mozbuild/mozbuild/base.py#656

The two detected objdirs were "<objdir>" and "<objdir>/x86_64". I believe this should not
count as a mismatch, otherwise it would not be possible to run a mach command from the
x86_64 directory.

Review commit: https://reviewboard.mozilla.org/r/82776/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/82776/

Comment 13

[Andrew Halberstadt [:ahal]]

Comment on attachment 8796274 [details]
Bug 1304593 - Fix broken test_entry_point.py in mach,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/82182/diff/1-2/

Comment 14

[Andrew Halberstadt [:ahal]]

Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/82184/diff/1-2/

Comment 15

[Andrew Halberstadt [:ahal]]

Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review80940

> I think we'd be better off just removing the rules here, and sticking a top-level `check::` target, either in $topsrcdir/Makefile.in or testsuite-targets.mk that invokes `mach python-test` to run all the Python unittests in one go. (Then maybe we could figure out if there's anything else useful hanging off of `make check` and maybe just stop calling it?)
> 
> You should also sanity check that the `python-test` target exits with a non-zero return code when tests fail.

There's still a fair bit of misc things other than python unittests using make check, so we'll still need to do a decent chunk of work before we can stop calling it altogether. And python-test does return non-zero on failure.

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review81416

Nice!

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8796274 [details]
Bug 1304593 - Fix broken test_entry_point.py in mach,

https://reviewboard.mozilla.org/r/82182/#review81422

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8797177 [details]
Bug 1304593 - Take universal binaries into account when detecting objdir mismatches,

https://reviewboard.mozilla.org/r/82776/#review81424

I can't remember what I was doing exactly, but I managed to hit this very exception on try in the past 2 weeks. Yay for fixing it!

Comment 19

[Gregory Szorc [:gps]]

Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review81426

I think the code in the moz.build backend to emit `PYTHON_UNIT_TESTS` in generated backend.mk files can also be removed, as I think this was the only reference to `PYTHON_UNIT_TESTS` in make files.

Comment 20

[Andrew Halberstadt [:ahal]]

Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review81426

I filed bug 1307200 as a follow up because I couldn't quite figure out where PYTHON_UNIT_TESTS was being written. I found recursivemake.py, but the _process_test_manifests function doesn't seem to write anything to backend.mk. So I'm going to land what I have for now and punt on this.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/180d4a79ad6f
Fix broken test_entry_point.py in mach, r=gps
https://hg.mozilla.org/integration/autoland/rev/9d5982f805c9
Take universal binaries into account when detecting objdir mismatches, r=gps
https://hg.mozilla.org/integration/autoland/rev/27f488a1f9ef
Use |mach python-test| to run python unittests from make check, r=ted

Comment 22

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/180d4a79ad6f
https://hg.mozilla.org/mozilla-central/rev/9d5982f805c9
https://hg.mozilla.org/mozilla-central/rev/27f488a1f9ef

Comment 23

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f4ca19daa701
https://hg.mozilla.org/releases/mozilla-aurora/rev/70218f843e80
https://hg.mozilla.org/releases/mozilla-aurora/rev/a55eeb63cbcc