Closed Bug 1304593 Opened 3 years ago Closed 3 years ago

Frequent taskcluster-only Windows build Intermittent 'make -k check' did not run successfully. Please check log for errors. after recipe for target 'mozlint/test/test_types.py-run' failed

Categories

(Taskcluster :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla52

People

(Reporter: intermittent-bug-filer, Assigned: ahal)

References

Details

(Keywords: intermittent-failure)

Attachments

(3 files)

Weird, so test_types.py runs and passes but then make check tries to run "test_types.py-run".. I have no idea what this file is or why make check is trying to run it. I'm investigating.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
I found where -run is getting added:
https://dxr.mozilla.org/mozilla-central/source/config/rules.mk#103

However it looks like my original assessment that the py-run file was being mistaken for a test is wrong. For some reason make check thinks the pytest test failed, even though all tests say PASSED.

I'm very tempted to ignore this bug and pull the python unittests out of make check instead. With the vcs checkout work gps has done recently, it might actually be fairly trivial to do this.
So I wrote a patch [1] that changes make check to call |mach python-test| instead of doing the weird '-run' hack, but that resulted in a much more verbose log [2]. And for some reason the mozbase tests dump structured jsonl messages to the log which is probably not acceptable. Sigh, I guess I should just figure out why make check thinks that test is failing. I'm still at a loss, and not sure the best way to test this out.

[1] https://hg.mozilla.org/try/rev/11f9065b7f33bfce8880604dfa316b4721463082
[2] https://archive.mozilla.org/pub/firefox/try-builds/ahalberstadt@mozilla.com-fb02ce373e19254df5546a8d1911111553a745b5/try-linux64/try-linux64-bm78-try1-build16722.txt.gz
Actually, I just wasn't paying close attention. Looks like the output with and without my patch in [1] is nearly the same. I can add --log-no-times to the mach invocation to make it even closer (though not quite identical). I'll get a patch up.
So my patch still caused problems on osx.. but it seems like this hasn't happened in a couple of days? The last occurrence seems to have been Sept. 25th, so I think I'm going to sit on my patch for bit and keep an eye out for this again.
Busted and hidden by the hg cert change, just got visible again this morning.
Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review80940

::: config/rules.mk:107
(Diff revision 1)
> +.PHONY: run-python-unittests
>  
> -RUN_PYTHON_UNIT_TESTS := $(addsuffix -run,$(PYTHON_UNIT_TESTS))
> +check:: run-python-unittests
>  
> -.PHONY: $(RUN_PYTHON_UNIT_TESTS)
> -
> +run-python-unittests::
> +	@$(MOZILLA_DIR)/mach --log-no-times python-test $(PYTHON_UNIT_TESTS)

I think we'd be better off just removing the rules here, and sticking a top-level `check::` target, either in $topsrcdir/Makefile.in or testsuite-targets.mk that invokes `mach python-test` to run all the Python unittests in one go. (Then maybe we could figure out if there's anything else useful hanging off of `make check` and maybe just stop calling it?)

You should also sanity check that the `python-test` target exits with a non-zero return code when tests fail.
Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review80940

> I think we'd be better off just removing the rules here, and sticking a top-level `check::` target, either in $topsrcdir/Makefile.in or testsuite-targets.mk that invokes `mach python-test` to run all the Python unittests in one go. (Then maybe we could figure out if there's anything else useful hanging off of `make check` and maybe just stop calling it?)
> 
> You should also sanity check that the `python-test` target exits with a non-zero return code when tests fail.

There's still a fair bit of misc things other than python unittests using make check, so we'll still need to do a decent chunk of work before we can stop calling it altogether. And python-test does return non-zero on failure.
Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review81416

Nice!
Attachment #8796275 - Flags: review?(ted) → review+
Comment on attachment 8796274 [details]
Bug 1304593 - Fix broken test_entry_point.py in mach,

https://reviewboard.mozilla.org/r/82182/#review81422
Attachment #8796274 - Flags: review?(gps) → review+
Comment on attachment 8797177 [details]
Bug 1304593 - Take universal binaries into account when detecting objdir mismatches,

https://reviewboard.mozilla.org/r/82776/#review81424

I can't remember what I was doing exactly, but I managed to hit this very exception on try in the past 2 weeks. Yay for fixing it!
Attachment #8797177 - Flags: review?(gps) → review+
Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review81426

I think the code in the moz.build backend to emit `PYTHON_UNIT_TESTS` in generated backend.mk files can also be removed, as I think this was the only reference to `PYTHON_UNIT_TESTS` in make files.
Blocks: 1307200
Comment on attachment 8796275 [details]
Bug 1304593 - Use |mach python-test| to run python unittests from make check,

https://reviewboard.mozilla.org/r/82184/#review81426

I filed bug 1307200 as a follow up because I couldn't quite figure out where PYTHON_UNIT_TESTS was being written. I found recursivemake.py, but the _process_test_manifests function doesn't seem to write anything to backend.mk. So I'm going to land what I have for now and punt on this.
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/180d4a79ad6f
Fix broken test_entry_point.py in mach, r=gps
https://hg.mozilla.org/integration/autoland/rev/9d5982f805c9
Take universal binaries into account when detecting objdir mismatches, r=gps
https://hg.mozilla.org/integration/autoland/rev/27f488a1f9ef
Use |mach python-test| to run python unittests from make check, r=ted
https://hg.mozilla.org/mozilla-central/rev/180d4a79ad6f
https://hg.mozilla.org/mozilla-central/rev/9d5982f805c9
https://hg.mozilla.org/mozilla-central/rev/27f488a1f9ef
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
Depends on: 1309060
You need to log in before you can comment on or make changes to this bug.