1462489 - Run mochitests from the tup build

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Chris Manchester (limited bugmail, email directly)]

I've been able to run xpcshell tests from a tup build just fine, but the mochitests fail due to: "invalid setup: missing mochikit extension"

Comment 1

[Chris Manchester (limited bugmail, email directly)]

Aha, there's a stray libs rule that copies mochijar from dist/xpi-stage to _tests/testing/mochitest: https://searchfox.org/mozilla-central/rev/da499aac682d0bbda5829327b60a865cbc491611/testing/mochitest/Makefile.in#11

Comment 2

[(not currently active) Ted Mielczarek]

I wonder if we could just fix the default path in runtests.py to look in dist/xpi-stage, and fix the test packaging bits to copy it out of there into the current place in the test package?

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> I wonder if we could just fix the default path in runtests.py to look in
> dist/xpi-stage, and fix the test packaging bits to copy it out of there into
> the current place in the test package?

+1. I think the main reason we put everything in those staging test dirs is because it was previously a PITA to create the test package in testsuite-targets.mk. But now with test_archive.py, it's trivial to pull in things from all over the sourcedir/objdir.

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1462489 - Process some chrome manifest entries from dist/xpi-stage from the tup backend.

Review commit: https://reviewboard.mozilla.org/r/246616/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/246616/

Comment 5

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1462489 - Use test addons from dist/xpi-stage when running mochitests locally.

Review commit: https://reviewboard.mozilla.org/r/246618/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/246618/

Comment 6

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1462489 - Put packaged test addons in the test package with the test packager rather than Makefile.in

Review commit: https://reviewboard.mozilla.org/r/246620/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/246620/

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1462489 - Re-generate the build backend when a build backend output is missing.

Review commit: https://reviewboard.mozilla.org/r/246622/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/246622/

Comment 8

[Andrew Halberstadt [:ahal]]

Comment on attachment 8980455 [details]
Bug 1462489 - Use test addons from dist/xpi-stage when running mochitests locally.

https://reviewboard.mozilla.org/r/246618/#review252926

Lgtm!

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8980454 [details]
Bug 1462489 - Process some chrome manifest entries from dist/xpi-stage from the tup backend.

https://reviewboard.mozilla.org/r/246616/#review254188

Comment 10

[Michael Shal [:mshal]]

Comment on attachment 8980456 [details]
Bug 1462489 - Put packaged test addons in the test package with the test packager rather than Makefile.in

https://reviewboard.mozilla.org/r/246620/#review254190

::: python/mozbuild/mozbuild/action/test_archive.py:372
(Diff revision 1)
> +            'source': buildconfig.topobjdir,
> +            'base': 'dist/xpi-stage',
> +            'pattern': 'mochijar/**',
> +            'dest': 'mochitest'
> +        },
> +        {

Is this entry intended to be something else? It's a duplicate of the one above.

Comment 11

[Michael Shal [:mshal]]

Comment on attachment 8980457 [details]
Bug 1462489 - Re-generate the build backend when a build backend output is missing.

https://reviewboard.mozilla.org/r/246622/#review254194

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980456 [details]
Bug 1462489 - Put packaged test addons in the test package with the test packager rather than Makefile.in

https://reviewboard.mozilla.org/r/246620/#review254190

> Is this entry intended to be something else? It's a duplicate of the one above.

Nope, it's just duplicated for no reason. Thanks for noticing that!

Comment 13

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980454 [details]
Bug 1462489 - Process some chrome manifest entries from dist/xpi-stage from the tup backend.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246616/diff/1-2/

Comment 14

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980455 [details]
Bug 1462489 - Use test addons from dist/xpi-stage when running mochitests locally.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246618/diff/1-2/

Comment 15

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980456 [details]
Bug 1462489 - Put packaged test addons in the test package with the test packager rather than Makefile.in

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246620/diff/1-2/

Comment 16

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980457 [details]
Bug 1462489 - Re-generate the build backend when a build backend output is missing.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246622/diff/1-2/

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dfed23857cea
Process some chrome manifest entries from dist/xpi-stage from the tup backend. r=mshal
https://hg.mozilla.org/integration/autoland/rev/3cd6a8ee8f66
Use test addons from dist/xpi-stage when running mochitests locally. r=ahal
https://hg.mozilla.org/integration/autoland/rev/daa33716b140
Put packaged test addons in the test package with the test packager rather than Makefile.in r=mshal
https://hg.mozilla.org/integration/autoland/rev/41cf0d53a95c
Re-generate the build backend when a build backend output is missing. r=mshal

Comment 18

[Dorel Luca [:dluca]]

Backed out 4 changesets (bug 1462489) for breaking mochitest on testing/mochitest/tests/python/test_basic_mochitest_plain.py

Log:
https://treeherder.mozilla.org/logviewer.html#?job_id=181011268&repo=autoland&lineNumber=355

 0:11.62 ===================== 1 passed, 1 warnings in 0.28 seconds =====================
[task 2018-05-30T23:11:04.028Z]  0:11.72 /builds/worker/checkouts/gecko/testing/mochitest/tests/python/test_basic_mochitest_plain.py
[task 2018-05-30T23:11:04.639Z]  0:12.33 ============================= test session starts ==============================
[task 2018-05-30T23:11:04.639Z]  0:12.33 platform linux2 -- Python 2.7.12, pytest-3.2.5, py-1.5.3, pluggy-0.4.0 -- /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2018-05-30T23:11:04.639Z]  0:12.33 rootdir: /builds/worker/checkouts/gecko, inifile:
[task 2018-05-30T23:11:04.667Z]  0:12.36 collecting ... collected 6 items
[task 2018-05-30T23:11:04.668Z]  0:12.36 
[task 2018-05-30T23:11:13.553Z]  0:21.24 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_pass TEST-UNEXPECTED-FAIL
[task 2018-05-30T23:11:21.572Z]  0:29.26 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_fail TEST-UNEXPECTED-FAIL
[task 2018-05-30T23:11:21.573Z]  0:29.26 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_crash SKIPPED
[task 2018-05-30T23:11:31.213Z]  0:38.90 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_asan TEST-UNEXPECTED-FAIL
[task 2018-05-30T23:11:31.213Z]  0:38.90 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_assertion SKIPPED
[task 2018-05-30T23:11:31.216Z]  0:38.91 testing/mochitest/tests/python/test_basic_mochitest_plain.py::test_output_leak SKIPPED
[task 2018-05-30T23:11:31.217Z]  0:38.91 
[task 2018-05-30T23:11:31.217Z]  0:38.91 =================================== FAILURES ===================================
[task 2018-05-30T23:11:31.217Z]  0:38.91 _______________________________ test_output_pass _______________________________
[task 2018-05-30T23:11:31.218Z]  0:38.91 
[task 2018-05-30T23:11:31.218Z]  0:38.91 runtests = <function inner at 0x7f7e2e05c500>

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=41cf0d53a95cc8966659a5eafb93d3826b97ea62&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified

Backout:
https://hg.mozilla.org/integration/autoland/rev/aca4b42750423341b74ba8cfbd77d27069f70fb2

Comment 19

[Chris Manchester (limited bugmail, email directly)]

I would have fixed it this anyway, so it's not really much of an inconvenience, but these tests are marked as tier-2, so as far as I know this failure should not have resulted in a backout.

Comment 20

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980455 [details]
Bug 1462489 - Use test addons from dist/xpi-stage when running mochitests locally.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246618/diff/2-3/

Comment 21

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980456 [details]
Bug 1462489 - Put packaged test addons in the test package with the test packager rather than Makefile.in

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246620/diff/2-3/

Comment 22

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8980457 [details]
Bug 1462489 - Re-generate the build backend when a build backend output is missing.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246622/diff/2-3/

Comment 23

[Andrew Halberstadt [:ahal]]

Comment on attachment 8980455 [details]
Bug 1462489 - Use test addons from dist/xpi-stage when running mochitests locally.

https://reviewboard.mozilla.org/r/246618/#review254392

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7f3e1d87baa3
Process some chrome manifest entries from dist/xpi-stage from the tup backend. r=mshal
https://hg.mozilla.org/integration/autoland/rev/5133f89842f6
Use test addons from dist/xpi-stage when running mochitests locally. r=ahal
https://hg.mozilla.org/integration/autoland/rev/ce2105ffae17
Put packaged test addons in the test package with the test packager rather than Makefile.in r=mshal
https://hg.mozilla.org/integration/autoland/rev/e0b837323b48
Re-generate the build backend when a build backend output is missing. r=mshal

Comment 25

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/7f3e1d87baa3
https://hg.mozilla.org/mozilla-central/rev/5133f89842f6
https://hg.mozilla.org/mozilla-central/rev/ce2105ffae17
https://hg.mozilla.org/mozilla-central/rev/e0b837323b48