1141771 - Automation runs the wrong mochitests when asked to run an orphaned mochitest manifest

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Kelsey Gilbert [:jgilbert]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a75781ee857

We have an issue where the webgl subsuite of mochitests is run twice on CI: Once in mochitest-gl (explicitly requested) and once again in mochitest-1 as part of normal chunking.

Needless to say, this should only be run once, so I tried orphaning the subsuite manifest from its moz.build file. On Try though, it runs the wrong tests (lots of DOM tests?) when asked to run the webgl subsuite mochitest manifest.

Comment 1

[Kelsey Gilbert [:jgilbert]]

:ahal volunteer as knowing about runtest.py, which seems to be the problem.

The test logs show that we're asking for our manifest, but it seems to be being ignored?

Comment 2

[Andrew Halberstadt [:ahal]]

Removing that manifest from moz.build will stop it from getting copied into the tests.zip. So while it will stop them from getting run in mochitest-1, it will also stop them from running anywhere else.

It looks like runtests.py defaults back to the master mochitest.ini in the case that --manifest doesn't exist (which it doesn't in your try push):
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#2477

To me that is a bug, the harness should error out if the path pointed to by --manifest doesn't exist. We should add an existence check here:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mochitest_options.py#608

What you wanted to do was remove that manifest from MOCHITEST_MANIFESTS, but still copy it to the test bundle. I think you could do that by adding:
TEST_HARNESS_FILES.testing.mochitest += ['test/mochitest-subsuite-webgl.ini']

But there's another way that will give you a bit more flexibility. Mochitest has a concept of "subsuites". E.g mochitest-devtools is a subsuite of mochitest-browser-chrome. You basically need to add `subsuite='webgl'` to any test (or manifest in the DEFAULT section) that you want to run (see [1] for an example). This will prevent those tests from being run as part of the default set. Then you need to pass in --subsuite=webgl into runtests.py, this would involve a mozharness change [2].

[1] https://dxr.mozilla.org/mozilla-central/source/browser/devtools/animationinspector/test/browser.ini#2
[2] https://dxr.mozilla.org/build:mozharness/search?q=%22mochitest-gl%22%3A&case=true&redirect=true

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

some questions:
1) do we care that the webgl tests are in their own job?  They are stable and historical data shows that they do not catch unique failures.
2) if we want them in a separate job, then we should mark them with: "subsuite=webgl" and then the 'gl' jobs can run with that.

thoughts?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

for reference, bug 1083347 removes the unique way gl tests have been run on android, so whatever decision we make will work on android+desktop.

Comment 5

[Kelsey Gilbert [:jgilbert]]

(In reply to Joel Maher (:jmaher) from comment #3)
> some questions:
> 1) do we care that the webgl tests are in their own job?  They are stable
> and historical data shows that they do not catch unique failures.
> 2) if we want them in a separate job, then we should mark them with:
> "subsuite=webgl" and then the 'gl' jobs can run with that.
> 
> thoughts?

I would prefer to keep them as their own jobs. WebGL changes don't usually cause non-webgl test failures, and vice-versa. It's extremely handy to be able to ask for just webgl tests to run on Try, particularly since WebGL is not something I can test locally and assume it'll work on Try.

Comment 6

[Kelsey Gilbert [:jgilbert]]

So yes, marking them with subsuite=webgl is part of the path forward. We literally have a "mochitest-subsuite-webgl.ini" manifest, but it does not claim to be subsuite=webgl yet.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

great, then we should make all webgl tests have an entry in the [default] section of the .ini files:
subsute = webgl

then we can adjust the mozharness scripts to have "--subsuite webgl" in the commandline options for the 'gl' jobs.

Comment 8

[Kelsey Gilbert [:jgilbert]]

I filed bug 1143218 and supplied a patch for marking webgl tests with `subsuite`.