1070057 - Make TestResolver.resolve_tests filter by subsuite

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nick Alexander :nalexander [he/him]]

I'm implementing a new test suite: Android instrumentation tests.  This suite has natural subsuites that correspond to different instrumentation APK files; right now we have browser and background; I expect to grow a few more.  I want to filter by subsuite both in my test runner (which parses manifests directly) and in my mach command (which uses all-tests.json and TestResolver).  I'd like to make TestResolver.resolve_tests filter by subsuite as well.

There's a little wrinkle with devtools being a subsuite but handled specially; but I don't think I need to touch that.

Comment 1

[(not currently active) Ted Mielczarek]

It would be nice if subsuites were handled the same across all test harnesses. Having two different methods of dealing with them sounds confusing.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> It would be nice if subsuites were handled the same across all test
> harnesses. Having two different methods of dealing with them sounds
> confusing.

I agree, but I don't understand this special processing for devtools and frankly I don't care to shave that yak.

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: Make TestResolver.resolve_tests filter by subsuite. r=ted

You can see the devtools funkyness just above.  I think the thing is
that devtools is a subsuite that for historical reasons is treated as
a different flavour.  Very confusing.

Comment 4

[Andrew Halberstadt [:ahal]]

I think it was actually a request from the devtools team to have their own symbol on tbpl but instead of implementing a new mochitest flavour it was done via manifest hacking (which I'm guessing required fewer buildbot related changes).

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 8492434 [details] [diff] [review]
Make TestResolver.resolve_tests filter by subsuite. r=ted

Review of attachment 8492434 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/testing.py
@@ +90,5 @@
>          ``xpcshell``.
> +
> +        If ``subsuite`` is a string, it will be used to filter returned tests
> +        to only be in the subsuite specified. A subsuite is something like
> +        ``browser`` or ``background``.

I'm not sure that the examples in the docstring are super-useful, given that they vary from suite to suite.

Comment 6

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/integration/fx-team/rev/b28b8ec17fba

Comment 7

[:Gijs (he/him)]

https://hg.mozilla.org/mozilla-central/rev/b28b8ec17fba