996235 - Need a more robust way for defining mochitest subsuites

Status: RESOLVED INCOMPLETE

(Testing :: Mochitest, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

As shown in bug 996003, it's currently possible for tests to slip through the cracks since the subsuite definition lives at the manifest level. It would be nice if we had a way to define the subsuite at a higher level so, for example, all tests found in browser/devtools can get that definition whether each manifest has the declaration or not.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure if there is a good solution here.  If browser/devtools/browser.ini had a bunch of include:manifest directives, we could make it work, but right now we just include all the directories in the master manifest.

I agree this is a problem in general, I just don't know the best solution.

Comment 2

[Ed Morley [:emorley]]

Could a test manifest not just inherit any subsuite deinitions that were defined in a manifest found in a directory at a higher up level in the tree?

Comment 3

[Ed Morley [:emorley]]

ie:

browser/devtools/browser.ini:
[DEFAULT]
subsuite = devtools

browser/devtools/debugger/test/browser.ini:
[DEFAULT]
# Unnecessary since inherits from the first manifest that specifies a subsuite, when traversing parent directories.
# subsuite = devtools

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

we don't have a concept of directories in the manifests, but we do have a hierarchy when it comes to the includes:

mochitest.ini:
[include:devtools.ini]
[include:browser/.../browser.ini]

devtools.ini:
subsuite = devtools

[include:browser/devtools/.../browser.ini]
[include:toolkit/devtools.../browser.ini]


In the above case all the includes in devtools would automatically be part of the subsuite devtools.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

We've lived with this for this long so I think it's safe to say it's not something we're likely to ever try to address.