1527167 - Don't filter performance tasks when specifying a path

Status: NEW

(Developer Infrastructure :: Try, defect, P3)

Description

[Bianca Danforth [:bdanforth]]

Steps to Reproduce:

1. Add some tests (and register them via moz.build(s)) to a testing directory.

• For example, following closing of Bug 1517083, I drop an extension XPI into ./testing/profiles/common/extensions and a moz.build file, *.ini files and test files into ./testing/extensions as described in that bug.

2. Push to Try with the following syntax, ostensibly to run the custom tests in ./testing/extensions and Raptor tp6 page load tests at the same time:

./mach try fuzzy testing/extensions -q "'test-linux64/opt-mochitest-browser-chrome-e10s" -q "linux64 opt raptor-tp6 firefox"

3. Observe in Treeherder (example) that only the first query argument was picked up and the performance tests are not run.

Expected Results:

Both the test(s) in ./testing/extensions and the Raptor tests should run on the Try server.

Actual Results:

Only the test(s) in ./testing/extensions run.

---

My understanding from ahal is that this is because:

when you specify a path, it filters down the list of tasks to only tasks that contain tests in that directory.

It would be very helpful if multiple queries could be passed into a single ./mach try fuzzy command where the provided path can be applied to only one or a subset of the queries rather than to all of them.

In my particular use case, I would like to be able to run custom tests in the ./testing/extensions directory and existing tests (such as browser startup (Talos, tspaint) and pageload (Raptor, tp6) performance tests) from a single ./mach try fuzzy command in a non-interactive shell (i.e. making use of queries).

Comment 1

[Andrew Halberstadt [:ahal]]

(In reply to Bianca Danforth [:bdanforth] from comment #0)

It would be very helpful if multiple queries could be passed into a single ./mach try fuzzy command where the provided path can be applied to only one or a subset of the queries rather than to all of them.

I don't think there is a good way to implement this on the command line.. How could we tell whether the path should filter down both queries, or just a single one? If just a single query, which one does it apply to? What if you pass in 10 queries? What if you pass in 10 paths? I think supporting this would just be too complex and confusing.

But I do think we can solve your use case by special casing performance related tasks. These aren't associated with a path, so it doesn't make sense to filter them out based on a path. So I'm thinking we can solve this by just select all "raptor" and "talos" tasks regardless of which path was chosen.

Comment 2

[Bianca Danforth [:bdanforth]]

I don't think there is a good way to implement this on the command line.. How could we tell whether the path should filter down both queries, or just a single one? If just a single query, which one does it apply to? What if you pass in 10 queries? What if you pass in 10 paths? I think supporting this would just be too complex and confusing.

Hm... yeah I was thinking perhaps the "easiest" way would just be that the path must always precede the query and only applies to the query immediately following it, so if I wanted different paths with multiple queries, it'd be something like this:

./mach try fuzzy path/to/first/set/of/tests -q "first query" path/to/second/set/of/tests -q "second query"

That would allow folks to run tests at multiple different paths in the same query (performance or otherwise). I could see this being useful say if I wanted to run my custom tests at ./testing/extensions/ and some existing unit tests elsewhere along with some performance tests.

Anyway, I don't claim to understand how complicated that would be to implement, nor do I have specific existing tests in mind at this time, so I don't want to scope creep on this bug if it's not feasible in your estimation. As you said, it does make sense for performance tests in particular.

Comment 3

[Andrew Halberstadt [:ahal]]

Unfortunately I don't think that would be possible without some deep argparse hackery. Argparse will return them both as lists, so both of these commands:

$ ./mach try path/foo path/bar -q "first query" -q "second query"
$ ./mach try path/foo -q "first query" path/bar -q "second query"

will be parsed into this:

paths = ["path/foo", "path/bar"]
queries = ["first query", "second query"]

By the time we get the result from argparse, there would be no way to tell what order those were specified in. I think if you truly need to run a different set of tasks on a different set of paths, at that point it's best to do two pushes :).