After bug 1250904 is fixed, we should make sure developers running interactive workers have access to a mach environment that is set up to run the relevant test harness. This is in addition to (not instead of) the ability to resume the job as is. This might be useful in the case that developers just need to replicate the environment, and don't necessarily care about replicating the exact same command line/test paths. There are two approaches we can do: 1) Make use of the mach in the tests.zip. This is by the easiest approach, though some work will be needed to get it working again. Also there is only currently mochitest support. But it should be do-able, such that the commands there will "just work" in the worker environment. 2) Wait for running tests from source. This is the cleaner approach, but has a very large dependency: stop requiring the objdir to run tests. This is something we want to work on anyway, but is likely a long way out. So option 1 is likely better in the short term. To clarify what this intends to provide, upon getting a loaner and opening the taskcluster browser shell, developers might see a MOTD similar to: Either run 'resume-task' to continue running the current job, or run 'mach mochitest -f plain' to experiment with different test configurations.
Adding gps and chmanchester, who might comment on the viability of running tests from source in automation.
I think we can take baby steps towards the "run everything from the source directory" goal, like: 1) Just have a source checkout matching the built changeset on the test machine. This is immediately useful for debugging, since you have the matching source handy. 2) Invoke the test harnesses from the source tree instead of from the test package. Use the same commandlines as we currently use, but point to `$srcdir/testing/mochitest/runtests.py` instead of `$testpackage/mochitest/runtests.py`. Pretty minor change, not a huge benefit. 3) Change test invocation on the testers to be e.g. `./mach mochitest <args>` from the srcdir, but using the downloaded build and test package, maybe via another argument to the mach command, maybe via some mach configuration command beforehand. This has the benefit of making the test invocation way more familiar to developers. It also simplifies the "debug this test run" flow by using the existing `--debugger` flag that our mach commands support. 4) Finally, change test harnesses to use test files from the srcdir instead of from the test package. This is harder for some harnesses (Mochitest and xpcshell, notably) because they use a mix of srcdir+objdir files, but it has the biggest payoff.
Something we'll have to watch out for is a dependency on moz.build being read (read: running configure). Right now, the mach testing commands need state from a configured build system. I'm not sure we'll be able to avoid scanning moz.build files completely (we need to know what tests are active and where the files are). We could probably hack something together where we continue to share state from the build to the test job. But long term we should avoid that.
I remember Chris mentioned landing some changes that would make a file system only TestResolver possible. I'm not super familiar with how the current test resolving works, but if we had that maybe we could invoke it without reading moz.build? I'm sure there are downsides to that I'm unaware of :). It's also not strictly true that we need moz.build for this. Any manifestparser based harness can support passing in directories via a crude pathprefix filter: https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/manifestparser/manifestparser/filters.py#342 And reftest has similar path filtering capabilities. I've always been a bit confused between the relation of moz.build for tests and test manifests. They seem to be trying to solve the same thing.
Summary: Make test harness mach commands available on the $PATH for interactive jobs → [Tracking] Make test harness mach commands available on the $PATH for interactive jobs
I would like to drop this out of the scope for this quarter: * Bug 1234728 - [meta] Run tests from source directory
Oh yes, for sure. I don't think anyone has even come close to committing to working on that. It probably doesn't need to be a dependency of this bug.. kind of a "it would have made this bug trivial if we had it" type of thing.
Thanks Andrew! Removing it for now. jgriffin, we believe we're in unison. If not we can chat in our meeting tomorrow.
No longer depends on: 1234728
Assignee: nobody → ahalberstadt
Priority: -- → P1
No more dependents, calling this fixed.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.