Closed Bug 1352000 Opened 7 years ago Closed 6 years ago

--try-test-paths sometimes not respected

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: florian, Unassigned)

References

Details

On this try push, the "Windows 7 VM opt Mochitests executed by TaskCluster test-windows7-32-vm/opt-mochitest-browser-chrome-1 tc-M(bc1)" job ran tests that are not from the folder I specified using --try-test-paths (and didn't run the tests I was actually interested in) :

https://treeherder.mozilla.org/#/jobs?repo=try&revision=076424360135d36d5da9ce8f33f1bed9e6b3b01b
Component: Tools → General
The problem has now spread to the OS X 10.10 tc-M* jobs, making it harder to run specific tests lots of times without wasting lots of resources on unrelated tests.
Gregory, is this something you can help with? Or do you know who can help with this?
Flags: needinfo?(gps)
This smells like something that was supported in a buildbot world but not in TaskCluster. I don't see any references to --try-test-paths in the decision graph code. So my guess is this option is being dropped on the floor by the decision graph.
Component: General → Build Config
Flags: needinfo?(gps)
Product: Release Engineering → Core
I've never heard of it -- is it something that gets passed along to mozharness? I think Wander just landed something to help with that.
This bug makes debugging individual tests on try really painful; is there any chance of fixing it soon?

Eg. on https://treeherder.mozilla.org/#/jobs?repo=try&revision=f6175c789256e5e043cc723c81187eb55ce4e395 I'm getting the right jobs on Linux 64, but the Windows 7 VM test jobs are not what I requested. In both cases they are tc-M/tc-M-e10s) jobs.
gps, dustin, this feature was very useful, can we get this bug prioritized so that it doesn't wait several more months before getting fixed?
Blocks: 1193215
Flags: needinfo?(gps)
Flags: needinfo?(dustin)
If you can dig into how this option used to work, we can probably figure out how to implement it.  But absent any info or documentation on what it does/did it's hard to "fix" this.
Flags: needinfo?(dustin)
(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> If you can dig into how this option used to work, we can probably figure out
> how to implement it.  But absent any info or documentation on what it
> does/did it's hard to "fix" this.

The way I used it was with a try syntax like this:
try: -b do -p linux,linux64,macosx64,win32,win64 -u mochitest-bc,mochitest-e10s-bc -t none --try-test-paths browser-chrome:browser/base/content/test/performance

This still triggered all the bcN chunks, but each of them was running only the tests in the browser/base/content/test/performance directory. With sometimes an additional --rebuild 5, it was a very convenient way to run plenty of times the test I was debugging. Either for new tests I was about to land to ensure there was no intermittent failures, or to debug known oranges.
(In reply to Florian Quèze [:florian] [:flo] from comment #8)
> it was a very convenient way to run
> plenty of times the test I was debugging.

It also gave very quick results, as running only the folder I care about usually takes less than 5 minutes. Also nice to save resources I guess.
It looks like testing/mozharness/mozharness/mozilla/testing/try_tools.py looks for this in the commit message.

_extract_try_message looks in os.environ['TRY_COMMIT_MSG']

Looking at a random test job,
    https://tools.taskcluster.net/groups/Ty6WCNvZQmGycBtthXHXig/tasks/RWGc6ejKSjWRUwpoYtjXVA/details
I see
    "TRY_COMMIT_MSG": "try: -b do -p linux64 -u all[linux64-qr] -t none",

So I would think this should all "just work".

Can you make a try push that uses this feature, and we can dig into where that info is getting lost?
(In reply to Dustin J. Mitchell [:dustin] from comment #10)

> Can you make a try push that uses this feature, and we can dig into where
> that info is getting lost?

I just did (https://treeherder.mozilla.org/#/jobs?repo=try&revision=18a2b60521c3a750027e0b4a590e53eaba1cf672) it looks like things work as expected today. Sorry for the noise, I don't know when this got fixed, but it's awesome to be able to use this again (this try push gave results within 20 minutes!)
Status: NEW → RESOLVED
Closed: 6 years ago
Flags: needinfo?(gps)
Resolution: --- → WORKSFORME
Haha, I don't think anything changed that would have affected it.  There's an outside chance that this failed on particularly old revisions (like many weeks old now).  Anyway, glad it works!
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.