1672100 - Same jobs running on the same changeset should trigger the same tests on try

Status: RESOLVED WONTFIX

(Developer Infrastructure :: Try, defect)

Description

[Mike Hommey [:glandium]]

In this push:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=OiDvRSeKSzyyOCrif4pN5A.0&searchStr=mochitest%2Ctsan&revision=2d74ee5706f470d66c1969a74008b1024566796a

which is essentially the same thing as https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=mochitest&revision=609ac687f7e38023eae03bad472b1632d7d04768

I originally triggered test-linux1804-64-tsan/opt-mochitest-plain-e10s-1 because that's what the backfill was running. Turns out, that didn't run the same set of tests, so I looked further, and saw that test-linux1804-64-tsan/opt-mochitest-plain-e10s-4 on autoland had the same error as the first error in the backfill, so I triggered that one on try... which didn't fail. And looking at the log, the test that failed on autoland (test_bug1303704.html) didn't even run on try. Which led me, right now, to trigger all the mochitests.

Comment 1

[Mike Hommey [:glandium]]

test_bug1303704.html ended up in test-linux1804-64-tsan/opt-mochitest-plain-e10s-8....

Comment 2

[Andrew Halberstadt [:ahal]]

Yes, with manifest scheduling chunk numbers are essentially meaningless. This is an unavoidable consequence of manifest scheduling, but fortunately there's an easy work around:

$ ./mach try {fuzzy|chooser} path/to/dir/containing/tests

On treeherder you can use the "Test Path Filter":
https://wiki.mozilla.org/Sheriffing/Manifest_Scheduling#Push_Continuity

note: due to layoffs it is incomplete and only works if you specify a path to the manifest.ini, these are linked from the Test Groups tab of a test task.

We're kind of in a weird spot right now where chunks are still front and centre, but when time permits they will be de-emphasized. In bug 1557848 I'd like to put them behind a flag for the various mach try interfaces, and there have been talks about removing them from the default treeherder view as well.

Comment 3

[Mike Hommey [:glandium]]

So how is one supposed to reproduce the exact set of tests that run on autoland in a given task? Are we to assume that everything is fine as far as independence between tests is concerned? What if one wants to reproduce a set of test errors from autoland? Say, on something-2 and something-7. Do we now need to list all the paths of tests that failed?

Comment 4

[Andrew Halberstadt [:ahal]]

(In reply to Mike Hommey [:glandium] from comment #3)

So how is one supposed to reproduce the exact set of tests that run on autoland in a given task?

Is there a use case you have where this would be needed?

Are we to assume that everything is fine as far as independence between tests is concerned?

Yes. Manifests will never be split between chunks, and each manifest runs in a separate browser instance with a fresh profile. It's still technically possible for bad test interactions, but it's very rare.

What if one wants to reproduce a set of test errors from autoland? Say, on something-2 and something-7. Do we now need to list all the paths of tests that failed?

Yes you'd need to specify each path. Though you could also tackle one issue at a time, should be the same thing.

Comment 5

[Mike Hommey [:glandium]]

(In reply to Andrew Halberstadt [:ahal] from comment #4)

(In reply to Mike Hommey [:glandium] from comment #3)

So how is one supposed to reproduce the exact set of tests that run on autoland in a given task?

Is there a use case you have where this would be needed?

Reproducing failures from autoland. I assume a lot of people would expecting to be able to do that rather than go and pick individual tests manually. If it were automated, that would be a different story.