Open
Bug 1183144
Opened 9 years ago
Updated 2 years ago
When running tests on Try with --tag option, chunking behaviour is not predictable and makes no sense
Categories
(Testing :: Mochitest, defect)
Testing
Mochitest
Tracking
(Not tracked)
NEW
People
(Reporter: kats, Unassigned)
References
Details
See try push at [1]. I ran B2G emulator mochitests with --tag apz (so it is supposed to run all of the tests listed at [2]). However, if you look at the log files for the individual mochitests chunks (M1..M8) you will see the following: M1: ran test_a, test_b M2: ran test_c, test_d M3: ran test_e, test_f M4: ran test_g, test_h, test_i M5: ran test_j, test_k M6: ran test_l, test_m M7: ran test_n, test_o M8: ran test_bug1151663, test_p In particular test_basic_pan never got run, neither did test_bug982141. And in M8 test_bug1151663 ran *before* test_p despite being listed *after* test_p in the mochitest.ini file. This was just one of many attempts I pushed over the weekend to work around bug 1182817. My goal was to somehow run test_bug1151663 followed by test_basic_pan in the same chunk, with logging enabled. This would confirm/refute my theory that something in the former test was interfering with the latter test and causing it to fail. However after 5+ try pushes I was unable to get this working, because the behaviour of tagged tests running in chunks is just completely WTF. [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=f91134f3e97a [2] https://hg.mozilla.org/try/file/f91134f3e97a/gfx/layers/apz/test/mochitest.ini#l13
Reporter | ||
Comment 1•9 years ago
|
||
Also while looking at the commands needed to run this on a loaner, I noticed that the command has this: "--this-chunk 8 --total-chunks 9". Sounds like the job is getting split into 9 chunks but only 8 of them are actually getting run on Try.
Comment 2•9 years ago
|
||
Yeah, chunking no longer makes sense when you use --tags, everything should go into a single chunk, try using -u mochitest-1 instead. I think if you use |mach try| it will do something smart and convert your push to a single chunk automatically for you (though I could be wrong, double check with chmanchester). This might help understand what is happening a little better: https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1915 The test harness has a concept of filters. A filter basically just takes a list of tests, modifies or removes some of them, and returns a new list of tests. Filters get executed in the order they're applied. You'll notice that the "tags" filter is added before the "chunking" filter (by design). This means: 1. The harness will remove every test that doesn't have the apz tag (~20 tests remaining) 2. The harness will chunk whatever is left (20 / 8 -> ~2-3 tests per chunk) (In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #1) > Also while looking at the commands needed to run this on a loaner, I noticed > that the command has this: "--this-chunk 8 --total-chunks 9". Sounds like > the job is getting split into 9 chunks but only 8 of them are actually > getting run on Try. I'm not sure why this is happening, but it's probably why those last two tests didn't run.. I'll look into it.
Comment 3•9 years ago
|
||
Actually -u mochitest-1 won't work either, you need to not pass in any chunking stuff at all. :chmanchester designed --tags to be used with |mach try|. I guess when using it directly in a try syntax results are less than stellar... Bug 1177653 added --tags to trychooser, but if the UX is this bad maybe it should be removed again.. Chris, any thoughts on this?
Flags: needinfo?(cmanchester)
Comment 4•9 years ago
|
||
The behavior of putting all the tests in a single chunk is applied when filtering down to individual directories with the mach try command, not when using --tag. The UX here hasn't had any design applied to it -- putting tags in try syntax just forwards arguments to the harness. If we don't think this is useful enough to tell people about I guess that's disappointing, but I understand that creating no-op chunks that turn red and look like failures is a big opportunity for confusion, and we've uncovered a few more sharp edges here. Either way, comment 1 and bug 1182817 seem like the thing to address.
Flags: needinfo?(cmanchester)
Comment 5•9 years ago
|
||
I'll have a fix for bug 1182817 soon, but I don't think it'll have much impact on this workflow. Would it make sense to apply the "single chunk" behaviour to --tag as well as paths? Otherwise, people would have to edit the in-tree configs to get a single chunk.
Reporter | ||
Comment 6•9 years ago
|
||
I think fixing bug 1182817 and the issue in comment 1 would go a long way towards fixing the workflow. Also if there is a bug with respect to test ordering (see my note about test_p in comment 0) fixing that would be nice too.
Comment 7•9 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #5) > I'll have a fix for bug 1182817 soon, but I don't think it'll have much > impact on this workflow. Would it make sense to apply the "single chunk" > behaviour to --tag as well as paths? Otherwise, people would have to edit > the in-tree configs to get a single chunk. It's probably safe to do when we have control of the entire try syntax (I'll do it in bug 1183232). Having --tag always override chunking would lead to other counter-intuitive results. The mach command intervenes with chunking at mozharness time, so if the try-syntax otherwise results in chunks, each of those would end up running the same selection of tests but under different symbols on treeherder, which would be extremely confusing.
Comment 8•9 years ago
|
||
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #1) > Also while looking at the commands needed to run this on a loaner, I noticed > that the command has this: "--this-chunk 8 --total-chunks 9". Sounds like > the job is getting split into 9 chunks but only 8 of them are actually > getting run on Try. This is just because chunk 9 is hidden for some reason. One of the sheriffs might be able to tell you why, but for now just check "show excluded jobs" in the top right.
Reporter | ||
Comment 9•9 years ago
|
||
I asked RyanVM on IRC and he pointed me to bug 1168224 as the reason M-9 is hidden by default. I guess this is another point of incompatibility with the way --tag currently works on trychooser pushes with the way chunking is done. M-9 is hidden because it has a high rate of failures, but with --tag the tests that get run in it are completely different so it really shouldn't be hidden in that case.
Comment 10•9 years ago
|
||
Agreed, but the hiding mechanism is implemented in treeherder and as far as I know, you can't just "temporarily unhide a job for a given push". Hiding a job hides it from every push on that branch. Probably the easiest thing to do would be to modify the treeherder url that gets printed out to include ?exclusion_profile=false.
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•