1308789 - Win8 tests don't run on try pushes with "-p all -u all[Windows 8]"

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

I can't even.

With an explicit list of platforms like in https://treeherder.mozilla.org/#/jobs?repo=try&revision=243c762f538197753011b79e68914fe56b586508 we run the Win8 tests for -u all[Windows 8].

With -p all like in https://treeherder.mozilla.org/#/jobs?repo=try&revision=1e0d5d54574f0cc7fa867faf41ec192d7be1156d we don't.

That's not true of the other try-nondefault platforms, -p all -u all[Windows XP,10.6,Windows 8] will run WinXP and 10.6 tests, just not Win8. -p all -u all[-quux] gives the same result, 10.6 and WinXP but no Win8.

Comment 1

[Chris AtLee [:catlee]]

A few notes:
* We set win64-debug try_by_default to False here: http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l1777
* Try push here confirms this issue: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6120a9d6cde584b96f3a16eb051d14e3adcd57ab

Interestingly, the try push got neither opt nor debug tests for Win8.

I think we should at least update win64-debug to remove that try_by_default line and let it use the default (True). I'm not sure yet if that will actually help, but it shouldn't hurt either.

Comment 2

[Chris AtLee [:catlee]]

Interestingly the unittests fail if win64-debug is added as try-nondefault (win64 opt is try by default). If both win64 and win64-debug are either try-nondefault or try by default then the tests pass. I wonder if this points to where the problem is with try parser.

Comment 3

[Phil Ringnalda (:philor)]

Ah, bug 914914 comment 2.

So, again, like bug 914914 comment 3 we don't have to fix it, because both platforms should have no try_by_default in mozilla/config.py, because what we want is only the try_by_default: False in https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla-tests/config.py#116 so we get by-default Win64 opt and debug builds, but not-by-default Win8 tests.

Comment 4

[Phil Ringnalda (:philor)]

Attachment: WTWTCH?

Really, surprisingly few of my "Completely untested, of course" patches have been backed out.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8800520 [details] [diff] [review]
WTWTCH?

Review of attachment 8800520 [details] [diff] [review]:
-----------------------------------------------------------------

what could possibly go wrong?

Comment 6

[Phil Ringnalda (:philor)]

Comment on attachment 8800520 [details] [diff] [review]
WTWTCH?

https://hg.mozilla.org/build/buildbot-configs/rev/a9bbcd872ad9aa70ca952d3571142aad6a5dc26e

Comment 7

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/a9bbcd872ad9

Comment 8

[Phil Ringnalda (:philor)]

And https://treeherder.mozilla.org/#/jobs?repo=try&revision=ba1c42dc3ddf - no joy, no win8 tests.

Comment 9

[Phil Ringnalda (:philor)]

I got nothin' more than that.

Comment 10

[Chris AtLee [:catlee]]

Attachment: Bug 1308789 - Add test case to reproduce failure

Review commit: https://reviewboard.mozilla.org/r/86244/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/86244/

Comment 11

[Chris AtLee [:catlee]]

Attachment: Bug 1308789 - Fix setting of default platforms for tests

The previous logic meant that any top-level platform (e.g. 'win64') without any
try-by-default slave platforms wouldn't get test runs unless 'win64' was
specified as an argument to -p.

This patch also flips the test case logic around now to demonstrate that the
issue is fixed.

Review commit: https://reviewboard.mozilla.org/r/86246/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/86246/

Comment 12

[Chris AtLee [:catlee]]

I think what was happening was that try_parser was filtering out any platforms which had *no* default-on-try test platform. Neither of win64's test platforms are default on try, which is why win64 was then excluded from the set of default platforms...or something.

I think these patches fix the issue.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8801533 [details]
Bug 1308789 - Add test case to reproduce failure

https://reviewboard.mozilla.org/r/86244/#review85298

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8801534 [details]
Bug 1308789 - Fix setting of default platforms for tests

https://reviewboard.mozilla.org/r/86246/#review85308

Not fully groking this but benefit of the doubt. Do we have test(s) that we're not turning on more than we think here ?

Comment 15

[Chris AtLee [:catlee]]

I _think_ the existing tests cover that we're not over-enabling. If we do discover this has unintended consequences in the wild, I'll fix it up and add more tests.

Comment 16

[Chris AtLee [:catlee]]

https://hg.mozilla.org/build/buildbotcustom/rev/5a283b5466d69441dc25b1e1f6fdc02dd0405465
Bug 1308789 - Add test case to reproduce failure r=nthomas

https://hg.mozilla.org/build/buildbotcustom/rev/bfe42c243d4a7c6f1a15fb2d36567d715a5447e4
Bug 1308789 - Fix setting of default platforms for tests r=nthomas

Comment 17

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/5a283b5466d6

Comment 18

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/bfe42c243d4a

Comment 19

[Chris AtLee [:catlee]]

Looks like this is working:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fde1934023456b0b63069dc22f3a84089b36f9cd

Comment 20

[Phil Ringnalda (:philor)]

And for talos too.