1397229 - move mochitest-browser-chrome tests from win8->win10

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we have 1 test failure to fix, then we can move:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d91ebd8cd18f92aba7b87a23650c3422b695ff36&filter-tier=1&filter-tier=2&filter-tier=3&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&filter-resultStatus=success&filter-resultStatus=coalesced&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&group_state=expanded&exclusion_profile=false&filter-searchStr=browser-chrome

:ahal, could you get a fix for the test failure and give a rough estimate of how many machine hours we need per push?  we could then multiply that by 10 to give us a hint as to how many machines we would need initially.

Comment 1

[Andrew Halberstadt [:ahal]]

Sure! I'm really close to getting reftest selftests working, I'll take a look when I'm done that.

Comment 2

[Andrew Halberstadt [:ahal]]

So this test was skipped on Windows 8 within the test itself:
http://searchfox.org/mozilla-central/source/browser/base/content/test/performance/browser_windowopen_reflows.js

I guess because manifests can't skip on specific windows versions? (we should maybe fix that)

That being said, digging into that test it looks like on Windows 10 there is 1 fewer reflow than before (which is a good thing). So I deleted that expected reflow and everything looks like it works:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1c0739f6cc46c4218d4286fb6f9cd29dc307a02

I'll get review from mconley here to make sure this is ok. If it's not ok I guess we can just update the skip-if line to also include Win10.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

if needed in browser.ini we could:
skip-if = os == 'win' && bits=64

Comment 4

[(not currently active) Ted Mielczarek]

mozinfo adds the `os_version` key by default, and that's what we feed into the manifest parser, so you should be able to use `os_version == '10.0'` or similar.

On my local Windows 10 machine os_version == '10.0':
>>> import mozinfo
>>> mozinfo.os_version
StringVersion ('10.0')

Comment 5

[Andrew Halberstadt [:ahal]]

True! I think the win8 tests are running on a win10 generated build though, so the mozinfo.json probably has the wrong os_version in it.

The fix worked though, so I think it's moot (for this case). The test can probably be re-enabled.

Comment 6

[(not currently active) Ted Mielczarek]

Just for completeness, I want to note that we don't put os_version in the mozinfo.json that the build generates. Random example from an inbound Windows build:
https://public-artifacts.taskcluster.net/bXT8PmuISYGgwBiQo7ge3w/0/public/build/target.mozinfo.json

os_version will be determined by mozinfo on the test machine, so it should work fine.

Comment 7

[Andrew Halberstadt [:ahal]]

Ah, neat! I'll move the skip over to the manifest as part of this patch.

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

This test was previously skipped on Windows 8. This bug is migrating the test over to Windows 10
where we discovered the unused reflow.

Review commit: https://reviewboard.mozilla.org/r/178464/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/178464/

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1397229 - Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8,

Review commit: https://reviewboard.mozilla.org/r/178466/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/178466/

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8906731 [details]
Bug 1397229 - Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8,

https://reviewboard.mozilla.org/r/178466/#review183466

this looks great; lets ensure we have capacity (i.e. bug 1398884)

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

https://reviewboard.mozilla.org/r/178464/#review183470

Great! I assume this change survives retriggers on try? If so, yeah, let's take it, and please WONTFIX bug 1381521. Thanks for your work here!

Comment 12

[Andrew Halberstadt [:ahal]]

Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

https://reviewboard.mozilla.org/r/178464/#review183470

Yep, it's permanently fixed somehow!

Should I just remove the skip-if win 8 from the manifest then, given this won't be running anywhere on Windows 8? If for some weird reason we ever start running this there again, I guess we can just re-add the skip-if and file a new bug then.

Comment 13

[Andrew Halberstadt [:ahal]]

We're still waiting for the two blockers here to be fixed before landing this.

There were also a few intermittents I noticed in some of my try pushes. They all look known, but I'll do a bunch more retriggers to make sure they didn't become really frequent.

Comment 14

[Andrew Halberstadt [:ahal]]

Dang, looks like there's some new problems:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b3435aba0898e2294debf84d2a078bb9c672c180

Comment 15

[Andrew Halberstadt [:ahal]]

There's two perma failures in that push:

1. browser/base/content/test/performance/browser_startup_images.js

This is a new test and is already skipped on Windows 7. It's unclear if it was running/working on Windows 8 or not, but maybe this Win10 issue is similar and it just needs to be skipped there too.


2. toolkit/components/tooltiptext/tests/browser_input_file_tooltips.js

This looks like a pre-existing intermittent turned perma-fail (bug 1295497). The test seems to be timing out here:
http://searchfox.org/mozilla-central/source/toolkit/components/tooltiptext/tests/browser_input_file_tooltips.js#38

It's worth noting this test also bypasses the no-arbitrary-setTimeout eslint rule, though looks unrelated to this specific issue.

Comment 16

[Andrew Halberstadt [:ahal]]

Hey Johann, we're in the process of migrating some tests from Windows 8 -> Windows 10. It looks like the hdpi version of browser_startup_images.js is failing on Windows 10:
https://treeherder.mozilla.org/logviewer.html#?job_id=131099613&repo=try&lineNumber=4225

I noticed this test was already disabled on Windows 7, does this look like a similar issue? Can we also disable it on Windows 10 for the time being? Let me know if you see a fix and/or how you'd like to proceed! Thanks

Comment 17

[Andrew Halberstadt [:ahal]]

(In reply to Andrew Halberstadt [:ahal] from comment #15) 
> 2. toolkit/components/tooltiptext/tests/browser_input_file_tooltips.js
> 
> This looks like a pre-existing intermittent turned perma-fail (bug 1295497).
> The test seems to be timing out here:
> http://searchfox.org/mozilla-central/source/toolkit/components/tooltiptext/
> tests/browser_input_file_tooltips.js#38
> 
> It's worth noting this test also bypasses the no-arbitrary-setTimeout eslint
> rule, though looks unrelated to this specific issue.

Looking at bug 1295497, it looks like the timeout is happening at a different place, so this could be a different issue. Gijs, I saw you did a bit of digging in bug 1295497, do you have any idea what might be happening in:
https://treeherder.mozilla.org/logviewer.html#?job_id=131094283&repo=try&lineNumber=5406

For context, this started failing after migrating it from Win8 -> Win10.

Comment 18

[Johann Hofmann [:johannh]]

(In reply to Andrew Halberstadt [:ahal] from comment #16)
> Hey Johann, we're in the process of migrating some tests from Windows 8 ->
> Windows 10. It looks like the hdpi version of browser_startup_images.js is
> failing on Windows 10:
> https://treeherder.mozilla.org/logviewer.
> html#?job_id=131099613&repo=try&lineNumber=4225
> 
> I noticed this test was already disabled on Windows 7, does this look like a
> similar issue? Can we also disable it on Windows 10 for the time being? Let
> me know if you see a fix and/or how you'd like to proceed! Thanks

I think that issue is slightly different than the one from Windows 7 (we probably just need to whitelist maximize.svg), but I don't have the time to deal with that immediately. I'm ok with disabling it for now if you open a bug for me to follow up on :)

Comment 19

[:Gijs (he/him)]

I replied on bug 1400368. We can disable the browser_input_file_tooltips.js test on win10 for now.

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,

This started failing when migrating from Windows 8 -> Windows 10. Fix is
tracked in bug 1400357.

Review commit: https://reviewboard.mozilla.org/r/180868/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180868/

Comment 21

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1397229 - Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10,

This started failing when migrating from Windows 8 -> Windows 10. Fix is
tracked in bug 1400368.

Review commit: https://reviewboard.mozilla.org/r/180870/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180870/

Comment 22

[Andrew Halberstadt [:ahal]]

Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/178464/diff/1-2/

Comment 23

[Andrew Halberstadt [:ahal]]

Comment on attachment 8906731 [details]
Bug 1397229 - Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/178466/diff/1-2/

Comment 24

[Andrew Halberstadt [:ahal]]

Here's a try run with those tests disabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b4d3794b17bf8b93c23eb0d50e8b54b7cf124946

There's a few intermittents, seem to all be on file. I'm sort of worried that the frequency has been increased, but I figured that's something we could make a decision on based on the orange factor data that comes in.

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8909293 [details]
Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,

https://reviewboard.mozilla.org/r/180868/#review185992

::: browser/base/content/test/performance/hidpi/browser.ini:6
(Diff revision 1)
>  [DEFAULT]
>  prefs =
>    layout.css.devPixelsPerPx='2'
>  
>  [../browser_startup_images.js]
> -skip-if = !debug || (os == 'win' && os_version == '6.1') # hidpi results in the toolbar overflowing on Win 7
> +skip-if = !debug || (os == 'win' && (os_version == '6.1' || os_version == '10.0'))  # hidpi results in the toolbar overflowing on Win 7; fails on Win 10 (bug 1400357)

have you confirmed that our version os '10.0' ?

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8909294 [details]
Bug 1397229 - Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10,

https://reviewboard.mozilla.org/r/180870/#review185994

assuming we match the os_version == 10.0 exactly.

Comment 27

[Andrew Halberstadt [:ahal]]

Comment on attachment 8909293 [details]
Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,

https://reviewboard.mozilla.org/r/180868/#review185992

> have you confirmed that our version os '10.0' ?

Yep!

Comment 28

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for failing browser-chrome's browser/base/content/test/performance/browser_windowopen_reflows.js on Windows 7:

https://hg.mozilla.org/integration/autoland/rev/93ac511811516cfcd3e3c759f0ce3afef8a6b371

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=cb09c5099e519f615038109f0da3da190555775b&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=131749901&repo=autoland

14:41:29     INFO -  Buffered messages finished
14:41:29    ERROR -  774 INFO TEST-UNEXPECTED-FAIL | browser/base/content/test/performance/browser_windowopen_reflows.js | unexpected uninterruptible reflow
14:41:29     INFO -  [
14:41:29     INFO -  	"handleEvent@chrome://browser/content/tabbrowser.xml:6864:19",
14:41:29     INFO -  	"EventListener.handleEvent*tabbrowser-tabs_XBL_Constructor@chrome://browser/content/tabbrowser.xml:6282:11",
14:41:29     INFO -  	""
14:41:29     INFO -  ]
14:41:29     INFO -   - false == true - JS frame :: chrome://mochitests/content/browser/browser/base/content/test/performance/head.js :: reflow :: line 115
14:41:29     INFO -  Stack trace:
14:41:29     INFO -  chrome://mochitests/content/browser/browser/base/content/test/performance/head.js:reflow:115
14:41:29     INFO -  chrome://browser/content/tabbrowser.xml:handleEvent:6864

Comment 29

[Andrew Halberstadt [:ahal]]

Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/178464/diff/2-3/

Comment 30

[Andrew Halberstadt [:ahal]]

Comment on attachment 8909293 [details]
Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/180868/diff/1-2/

Comment 31

[Andrew Halberstadt [:ahal]]

Comment on attachment 8909294 [details]
Bug 1397229 - Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/180870/diff/1-2/

Comment 32

[Andrew Halberstadt [:ahal]]

Comment on attachment 8906731 [details]
Bug 1397229 - Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/178466/diff/2-3/

Comment 33

[Andrew Halberstadt [:ahal]]

Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

This fixes the Windows 7 error from the backout:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b8d7a18ccc0bdfdd3bd50f1e9271140a69324320

Hey Mike, I'd appreciate a (trivial) re-review of this latest change. Using the user agent feels hacky, but this is how the test was previously being skipped and I don't know of a better way.

Alternatively if this test doesn't need to run on both Win7 and Win10, we could disable it on Win7.

Comment 34

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

https://reviewboard.mozilla.org/r/178464/#review186192

Yeah, let's do this. Thanks!

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d928ad6475d2
Remove unused reflow from browser_windowopen_reflows.js on Windows, r=mconley
https://hg.mozilla.org/integration/autoland/rev/fed3a10c6ffc
Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/1c9f780c4e9c
Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10, r=jmaher

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9f07c6210b34
Remove unused reflow from browser_windowopen_reflows.js on Windows, r=mconley
https://hg.mozilla.org/integration/autoland/rev/2671b6644f50
Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/bb89cd6f5507
Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/008e877ffdaa
Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8, r=jmaher

Comment 37

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/87b0dfdc4832
Backed out changeset cb09c5099e51 
https://hg.mozilla.org/mozilla-central/rev/1eaf47154762
Backed out changeset 1c9f780c4e9c 
https://hg.mozilla.org/mozilla-central/rev/9f2bddf428df
Backed out changeset fed3a10c6ffc

Comment 38

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/9f07c6210b34
https://hg.mozilla.org/mozilla-central/rev/2671b6644f50
https://hg.mozilla.org/mozilla-central/rev/bb89cd6f5507
https://hg.mozilla.org/mozilla-central/rev/008e877ffdaa