Closed Bug 1397229 Opened 6 years ago Closed 6 years ago

move mochitest-browser-chrome tests from win8->win10

Categories

(Testing :: General, enhancement)

Product:
Testing
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox57 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla57
Tracking Flags:
Tracking Status
firefox57 --- fixed

People

(Reporter: jmaher, Assigned: ahal)

References

Details

Attachments

(4 files)

Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,
59 bytes, text/x-review-board-request
mconley
: review+
Details
Bug 1397229 - Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8,
59 bytes, text/x-review-board-request
jmaher
: review+
Details
Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,
59 bytes, text/x-review-board-request
jmaher
: review+
Details
Bug 1397229 - Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10,
59 bytes, text/x-review-board-request
jmaher
: review+
Details 
Sure! I'm really close to getting reftest selftests working, I'll take a look when I'm done that.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
So this test was skipped on Windows 8 within the test itself:
http://searchfox.org/mozilla-central/source/browser/base/content/test/performance/browser_windowopen_reflows.js

I guess because manifests can't skip on specific windows versions? (we should maybe fix that)

That being said, digging into that test it looks like on Windows 10 there is 1 fewer reflow than before (which is a good thing). So I deleted that expected reflow and everything looks like it works:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1c0739f6cc46c4218d4286fb6f9cd29dc307a02

I'll get review from mconley here to make sure this is ok. If it's not ok I guess we can just update the skip-if line to also include Win10.
Flags: needinfo?(ahalberstadt)
if needed in browser.ini we could:
skip-if = os == 'win' && bits=64
mozinfo adds the `os_version` key by default, and that's what we feed into the manifest parser, so you should be able to use `os_version == '10.0'` or similar.

On my local Windows 10 machine os_version == '10.0':
>>> import mozinfo
>>> mozinfo.os_version
StringVersion ('10.0')
True! I think the win8 tests are running on a win10 generated build though, so the mozinfo.json probably has the wrong os_version in it.

The fix worked though, so I think it's moot (for this case). The test can probably be re-enabled.
Just for completeness, I want to note that we don't put os_version in the mozinfo.json that the build generates. Random example from an inbound Windows build:
https://public-artifacts.taskcluster.net/bXT8PmuISYGgwBiQo7ge3w/0/public/build/target.mozinfo.json

os_version will be determined by mozinfo on the test machine, so it should work fine.
Ah, neat! I'll move the skip over to the manifest as part of this patch.
Depends on: 1397829
Comment on attachment 8906731 [details]
Bug 1397229 - Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8,

https://reviewboard.mozilla.org/r/178466/#review183466

this looks great; lets ensure we have capacity (i.e. bug 1398884)
Attachment #8906731 - Flags: review?(jmaher) → review+
Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

https://reviewboard.mozilla.org/r/178464/#review183470

Great! I assume this change survives retriggers on try? If so, yeah, let's take it, and please WONTFIX bug 1381521. Thanks for your work here!
Attachment #8906730 - Flags: review?(mconley) → review+
Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

https://reviewboard.mozilla.org/r/178464/#review183470

Yep, it's permanently fixed somehow!

Should I just remove the skip-if win 8 from the manifest then, given this won't be running anywhere on Windows 8? If for some weird reason we ever start running this there again, I guess we can just re-add the skip-if and file a new bug then.
We're still waiting for the two blockers here to be fixed before landing this.

There were also a few intermittents I noticed in some of my try pushes. They all look known, but I'll do a bunch more retriggers to make sure they didn't become really frequent.
There's two perma failures in that push:

1. browser/base/content/test/performance/browser_startup_images.js

This is a new test and is already skipped on Windows 7. It's unclear if it was running/working on Windows 8 or not, but maybe this Win10 issue is similar and it just needs to be skipped there too.


2. toolkit/components/tooltiptext/tests/browser_input_file_tooltips.js

This looks like a pre-existing intermittent turned perma-fail (bug 1295497). The test seems to be timing out here:
http://searchfox.org/mozilla-central/source/toolkit/components/tooltiptext/tests/browser_input_file_tooltips.js#38

It's worth noting this test also bypasses the no-arbitrary-setTimeout eslint rule, though looks unrelated to this specific issue.
Hey Johann, we're in the process of migrating some tests from Windows 8 -> Windows 10. It looks like the hdpi version of browser_startup_images.js is failing on Windows 10:
https://treeherder.mozilla.org/logviewer.html#?job_id=131099613&repo=try&lineNumber=4225

I noticed this test was already disabled on Windows 7, does this look like a similar issue? Can we also disable it on Windows 10 for the time being? Let me know if you see a fix and/or how you'd like to proceed! Thanks
Flags: needinfo?(jhofmann)
(In reply to Andrew Halberstadt [:ahal] from comment #15) 
> 2. toolkit/components/tooltiptext/tests/browser_input_file_tooltips.js
> 
> This looks like a pre-existing intermittent turned perma-fail (bug 1295497).
> The test seems to be timing out here:
> http://searchfox.org/mozilla-central/source/toolkit/components/tooltiptext/
> tests/browser_input_file_tooltips.js#38
> 
> It's worth noting this test also bypasses the no-arbitrary-setTimeout eslint
> rule, though looks unrelated to this specific issue.

Looking at bug 1295497, it looks like the timeout is happening at a different place, so this could be a different issue. Gijs, I saw you did a bit of digging in bug 1295497, do you have any idea what might be happening in:
https://treeherder.mozilla.org/logviewer.html#?job_id=131094283&repo=try&lineNumber=5406

For context, this started failing after migrating it from Win8 -> Win10.
Flags: needinfo?(gijskruitbosch+bugs)
(In reply to Andrew Halberstadt [:ahal] from comment #16)
> Hey Johann, we're in the process of migrating some tests from Windows 8 ->
> Windows 10. It looks like the hdpi version of browser_startup_images.js is
> failing on Windows 10:
> https://treeherder.mozilla.org/logviewer.
> html#?job_id=131099613&repo=try&lineNumber=4225
> 
> I noticed this test was already disabled on Windows 7, does this look like a
> similar issue? Can we also disable it on Windows 10 for the time being? Let
> me know if you see a fix and/or how you'd like to proceed! Thanks

I think that issue is slightly different than the one from Windows 7 (we probably just need to whitelist maximize.svg), but I don't have the time to deal with that immediately. I'm ok with disabling it for now if you open a bug for me to follow up on :)
Flags: needinfo?(jhofmann)
See Also: → 1400357
See Also: → 1400368
I replied on bug 1400368. We can disable the browser_input_file_tooltips.js test on win10 for now.
Flags: needinfo?(gijskruitbosch+bugs)
Here's a try run with those tests disabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b4d3794b17bf8b93c23eb0d50e8b54b7cf124946

There's a few intermittents, seem to all be on file. I'm sort of worried that the frequency has been increased, but I figured that's something we could make a decision on based on the orange factor data that comes in.
Comment on attachment 8909293 [details]
Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,

https://reviewboard.mozilla.org/r/180868/#review185992

::: browser/base/content/test/performance/hidpi/browser.ini:6
(Diff revision 1)
>  [DEFAULT]
>  prefs =
>    layout.css.devPixelsPerPx='2'
>  
>  [../browser_startup_images.js]
> -skip-if = !debug || (os == 'win' && os_version == '6.1') # hidpi results in the toolbar overflowing on Win 7
> +skip-if = !debug || (os == 'win' && (os_version == '6.1' || os_version == '10.0'))  # hidpi results in the toolbar overflowing on Win 7; fails on Win 10 (bug 1400357)

have you confirmed that our version os '10.0' ?
Attachment #8909293 - Flags: review?(jmaher) → review+
Comment on attachment 8909294 [details]
Bug 1397229 - Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10,

https://reviewboard.mozilla.org/r/180870/#review185994

assuming we match the os_version == 10.0 exactly.
Attachment #8909294 - Flags: review?(jmaher) → review+
Comment on attachment 8909293 [details]
Bug 1397229 - Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10,

https://reviewboard.mozilla.org/r/180868/#review185992

> have you confirmed that our version os '10.0' ?

Yep!
Backed out for failing browser-chrome's browser/base/content/test/performance/browser_windowopen_reflows.js on Windows 7:

https://hg.mozilla.org/integration/autoland/rev/93ac511811516cfcd3e3c759f0ce3afef8a6b371

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=cb09c5099e519f615038109f0da3da190555775b&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=131749901&repo=autoland

14:41:29     INFO -  Buffered messages finished
14:41:29    ERROR -  774 INFO TEST-UNEXPECTED-FAIL | browser/base/content/test/performance/browser_windowopen_reflows.js | unexpected uninterruptible reflow
14:41:29     INFO -  [
14:41:29     INFO -  	"handleEvent@chrome://browser/content/tabbrowser.xml:6864:19",
14:41:29     INFO -  	"EventListener.handleEvent*tabbrowser-tabs_XBL_Constructor@chrome://browser/content/tabbrowser.xml:6282:11",
14:41:29     INFO -  	""
14:41:29     INFO -  ]
14:41:29     INFO -   - false == true - JS frame :: chrome://mochitests/content/browser/browser/base/content/test/performance/head.js :: reflow :: line 115
14:41:29     INFO -  Stack trace:
14:41:29     INFO -  chrome://mochitests/content/browser/browser/base/content/test/performance/head.js:reflow:115
14:41:29     INFO -  chrome://browser/content/tabbrowser.xml:handleEvent:6864
Flags: needinfo?(ahalberstadt)
Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

This fixes the Windows 7 error from the backout:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b8d7a18ccc0bdfdd3bd50f1e9271140a69324320

Hey Mike, I'd appreciate a (trivial) re-review of this latest change. Using the user agent feels hacky, but this is how the test was previously being skipped and I don't know of a better way.

Alternatively if this test doesn't need to run on both Win7 and Win10, we could disable it on Win7.
Flags: needinfo?(ahalberstadt)
Attachment #8906730 - Flags: review+ → review?(mconley)
Comment on attachment 8906730 [details]
Bug 1397229 - Remove unused reflow from browser_windowopen_reflows.js on Windows,

https://reviewboard.mozilla.org/r/178464/#review186192

Yeah, let's do this. Thanks!
Attachment #8906730 - Flags: review?(mconley) → review+
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d928ad6475d2
Remove unused reflow from browser_windowopen_reflows.js on Windows, r=mconley
https://hg.mozilla.org/integration/autoland/rev/fed3a10c6ffc
Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/1c9f780c4e9c
Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10, r=jmaher
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9f07c6210b34
Remove unused reflow from browser_windowopen_reflows.js on Windows, r=mconley
https://hg.mozilla.org/integration/autoland/rev/2671b6644f50
Disable browser/base/content/test/performance/browser_startup_images.js hidpi on Windows 10, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/bb89cd6f5507
Disable toolkit/browser/components/tooltiptext/browser_input_file_tooltips.js on Win10, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/008e877ffdaa
Schedule mochitest-browser-chrome tests on Windows 10 instead of Windows 8, r=jmaher
Backout by kwierso@gmail.com:
https://hg.mozilla.org/mozilla-central/rev/87b0dfdc4832
Backed out changeset cb09c5099e51 
https://hg.mozilla.org/mozilla-central/rev/1eaf47154762
Backed out changeset 1c9f780c4e9c 
https://hg.mozilla.org/mozilla-central/rev/9f2bddf428df
Backed out changeset fed3a10c6ffc
You need to log in before you can comment on or make changes to this bug.