1433438 - Update win10 version for mochitest-chrome [and other tests: wpt, reftests, etc] to the Fall Creators Update release or later

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Jonathan Kew [:jfkthame]]

Bug 1323743 will land a chrome mochitest that requires OpenType Variation Font support, which only becomes available on Windows with the Win10 Fall Creators Update.

The test passes for me locally, on a fully-updated Windows machine, but fails on tryserver; I presume this is because we're running an older Win10 release (but don't know how to identify the exact release from within the test, nor do I see it obviously reported in the log file).

For now, I intend to land the test with an annotation so it is skipped on Windows, but in order to be able to have test coverage for new features like this -- other examples will no doubt appear in due course -- we should update the test environment to the latest public release of the OS.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

We're currently running 1703 on our Win10 machines. Last I heard, the tentative plan for the next update was to wait for 1803 before updating again.

Comment 2

[Jonathan Kew [:jfkthame]]

Build 1703 is Windows 10 Creators Update, so that confirms why I'm seeing the failure in bug 1323743; we'd need to be on 1709 for that to work.

If we can't readily do the update for at the moment (I realize it might involve significant work/disruption), is it possible to identify the specific Windows build either within the test itself (navigator.oscpu, for example, just says "Windows NT 10.0; Win64; x64", which doesn't distinguish the various Win10 updates) or in a chrome.ini manifest conditional?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

we support os_version and version, unfortunately version only has 10.0.1703 when we have major version >6 and minor version>2 so it will not be for win10:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozinfo/mozinfo/mozinfo.py#90

possibly we could change that- not sure why it is coded that way- lets see if there is a reason for it or objections to changing that.

Comment 4

[James Graham [:jgraham]]

I know nothing. More precisely 100% of my knowledge on this subject comes from the comments in https://bugzilla.mozilla.org/show_bug.cgi?id=1193033 which is what I used to write that patch.

From the wpt of view, we use something like (os == "win") and (version == "10.0.15063") in metadata to identify the Windows version. So if you change the meaning of version, we would need to regenerate existing metadata. Probably not an issue. If you add a new field that distinguishes two configurations then we may need to account for that in the metadata.

Comment 5

[(not currently active) Ted Mielczarek]

That code is written that way because Microsoft froze the version numbers that some APIs returned because software did dumb things with higher versions. On my Windows 10 machine (updated to FCU), I get:
>>> import os
>>> os.sys.getwindowsversion()
sys.getwindowsversion(major=6, minor=2, build=9200, platform=2, service_pack='')

Oddly, for some reason mozinfo only sets `os_version` to have major.minor for Windows:

>>> import mozinfo
>>> mozinfo.os_version
StringVersion ('10.0')

...but it sets `version` to major.minor.build, which looks correct:
>>> mozinfo.version
'10.0.16299'

I don't know what (if anything) the "Build 1703" maps to, TBH, but per Wikipedia FCU is build 16299:
https://en.wikipedia.org/wiki/Windows_10_version_history#Version_1709_(Fall_Creators_Update)

Comment 6

[Ryan VanderMeulen [:RyanVM]]

1703 is the Creators Update from last April, version 10.0.15063.

Comment 7

[Andrew Halberstadt [:ahal] (PTO Until Apr 3rd)]

It looks like there's a bug in that code:

    if major >= 6 and minor >= 2:

Should be:

    if (major == 6 and minor >= 2) || major > 6:

Comment 8

[(not currently active) Ted Mielczarek]

(In reply to Andrew Halberstadt [:ahal] from comment #7)
> It looks like there's a bug in that code:

It's weird, but no. This API is hardcoded to return 6.2 as the OS version. (I think it can maybe return 6.3 if you updated to Windows 10 from Windows 8 or something like that.)

Comment 9

[Andrew Halberstadt [:ahal] (PTO Until Apr 3rd)]

Oh I see what you meant now. So looks like we do have build numbers for Windows 10, but we probably can't use it with < or > in a manifest. We could either put it into its own mozinfo property, or better yet teach manifestparser to compare version numbers.

Comment 10

[Jonathan Kew [:jfkthame]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #1)
> We're currently running 1703 on our Win10 machines. Last I heard, the
> tentative plan for the next update was to wait for 1803 before updating
> again.

Release 1803 (build number 10.0.17134) is out. Can we try to move this forward fairly soon? In particular, Variable Fonts is planned to ship in Fx62, but currently has no test coverage at all in automation - which is somewhat concerning - because we're not running a recent enough Windows build to support it.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

:fubar, can you share your thoughts on plans to update windows (or other OS).  I know it is a lot of work and we are still finishing up data center migrations.

:jfkthame, is it possible if this is not realistic to get in automation that we solve it manually?  Maybe submit a PI request and get some manual coverage.

Comment 12

[Jonathan Kew [:jfkthame]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #11)
> :fubar, can you share your thoughts on plans to update windows (or other
> OS).  I know it is a lot of work and we are still finishing up data center
> migrations.
> 
> :jfkthame, is it possible if this is not realistic to get in automation that
> we solve it manually?  Maybe submit a PI request and get some manual
> coverage.

We've got a PI request open, and manual testing is ongoing right now. But without CI coverage once the initial PI testing is complete, I'm concerned that we'll always be liable to inadvertently break the functionality at random times going forward (e.g. as we move to webrender, it'll be at high risk of disruption), and we may not find out about breakage until it happens to be manually noticed and bugs filed.

Having CI testing will let developers know immediately a breaking change lands (or rather, prior to even trying to land a breaking change, thanks to tryserver), instead of finding out days or weeks later.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

FYI, version 1803 is already on 12% of our Win10-running release users and 25% on Nightly. For a bit over a month post-release, that's a pretty fast rollout on Microsoft's part.

Comment 14

[Kendall Libby [:fubar] (he/him)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #11)
> :fubar, can you share your thoughts on plans to update windows (or other
> OS).  I know it is a lot of work and we are still finishing up data center
> migrations.

We're just launching a project to completely revamp how we create win10 images and manage the OS; it's been in a not great state from the start and it's caused us issues. It's still very early on, but I think we can look at working on both 1703 (current) and 1803 while we're going through the process. I don't have an ETA offhand, though. And Joel and I will need to figure out how we (eventually) deal with multiple OS versions..

Comment 15

[Ryan VanderMeulen [:RyanVM]]

FYI, reports are that version 1803 is already up to 50% rollout by some metrics.
https://arstechnica.com/gadgets/2018/05/half-of-windows-10-pcs-already-updated-to-the-april-2018-update/

Our own Telemetry data puts the proportion of Firefox release users around 25%, but it's safe to say that the rate of uptake is pretty fast. It's also worth noting that version 1703, our current version in CI, is down to 6% of release users at this point (1709 is 58%).

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

:jfkthame, now that we are running 1803, is there any more work to do on this bug?

Comment 17

[Jonathan Kew [:jfkthame]]

No, that should cover the update we were asking for here. (Thanks!)

Now we should see what currently-skipped tests (e.g. as mentioned in comment 0) can be enabled on CI now that we're running a newer release.

Comment 18

[Jonathan Kew [:jfkthame]]

So I guess my remaining question is how should a chrome.ini manifest (for example) distinguish Windows versions? In layout/inspector/tests/chrome/chrome.ini we currently have skip-if = (os == 'win'). I'd like to enable that test on win10, but it'll still fail on win7. Is there a preferred manifest check for win7 vs win10?

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

I found one test case referenced in the patches in comment 0 and I pushed to try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a7fb6fc06b6564c0b086459bc641a34b33690a15

while we are not perfect, we do know all our test automation machines are on 1803 (otherwise many other tests would fail that are expected to pass, etc.)

Comment 20

[Jonathan Kew [:jfkthame]]

Yeah, I realized we could use a check for bits!=64, and that should work OK in automation. It's not really the correct condition here, but given that we don't run tests on win7/64 or on win10/32, we could get away with it.

I still think it'd be useful to expose the actual windows version in some way that could be easily referenced in the manifest, but it doesn't need to block us here for now.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

This should work for only running on Win10:
skip-if = (os == 'win' && os_version != '10.0')

Usage share for versions <1803 is small enough that we probably don't need to care about having a better version number. There's also a get_windows_version function in mozinfo.py - not sure if that could be extended here or not. We already have similar android_version usage in our test manifests.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

FWIW, ~90% of Firefox Win10 users are on 1709 or newer.