Closed Bug 1679929 Opened 4 years ago Closed 4 years ago

Cap the User-Agent string's reported macOS version at 10.15

Categories

(Core :: Networking: HTTP, defect, P2)

Product:
Core
Component:
Networking: HTTP
Platform:
All
macOS
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
87 Branch
Tracking Flags:
Tracking Status
firefox-esr78 --- fixed
firefox83 --- wontfix
firefox84 --- wontfix
firefox85 --- wontfix
firefox86 --- wontfix
firefox87 --- fixed

People

(Reporter: emk, Assigned: cpeterson)

References

(Blocks 1 open bug)

Details

Attachments

(5 files)

Bug 1679929 - Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r?valentin
47 bytes, text/x-phabricator-request
Details | Review
Bug 1679929 - Add tests for the default User-Agent string. r?valentin
47 bytes, text/x-phabricator-request
Details | Review
Bug 1679929 - Rename SPOOFED_APPNAME to CONST_APPNAME because navigator.appName isn't spoofed. r?valentin
47 bytes, text/x-phabricator-request
Details | Review
Bug 1679929 - Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r=valentin,hsivonen
48 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-esr78+
Details | Review
Bug 1679929 - Add tests for the default User-Agent string. r=valentin
48 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-esr78+
Details | Review

According to bug 1678061, our user-agent string will report 11.0 on Apple Silicon, 10.16 on Intel. (But I can't verify it myself. Feel free to close this bug if it is incorrect.) We should not expose the CPU difference.

Safari hard-codes 10_15_7 for Big Sur (or later) at the moment:
https://bugs.webkit.org/show_bug.cgi?id=216593
https://bugs.webkit.org/show_bug.cgi?id=217364

Why are we exposing the osx version in the user agent in the first place?

I am trying to figure out who is working on this.

(In reply to Mike Hommey [:glandium] from comment #1)

Why are we exposing the osx version in the user agent in the first place?

Because nobody cared about fingerprinting at that point. But we can't unexpose the version string. It will break many websites. What we can do is freezing the version string.

Flags: needinfo?(dd.mozilla)
See Also: → 1680516

Cap the User-Agent string's reported macOS version at 10.15 like Safari does (on macOS 10.15 Catalina and macOS 11.0/10.16 Big Sur).

This will fix webcompat bugs like bug 1680516. Some UA-sniffing sites assume the User-Agent string's macOS version always begin with "10." and they break if the User-Agent string reports version 11.0 on macOS Big Sur (11.0 aka 10.16).

Assignee: nobody → cpeterson
Blocks: 1680516
Severity: -- → S3
status-firefox83: --- → wontfix
status-firefox84: --- → ?
status-firefox85: --- → affected
status-firefox86: --- → affected
status-firefox-esr78: --- → ?
Priority: -- → P2
Hardware: ARM64 → All
See Also: 1680516 →
Summary: Do not report different user-agent string depending on the CPU → Cap the User-Agent string's reported macOS version at 10.15

Try run with the new UA string and UA test:

https://treeherder.mozilla.org/jobs?repo=try&revision=0c9a0c7f8b39ce2ceb4ee06a371b118cfb01ff67&selectedTaskRun=DBB3sUMZQBeaFrzGXi2vrg.0

Unfortunately, there are quite a few orange test failures, but none of them are related to the UA string or UA test.

Fixes webcompat bugs like bug 1680516. Some UA-sniffing sites assume the User-Agent string's macOS version always begin with "10." and they break if the User-Agent string reports version 11.0 on macOS Big Sur (11.0 aka 10.16).

We previously had no tests for the default User-Agent string values, just for the spoofed values when resistFingerprinting is enabled!

Depends on D99856

Interestingly Chrome reports Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 and has problems with downloads from https://www.origin.com/usa/en-us/store/download. As it looks like the world isn't ready yet for a major version bump, and the OS version should indeed be capped for now.

As it looks like the world isn't ready yet for a major version bump, and the OS version should indeed be capped for now.

Just to note that capping the OS version will not make the world more ready. :) Anything which is being introduced in the web platform will be used as-is for what it is (bad or good).

Capping will have the benefit of not increasing the fingerprinting surface, aka it will basically stay the same as of today, which is good.

But we need to check what are also the other values? aka not only UA string.

for(var property in navigator){ 
    var str = navigator[property]
    document.write(property+ ": "+str+"<br />");
}

on macOS 11.1 (20C69)

Firefox Nightly 86.0a1 (2021-01-03) (64-bit)

vibrate: function vibrate() { [native code] }
javaEnabled: function javaEnabled() { [native code] }
getGamepads: function getGamepads() { [native code] }
getVRDisplays: function getVRDisplays() { [native code] }
mozGetUserMedia: function mozGetUserMedia() { [native code] }
sendBeacon: function sendBeacon() { [native code] }
requestMediaKeySystemAccess: function requestMediaKeySystemAccess() { [native code] }
registerProtocolHandler: function registerProtocolHandler() { [native code] }
taintEnabled: function taintEnabled() { [native code] }
permissions: [object Permissions]
mimeTypes: [object MimeTypeArray]
plugins: [object PluginArray]
doNotTrack: unspecified
maxTouchPoints: 0
mediaCapabilities: [object MediaCapabilities]
oscpu: Intel Mac OS X 10.16
vendor: 
vendorSub: 
productSub: 20100101
cookieEnabled: false
buildID: 20181001000000
activeVRDisplays: 
mediaDevices: [object MediaDevices]
serviceWorker: [object ServiceWorkerContainer]
credentials: [object CredentialsContainer]
clipboard: [object Clipboard]
mediaSession: [object MediaSession]
webdriver: false
hardwareConcurrency: 8
geolocation: [object Geolocation]
appCodeName: Mozilla
appName: Netscape
appVersion: 5.0 (Macintosh)
platform: MacIntel
userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:86.0) Gecko/20100101 Firefox/86.0
product: Gecko
language: en
languages: en,en-US,fr,fr-FR
onLine: true
storage: [object StorageManager]

Safari Tech Preview Release 117 (Safari 14.1, WebKit 16611.1.7.2)

geolocation: [object Geolocation]
mediaCapabilities: [object MediaCapabilities]
webdriver: false
maxTouchPoints: 0
cookieEnabled: true
appCodeName: Mozilla
appName: Netscape
appVersion: 5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Safari/605.1.15
platform: MacIntel
product: Gecko
productSub: 20030107
userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Safari/605.1.15
vendor: Apple Computer, Inc.
vendorSub: 
language: fr-fr
languages: fr-fr
onLine: true
plugins: [object PluginArray]
mimeTypes: [object MimeTypeArray]
getStorageUpdates: function getStorageUpdates() { [native code] }
sendBeacon: function sendBeacon() { [native code] }
requestMediaKeySystemAccess: function requestMediaKeySystemAccess() { [native code] }
getGamepads: function getGamepads() { [native code] }
javaEnabled: function javaEnabled() { [native code] }

Microsoft Edge Canary Version 89.0.752.0 (Version officielle) Canary (64 bits)

vendorSub:
productSub: 20030107
vendor: Google Inc.
maxTouchPoints: 0
userActivation: [object UserActivation]
doNotTrack: null
geolocation: [object Geolocation]
connection: [object NetworkInformation]
plugins: [object PluginArray]
mimeTypes: [object MimeTypeArray]
webkitTemporaryStorage: [object DeprecatedStorageQuota]
webkitPersistentStorage: [object DeprecatedStorageQuota]
hardwareConcurrency: 8
cookieEnabled: true
appCodeName: Mozilla
appName: Netscape
appVersion: 5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4372.0 Safari/537.36 Edg/89.0.752.0
platform: MacIntel
product: Gecko
userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4372.0 Safari/537.36 Edg/89.0.752.0
language: en
languages: en,fr,fr-FR,en-GB,en-US
onLine: true
webdriver: false
getBattery: function getBattery() { [native code] }
getGamepads: function getGamepads() { [native code] }
javaEnabled: function javaEnabled() { [native code] }
sendBeacon: function sendBeacon() { [native code] }
vibrate: function vibrate() { [native code] }
clipboard: [object Clipboard]
credentials: [object CredentialsContainer]
keyboard: [object Keyboard]
mediaDevices: [object MediaDevices]
storage: [object StorageManager]
serviceWorker: [object ServiceWorkerContainer]
deviceMemory: 8
bluetooth: [object Bluetooth]
mediaCapabilities: [object MediaCapabilities]
locks: [object LockManager]
wakeLock: [object WakeLock]
mediaSession: [object MediaSession]
presentation: [object Presentation]
usb: [object USB]
scheduling: [object Scheduling]
xr: [object XRSystem]
permissions: [object Permissions]
registerProtocolHandler: function registerProtocolHandler() { [native code] }
unregisterProtocolHandler: function unregisterProtocolHandler() { [native code] }
getInstalledRelatedApps: function getInstalledRelatedApps() { [native code] }
clearAppBadge: function clearAppBadge() { [native code] }
setAppBadge: function setAppBadge() { [native code] }
getUserMedia: function getUserMedia() { [native code] }
requestMIDIAccess: function requestMIDIAccess() { [native code] }
requestMediaKeySystemAccess: function requestMediaKeySystemAccess() { [native code] }
webkitGetUserMedia: function webkitGetUserMedia() { [native code] }

(In reply to Karl Dubost💡 :karlcow from comment #11)

But we need to check what are also the other values? aka not only UA string.

Could you compare the information between ARM64 native and Intel on Rosetta 2? The problem is that (at least) the user-agent is different between them.

I should be able to have access to a M1 in a couple of days.

But we need to check what are also the other [navigator] values? aka not only UA string.

I agree, but I don't think that should block landing this patch to cap the macOS version in the UA string.

oscpu: Intel Mac OS X 10.16
userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:86.0) Gecko/20100101 Firefox/86.0

FWIW, my patch fixes the User-Agent HTTP header and the navigator.userAgent and navigator.oscpu APIs. (Those code paths all read from the same source of truth.)

(In reply to Masatoshi Kimura [:emk] from comment #0)

Safari hard-codes 10_15_7 for Big Sur (or later) at the moment:

At present, on Big Sur on Apple Silicon and on iPadOS, Safari reports 10_15_6. I think we should do the same.

Also, is there any expected benefit from capping the version at 10_15_6 compared to always reporting 10_15_6 even on older systems?

(In reply to Henri Sivonen (:hsivonen) from comment #15)

(In reply to Masatoshi Kimura [:emk] from comment #0)

Safari hard-codes 10_15_7 for Big Sur (or later) at the moment:

At present, on Big Sur on Apple Silicon and on iPadOS, Safari reports 10_15_6. I think we should do the same.

According to WebKit Bugzilla, they will update the user-agent version very soon. (I thought is was already released to the stable Safari release.)

(In reply to Henri Sivonen (:hsivonen) from comment #16)

Also, is there any expected benefit from capping the version at 10_15_6 compared to always reporting 10_15_6 even on older systems?

Safari is capping the version number. If you think we should do the same as what Safari is doing, we should do it ☺

(In reply to Henri Sivonen (:hsivonen) from comment #16)

Also, is there any expected benefit from capping the version at 10_15_6 compared to always reporting 10_15_6 even on older systems?

Some websites may be using the osx version to e.g. give different downloads based on it.

(In reply to Masatoshi Kimura [:emk] from comment #17)

Safari is capping the version number. If you think we should do the same as what Safari is doing, we should do it ☺

Is there a WebKit bug explaining why they are capping the number instead of returning a constant everywhere?

(In reply to Mike Hommey [:glandium] from comment #18)

(In reply to Henri Sivonen (:hsivonen) from comment #16)

Also, is there any expected benefit from capping the version at 10_15_6 compared to always reporting 10_15_6 even on older systems?

Some websites may be using the osx version to e.g. give different downloads based on it.

Does that happen in practice often enough to matter? I've seen some Mac apps offer manual selection of unsupported old versions for old versions of the OS, but, obviously, I wouldn't notice a site doing that kind of UA sniffing.

There is Limit user agent versioning to an upper bound, but I don't think that's it.

This commit about 11_ because of webcompat issues

Thanks. None of these say why Apple capped the version number instead of using the 10_15_6 constant on older systems as well.

Apple knows their platform. I think we should follow their lead.

OTOH, the risk of hard coding version 10.15 even on macOS 10.12 - 10.14 seems low. What could go wrong? A user tries to download some macOS binary from a website and the website bothers to sniff the UA string and insists on serving a binary intended for macOS 10.15 that is incompatible with macOS 10.12?

Here is the approximate distribution of macOS versions for Firefox users (using Fx84 crash stats as a proxy since I don't have real version telemetry handy):

OS X 11.2 	 0.01%
OS X 11.1 	 0.51%
OS X 11.0  	 0.07%
OS X 10.16 	14.01%
OS X 10.15 	40.76%
OS X 10.14 	15.13%
OS X 10.13 	22.40%
OS X 10.12 	 6.89%

(In reply to Henri Sivonen (:hsivonen) from comment #22)

Thanks. None of these say why Apple capped the version number instead of using the 10_15_6 constant on older systems as well.

They say. Please read the comments carefully.
https://bugs.webkit.org/show_bug.cgi?id=216593#c5

Why do it this roundabout way instead of hardcoding this to "10_15_6"_s always? Under what circumstances would osVersion not start with 10? And if we fully hardcoded it, maybe we could encode it directly in the template string, same as the AppleWebKit/ version number.

Oh, I understand now. This is to preserve original OS version on older macOS.

Yes, exactly. That was Charles's request.

(In reply to Henri Sivonen (:hsivonen) from comment #22)

Thanks. None of these say why Apple capped the version number instead of using the 10_15_6 constant on older systems as well.

I messaged Maciej on the Safari team and asked him if they knew of any specific compat problems spoofing 10_15 on older systems. He said: "We made the call pretty late cause 11_0 caused compat issues. We did not want to take unnecessary risk on older OS versions."

FWIW there's a related issue that Blink ran into while advertising 11_0: https://bugs.chromium.org/p/chromium/issues/detail?id=1171998

Jeff, This is Bug 1680516

(In reply to Jeff Gilbert [:jgilbert] from comment #26)

FWIW there's a related issue that Blink ran into while advertising 11_0: https://bugs.chromium.org/p/chromium/issues/detail?id=1171998

(In reply to Karl Dubost💡 :karlcow from comment #27)

Jeff, This is Bug 1680516

Unfortunately, bug 1680516 added just a single UA override to fix gamearter.com, but that Chromium bug lists a bunch of broken Unity games on other itch.io:

https://lucasteles.itch.io/slasherlock
https://graebor.itch.io/sort-the-court
https://asdehielo.itch.io/push-the-square
https://ysopprod.itch.io/spiderdoll
https://samhogan.itch.io/grey-box-testing
https://mattlawr.itch.io/contract-killer
https://dgeisert.itch.io/necro-magic
https://emilysadventure.itch.io/into-the-blizzard
https://pixelfr0ggie.itch.io/can-you-hear-me

We should probably file a new bug to add a UA override for itch.io or fix all the issues by capping the UA string's macOS version to 10.15 in bug 1679929.

When will this bug be ready to land? :tanvi has no activity since Nov. 2020 on Bugzilla and very few activities on Phabricator.

Flags: needinfo?(cpeterson)

(In reply to Masatoshi Kimura [:emk] from comment #29)

When will this bug be ready to land? :tanvi has no activity since Nov. 2020 on Bugzilla and very few activities on Phabricator.

I sent Tanvi email about these other sites. I recommended that we cap the macOS version (at 10.15) now instead of adding UA overrides to fix individual sites. If we want to implement a fancier solution, like revealing different OS versions in the User-Agent HTTP header and navigator.userAgent API, we can add that later.

Flags: needinfo?(cpeterson)

Dennis and I are discussing/wondering is there a way to identify deployed legacy scripts.
if yes We could deploy a site interventions targeting this specific script whichever domain it is on.

Blocks: 1690737

(In reply to Karl Dubost💡 :karlcow from comment #31)

Dennis and I are discussing/wondering is there a way to identify deployed legacy scripts.
if yes We could deploy a site interventions targeting this specific script whichever domain it is on.

Instead of that effort, I think we should proceed with this bug (once Tanvi has r+ed the patch) and make a tiny bit of privacy improvement while at it.

Google intends to cap the Chromium UA string's macOS version number at 10_15_7:

https://groups.google.com/u/1/a/chromium.org/g/blink-dev/c/hAI4QoX6rEo/m/qQNPThr0AAAJ

Tanvi, this seems to be blocked on your review?

Flags: needinfo?(tanvi)

(In reply to Chris Peterson [:cpeterson] from comment #33)

Google intends to cap the Chromium UA string's macOS version number at 10_15_7:

Makes sense to me. Let's do it as cpeterson has proposed.

[:tanvi] awaiting your review on the patch. Thanks!

status-firefox78=wontfix because we don't need to uplift this fix to ESR 78 because it runs in Rosetta on Apple Silicon and reports UA version 10.16, so it doesn't have the 11.0 version compat problem.

status-firefox84: ? → wontfix
status-firefox85: affected → wontfix
status-firefox86: affected → wontfix
status-firefox87: --- → affected
status-firefox-esr78: ? → wontfix
Attachment #9193403 - Attachment description: Bug 1679929 - Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r?valentin!,tanvi! → Bug 1679929 - Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r?valentin
See Also: → client-hints

If I got it right, Google intends to provide the real OS version number through the Sec-CH-UA-Platform-Version
https://groups.google.com/a/chromium.org/g/blink-dev/c/hAI4QoX6rEo/m/qQNPThr0AAAJ

Blink/Chrome has just enabled Client Hints on Feb 11, 2021
[UA-CH] Enable feature and runtime flags by default
https://chromium-review.googlesource.com/c/chromium/src/+/2525742

Client Hints is still being discussed for Gecko Bug 935216

Bumping this to tantek.

Flags: needinfo?(tanvi) → needinfo?(tantek)
See Also: → 1693295

Tantek delegated to me. r+. Let's get this landed.

Flags: needinfo?(tantek)
Pushed by cpeterson@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/86304e6b883a Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r=valentin,necko-reviewers,hsivonen https://hg.mozilla.org/integration/autoland/rev/5415efb793f4 Add tests for the default User-Agent string. r=valentin https://hg.mozilla.org/integration/autoland/rev/5e3d4b5ab7dd Rename SPOOFED_APPNAME to CONST_APPNAME because navigator.appName isn't spoofed. r=valentin

Verified on Nightly.

cpeterson, can you request ESR uplift? Not requesting myself, because I'm unsure if the tests uplift cleanly.

Status: RESOLVED → VERIFIED
Flags: needinfo?(dd.mozilla) → needinfo?(cpeterson)

(In reply to Henri Sivonen (:hsivonen) from comment #42)

cpeterson, can you request ESR uplift? Not requesting myself, because I'm unsure if the tests uplift cleanly.

OK. ESR 78 doesn't have the 11.0 UA compat problems (because Firefox didn't support Apple Silicon until Firefox 84. Firefox <= 83 run in Rosetta and report OS version "10.16"). But UA consistency is good.

ESR 78.8 will be released next week. Since I don't know of any compat problems reporting version 10.16, I will postpone this uplift to ESR 78.9 release (2021-03-23). That will give this 10.15 change more pre-release bake time to identify problems before uplifting to ESR.

Flags: needinfo?(cpeterson)

And always report "Intel" Mac OS X.

Fixes webcompat bugs like bug 1680516. Some UA-sniffing sites assume the User-Agent string's macOS version always begin with "10." and they break if the User-Agent string reports version 11.0 on macOS Big Sur (11.0 aka 10.16).

We previously had no tests for the default User-Agent string values, just for the spoofed values when resistFingerprinting is enabled!

Depends on D106729

D106730 and D106729 don't need re-review. I rebased the mozilla-central patches (including the UA string tests) onto mozilla-esr78 for uplift to ESR 78.

Green Try test run on ESR 78. Some random intermittents fail, but not the UA string tests or any other failures related to UA string:

https://treeherder.mozilla.org/jobs?repo=try&revision=4ca86f6912049e21a092d36a8f26c69a4814bcf9

status-firefox-esr78: wontfix → unaffected

Comment on attachment 9205928 [details]
Bug 1679929 - Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r=valentin,hsivonen

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: This is not a sec bug, but it will cap the Firefox UA string's macOS version to <= 10.15 (so no 10.16 or 11.0) to avoid webcompat issues. Firefox 87 and Safari (and soon Chrome) cap their UA strings' macOS version number at 10.15.
  • User impact if declined: ESR 78 running on macOS 11.0 will report UA string OS version "10.16". ESR 78 doesn't natively support Apple Silicon, so it will run in Rosetta x86 emulation and its UA string reports "10.16" instead of "11.0". The webcompat issues reported so far (like bug 1680516) are all about UA string "Mac OS X 11.0" on Apple Silicon and not about "10.16", but for consistency with Firefox 87 (and Safari and soon Chrome), it would be nice if ESR 78 also capped its UA string version at "10.15".
  • Fix Landed on Version: 87
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): Low risk because the patch is small and adds new UA string tests (Phabricator D106730 which we should also uplift to ESR 78).

Green Try run of mozilla-esr78:

https://treeherder.mozilla.org/jobs?repo=try&revision=4ca86f6912049e21a092d36a8f26c69a4814bcf9

  • String or UUID changes made by this patch: No string or UUID changes
Attachment #9205928 - Flags: approval-mozilla-esr78?
Attachment #9205929 - Flags: approval-mozilla-esr78?

Comment on attachment 9205928 [details]
Bug 1679929 - Cap the User-Agent string's reported macOS version at 10.15 like Safari does. r=valentin,hsivonen

Maintaining parity with mainline sounds like the right call here and the patch is well-scoped. Approved for 78.9esr.

Attachment #9205928 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
Attachment #9205929 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
See Also: → 1806592
See Also: 1806592 →
Depends on: 1841215
Blocks: 1841215
No longer depends on: 1841215
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: