Closed Bug 988725 Opened 10 years ago Closed 10 years ago

Override Firefox UA on *.mozilla.org to include the real version and channel

Categories

(Core :: Networking, enhancement)

Product:
Core
Component:
Networking
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox31 - ---

People

(Reporter: kohei, Unassigned)

References

Details

(Whiteboard: [fxgrowth]) 

http://www.mozilla.org/en-US/firefox/new/ has a warning for people who are using older Firefox releases and we are implementing the same waring site-wide (Bug 935719) but we have realized there's no way to detect ESR builds. Firefox 24.2.0 ESR is exposed as 24.0 in navigator.userAgent (Bug 870868). While we still can detect dot releases using navigator.buildID, this property is also at risk (Bug 583181).

It would be nice if Firefox has the real version number and channel name, so I've proposed a new DOM property called navigator.mozAppDetails (Bug 966030) but looks like a simple UA override might be enough and easier solution for us.

The override should happen only on white-listed domains, *.mozilla.org in particular, to prevent fingerprinting attempts. We believe it would contribute to improving user security, and would like to see this in Firefox 31 that will be the next ESR.

User agent string override examples:

Firefox 31.0 Nightly:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0a1; channel:nightly) Gecko/20100101 Firefox/31.0

Firefox 31.0 Aurora:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0a2; channel:aurora) Gecko/20100101 Firefox/31.0

Firefox 31.0 Beta 1:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0b1; channel:beta) Gecko/20100101 Firefox/31.0

Firefox 31.0 Release:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0; channel:release) Gecko/20100101 Firefox/31.0

Firefox 31.0 ESR:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0; channel:esr) Gecko/20100101 Firefox/31.0

Firefox 31.0.1 ESR:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.1.0; channel:esr) Gecko/20100101 Firefox/31.0
See Also: → 952549
This feels like it's the decision of Gerv and Laurence.
Adding lmandel and gerv.

* What does ESR mean here? Hmm I guess.
  https://www.mozilla.org/en-US/firefox/organizations/

>Firefox ESR is intended for groups who deploy and maintain the desktop environment in large organizations such as universities and other schools, county or city governments and businesses.

It means that users will know they don't use the last version, but they will have no way to adjust, update their browsers, aka it's under the control on who deploy the browser. Why would it help the administrator to have the users asking him to upgrade the park of browsers?

* A whitelisting of *.mozilla.org will encourage other services in Mozilla to use that information. If it's there it will be used and we do not want that. There's a bit of eat your own dog food in there. We want to reduce hooks on identification for others and ourselves. If we are behind a rule that we are the first ones to break, there will be misunderstanding outside.

* Could a notification service for Firefox ESR site administrators could solve your issue?
Flags: needinfo?(lmandel)
Flags: needinfo?(gerv)
Fact: Most enterprises that I know as part of my enterprise consulting use pathetically outdated browser and mail client versions. E.g. Thunderbird 3, Firefox 7, and Firefox 17. Very few have their ducks in row to always use the latest security release. It's very very hard to convince them to update in time.

My view: I don't think a message on http://www.mozilla.org is the right solution. Rather, howtos, builds etc. like http://www.fleetfox.org .
OK, so... site-specific UA overrides are generally an unfortunate necessity, to be avoided. However, I can see the case for mozilla.org being the only site on the web allowed to care exactly what Firefox version you are running, even if the general rendering is the same. (As we aren't exactly going to try and pwn you to work out which security vulnerabilities you have.)

So then the question is, how to encode this, and what to do? The channel is already encoded in the version number in the UA, with the exception of ESR. And adding whole new sections to the UA seems overkill. Not sure whether there would be a compat impact on such a small set of sites, but why take the risk? Alternative proposal: change the UA on mozilla.org only to be rv:31.0e for ESR only, just like we have "a" and "b" now. That's least likely to break any parsing code, I'd guess.

Gerv
Flags: needinfo?(gerv)
> Alternative proposal: change the UA on mozilla.org only to be
> rv:31.0e for ESR only, just like we have "a" and "b" now.
> That's least likely to break any parsing code, I'd guess.

Yes, we'd just like to distinguish ESR builds from Release builds so it will meet the needs of mozilla.org (and maybe SUMO). "channel:esr" might be verbose here.

A possible use case: once Firefox 32 is out, Firefox 31 Release channel users will see a message encouraging them to update now, while Firefox 31 ESR users will see no message or a different message encouraging them to ask their admin to update.

It's impossible for us to do the same thing now because ESR and Release channels share the same UA string. I think such a use case is acceptable on Mozilla properties. The UA string should remain on other sites in terms of privacy and compatibility.
Blocks: 939470
1) enterprise end users rarely go to mozilla.org 2) can't do anything 3) admins are very very hard to convince to update, because they have internal policies *against* updating. Thus, I think this isn't going to change much.
The energy is much better spent by educating admins how they can adapt their internal update policies to allow for security updates, without the (lengthy) internal testing procedure, and to test new ESR releases early in beta stage, not only when they are already released.
We may want to show ESR users no message. That's probably fine. The actual problem here is *non-ESR* users. We'd like to show Firefox 31 non-ESR users a warning once Firefox 32 is out. So we still need a way to distinguish ESR from non-ESR.
Note that if we decide to do anything here, we also need to pay attention to any third-party library loaded on any of the mozilla.org subdomains to make sure that their UA sniffing doesn't break.  Given the number of mozilla.org subdomains we have around, this is quite non-trivial.
Kohei: Not necessarily. You could just show it to all users, if you accept BenB's logic that ESR users don't regularly visit mozilla.org.

Ehsan: Right. Which is why minimal changes to the UA are a better idea. Hence my suggestion of 30.0e. It would have to be pretty detailed UA sniffing which was OK with 30.0a but barfed on 30.0e.

Gerv
We could also add a different API, exposed only on *.mozilla.org, to expose the information we want to instead of using UA overrides, since we presumably control the code that would be looking for the info and that code does not _have_ to use the UA string.
bz: isn't that bug 966030, which was WONTFIXed in favour of this bug (as being simpler to do)?

Gerv
Bug 966030 might be the better route, because it would also allow for bug 732527 to be settled through a common system.
> bz: isn't that bug 966030

Yes, looks like it.  ;)

> as being simpler to do

This part is not clear to me at this point.
(In reply to comment #13)
> > bz: isn't that bug 966030
> 
> Yes, looks like it.  ;)
> 
> > as being simpler to do
> 
> This part is not clear to me at this point.

So what I suggested in bug 966030 comment 6 was to send a special user agent string to a *specific* Mozilla server (aka, not *.mozilla.org).  This bug is most definitely not what I suggested there, and is much more difficult to pull off than my suggestion.
Sending a special UA string to *specific* Mozilla servers would be fine. Maybe only www.mozilla.org and SUMO need this.
I think we actually have all of the information that we need in the UA already. The ESR releases and their corresponding versions are updated alongside all new Firefox releases. The difference is in the content and the associated version bump. You can test for the latest version by either the latest Firefox release version or the latest ESR version. Sample latest version chart below:

Firefox 24 / ESR 24.0.0 / ESR 17.7.0
Firefox 25 / ESR 24.1.0 / ESR 17.8.0
Firefox 26 / ESR 24.2.0
Firefox 27 / ESR 24.3.0
Firefox 28 / ESR 24.4.0
Firefox 29 / ESR 24.5.0
Firefox 30 / ESR 24.6.0
Firefox 31 / ESR 31.0.0 / ESR 24.7.0
Firefox 32 / ESR 31.1.0 / ESR 24.8.0
Firefox 33 / ESR 31.2.0

This will require updating the UA detection logic. If the request is to only alert the user to outdated Firefox versions on a single site, updating the UA detection logic should not be a huge deal. Can you proceed with this information?
Flags: needinfo?(lmandel)
The problem is the minor version is hidden from the UA string.
The UA string for any 24.x versions is "24.0".
lmandel: Firefox non-ESR 24.0, ESR 24.0, ESR 24.1.0, ESR 24.2.0, ... all share the same UA string after Bug 870868 is in place. Unfortunately there's no way to distinguish them.

Currently mozilla.org treats Firefox 24.x as up-to-date. This is fine for the latest 24.x ESR users but non-ESR users older 24.x ESR users are seeing a wrong message saying they're using the latest version.

If we treat Firefox 24.x as an outdated version, the latest 24.x ESR users will see a wrong message saying they're using an insecure version.

To resolve those issues we need a way to detect ESR builds in some way, a custom DOM property (Bug 966030), UA string tweaks (this bug) or something else.
> Unfortunately there's no way to distinguish them.

That's right. On mozilla.org, you want to warn users who have outdated browser with security holes. The goal of the UA string reduction is to avoid that sites and see which security holes I have (or even if). So, you will never be able to use the normal UA string or other browser information for normal sites to fulfill the purpose. Thus, the only solution is to make a special exception for www.mozilla.org. (Or to use entirely different means to warn users.)
I think we should do this using the "30.0e" variant for as small a list of sites as we can get away with. Does anyone have a list of sites which need to know this information?

Gerv
Verdi: does SUMO need this?
Flags: needinfo?(mverdi)
(In reply to Kohei Yoshino [:kohei] from comment #21)
> Verdi: does SUMO need this?

Yes - this will allow sumo to have messaging consistent with mozilla.org and to direct people to the proper solution.
Flags: needinfo?(mverdi)
A couple of questions on proposed implementation:

1.) If we are to have a custom UA override on *.mozilla.org, would it require a secure connection to work? (I'm assuming yes). www.mozilla.org does not yet use to https by default, but I believe we're not that far away from this (This is more a note to us bedrock developers).

2.) We also have to consider a means of testing an override both for local development and QA. Would there be scope to include a custom preference in about:config to allow specifying a testing domain to add to the whitelist, and disabling SSL check if one exists? This would mean we could keep the white-list as small as possible (not needing to add things like demo or staging servers), as well as making local development & testing easier.
1) No, that's not the plan as far as I know.

2) The list is currently a set of prefs anyway, so adding more in that way is already easy.

Gerv
Can someone implement this feature? As I mentioned before, Firefox 31 ESR, the current Nightly, would be the target milestone.
tracking-firefox31: --- → ?
Wait, can't we use the navigator.buildID DOM property for this purpose?  The idea is to test what Firefox 24 returned from navigator.buildID, and then compare it against what navigator.buildID returns if the UA string indicates that you're on Firefox 24.  If the strings are the same, you're on the outdated Firefox 24.  If not, you're on ESR.
Unfortunately Firefox 24 and Firefox 24 ESR share the same navigator.buildID, so we cannot show different messages to those users. We can just detect if the version is up-to-date, though.

https://wiki.mozilla.org/Releases/Firefox_24/Test_Plan#Build_Info
https://wiki.mozilla.org/Releases/Firefox_24_ESR/Test_Plan#Firefox_24_ESR

Also, this property is at risk of removal (Bug 583181) as I noted in my comment 0.
(In reply to comment #27)
> Unfortunately Firefox 24 and Firefox 24 ESR share the same navigator.buildID,
> so we cannot show different messages to those users. We can just detect if the
> version is up-to-date, though.
> 
> https://wiki.mozilla.org/Releases/Firefox_24/Test_Plan#Build_Info
> https://wiki.mozilla.org/Releases/Firefox_24_ESR/Test_Plan#Firefox_24_ESR
> 
> Also, this property is at risk of removal (Bug 583181) as I noted in my comment
> 0.

Ah, bummer.

So I think the next step is for someone to respond to Gerv's comment 20.
(In reply to Kohei Yoshino [:kohei] from comment #25)
> Can someone implement this feature? As I mentioned before, Firefox 31 ESR,
> the current Nightly, would be the target milestone.

Was this something we had planned on landing and that would require blocking?
Flags: needinfo?(kohei.yoshino)
Yes: as soon as we have a list of sites which need to know this info, we can implement it. The list doesn't have to be complete, of course - we can start with some and expand as necessary, subject to shipping deadlines. But I think we should avoid doing it for "*.mozilla.org" if we can.

kohei: which domains did _you_ want to use this on?

Gerv
(In reply to Gervase Markham [:gerv] from comment #30)
> kohei: which domains did _you_ want to use this on?

Only www.mozilla.org and support.mozilla.org AFAIK.
Flags: needinfo?(kohei.yoshino)
OK. Let's get this done for those two domains; if other people turn up, we can evaluate how to help them at the time.

Gerv
Other sites that expressed interest in this are both MDN and AMO.
Blocks: 789731
Firefox 31 is already Aurora. How can we proceed this?
Kohei, just make a patch and ask the right reviewer :).
gerv already mentioned in comment 24 that there is a pref.
A quick search for "agent" in about:config shows: "general.useragent.site_specific_overrides" (true by default). If you search for that on Google and on http://lxr.mozilla.org , you'll also find "general.useragent.override.", which seesm to be the actual site prefs. The code is in
http://mxr.mozilla.org/comm-central/source/mozilla/netwerk/protocol/http/UserAgentOverrides.jsm

So, make a patch that adds www.mozilla.org and sumo.mozilla.org as hosts with your desired overrides to "general.useragent.override.".

A problem might be that you maybe can set only the full string, not parts of it. You may have to extend the above UserAgentOverrides.jsm.
You can set parts of it; look for the overrides which contain a "#" character - they are regexps.

Gerv
I think that what has been suggested is that we use the existing UA override mechanism for the override. AFAIK, this mechanism has only been used on B2G. I don't know if it can be used on desktop. It is worth mentioning that the mechanism and pref file for B2G are now dynamically updated via server side information. The simplest thing may be to modify the actual UA on the ESR branch. This does mean that we have to ensure that this change persists each time we create a new ESR branch. (That's currently every 7 releases.)
(In reply to Lawrence Mandel [:lmandel] from comment #37)
> I think that what has been suggested is that we use the existing UA override
> mechanism for the override. AFAIK, this mechanism has only been used on B2G.
> I don't know if it can be used on desktop. 

Here's the code:
http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/UserAgentOverrides.jsm
I can't immediately see why it couldn't be used on desktop.

> It is worth mentioning that the
> mechanism and pref file for B2G are now dynamically updated via server side
> information. 

Really? Awesome :-) When did that get done?

> The simplest thing may be to modify the actual UA on the ESR
> branch. This does mean that we have to ensure that this change persists each
> time we create a new ESR branch. (That's currently every 7 releases.)

Are you suggesting having the ESR UA different to the corresponding Firefox version's non-ESR UA for _every_ site on the web?

Gerv
31 is going to leave aurora to move to beta next Monday. So, this change will have to land soon to be part of 31. Who could take care of the change?
Not tracking for now. We will wait for an uplift request.
tracking-firefox31: + → -
(In reply to Gervase Markham [:gerv] from comment #38)
> > It is worth mentioning that the
> > mechanism and pref file for B2G are now dynamically updated via server side
> > information. 
> 
> Really? Awesome :-) When did that get done?

B2G 1.2. See bug 927429.
> 
> > The simplest thing may be to modify the actual UA on the ESR
> > branch. This does mean that we have to ensure that this change persists each
> > time we create a new ESR branch. (That's currently every 7 releases.)
> 
> Are you suggesting having the ESR UA different to the corresponding Firefox
> version's non-ESR UA for _every_ site on the web?

Erm, yes but unintentionally. Please ignore my comment. As Gerv said, That's going to apply the change to broadly.

On B2G we have the ability to hard code an UA override using a pref. These prefs get overwritten by the dynamic mechanism. I wonder if we can specify the same pref on desktop, where we don't use the dynamic mechanism, and have it persist.
Knowing the exact Firefox version of user on mozilla.org will also allow us to know if offering a reset option is appropriate. If a user is trying to download the same exact version of Firefox that they are running it is likely they are trying to fix a problem. For more info see Bug 1027318.
Rewriting URLs instead of UA sniffing.

This clicked when reading this blog post about "about:home"
http://www.mkelly.me/what-is-a-snippet.html

> Whenever you view about:home, Firefox checks to see if it 
> has fetched snippet code within the last 24 hours. If it 
> has, it loads the snippet code. If it hasn't, it makes a 
> request and fetches the snippet code, and saves it. The 
> URL used for this request is long and contains several 
> pieces of info useful for deciding which snippets to send 
> you. On one of my test copies of Firefox 30, about:home 
> fetches snippets with this URL:
> 
> https://snippets.mozilla.com/4/Firefox/30.0/20140605174243/Darwin_Universal-gcc3/en-US/release/Darwin%2013.2.0/default/default/
> 
> The most useful parts of the URL are the version (30.0), the 
> locale (en-US), and the release channel (release). These 
> let us do things like localize snippets for different locales 
> or create snippets that promote new features to the release 
> channels that already have them.


Why the browser would not rewrite on the client side the URI is requesting? 
The browser knows exactly what it is and it knows which URIs you are about to request. 

So when the user access http://www.mozilla.org/en-US/firefox/new/, the browser can say:
"Oh that's in my URI list that I should rewrite let me do that before sending the request
http://www.mozilla.org/en-US/firefox/new/?version=$VERSION&channel=esr"

This is sandboxed. 
It's based on request (aka client intent).
I think we have agreement here to UA override to contain this info for www.mozilla.org and support.mozilla.org. I agree Karl's mechanism would be as good or better, but it doesn't exist and the UA mechanism does. So if people still want this to happen, they should patch the UA override mechanism and see if it works on desktop :-)

Gerv
Blocks: 1062345
Flags: qe-verify?
Flags: firefox-backlog+
No longer blocks: 1062345
Whiteboard: [fxgrowth]
(In reply to Gervase Markham [:gerv] from comment #44)
> I think we have agreement here to UA override to contain this info for
> www.mozilla.org and support.mozilla.org. I agree Karl's mechanism would be
> as good or better, but it doesn't exist and the UA mechanism does. So if
> people still want this to happen, they should patch the UA override
> mechanism and see if it works on desktop :-)
> 
> Gerv

Yes, let's move forward to have www.mozilla.org and support.mozilla.org whitelisted to allow them to view full version numbers including ESR.
Note that bug 1065525 implemented an alternative method for www.mozilla.org and support.mozilla.org to get the version and other application info but that would depend on the JS library from https://github.com/Unfocused/mozilla-uitour
Blocks: 935719
:MattN - Should we be able to send "appinfo" as the configName param to Mozilla.UITour.getConfiguration today? Will that give us full version information?
Flags: needinfo?(MattN+bmo)
That should work as of bug 1065525 (in Nightly now).
Flags: needinfo?(MattN+bmo)
So Mozilla.UITour.getConfiguration("appinfo", callback) is available soon. That sounds good. We can quickly test this using the Error Console (not Web Console) of Firefox:

(Flip the devtools.errorconsole.enabled pref to true)

> Components.utils.import("resource://gre/modules/Services.jsm"); 
> Services.appinfo.defaultUpdateChannel;

And the results:

* defaultUpdateChannel: nightly
* distributionID: org.mozilla
* isOfficialBranding: false
* isReleaseBuild: false
* name: Firefox
* vendor: Mozilla
* version: 35.0a1

Both the channel and full version can be found here. Awesome. Let's use this then.
I'm moving the dependencies to Bug 1065525 and marking this bug WONTFIX. Thanks everyone who participated!
No longer blocks: 789731, 935719, 939470, 1027318
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
(In reply to Kohei Yoshino [:kohei] from comment #49)
> And the results:

Note bug 1070138, though.
(In reply to :Gavin Sharp [email: gavin@gavinsharp.com] from comment #51)
> Note bug 1070138, though.

Thanks for the info. We'll only use the defaultUpdateChannel and version anyway :)
You need to log in before you can comment on or make changes to this bug.