Closed Bug 649435 Opened 13 years ago Closed 12 years ago

Remove the frozen date from the User Agent String

Categories

(Core :: Networking: HTTP, defect)

Product:
Core
Component:
Networking: HTTP
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: sdwilsh, Assigned: sdwilsh)

References

(
URL
)

Details

Attachments

(1 file)

v1.0
1.60 KB, patch
khuey
: review+
bzbarsky
: review-
Details | Diff | Splinter Review
Attached patch v1.0 — — Splinter Review 
When dwitte blogged about us changing our user agent string for Firefox 4, the post also stated this:
If you are presently using the build date or otherwise depending on the exact format of the “Gecko/yyyymmdd” token, stop! In the next major release of Gecko, this will be replaced by the string “Gecko/”. If you rely on the fact that this will be followed by a date string, you will break.
For testing purposes, the unfrozen build date is still included in nightly builds of Firefox, but it will be removed in future nightly builds.

We missed this in Firefox 5, but we can get it in Firefox 6.The patch I'm attaching here (basically undoing attachment 470116 [details] [diff] [review] plus some removal in the http code) does just that.  We'll just have "Gecko/" in the useragent for now and evermore.  What my useragent looks like with this patch (local build):
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a1) Gecko/ Firefox/6.0a1

Note: this does *not* remove the buildid from navigator.buildId.  If someone wants that changed, you should file a new bug as this is *not* the place to discuss that.
Attachment #525452 - Flags: superreview?(jst)
Attachment #525452 - Flags: review?(khuey)
Attachment #525452 - Flags: review?(bzbarsky)
This was not meant to be a security bug...
Group: core-security
This was not meant to be a security bug...
Group: core-security
Try results will show up here: http://tbpl.mozilla.org/?tree=MozillaTry&rev=bf91ebda27b9
Whiteboard: [needs review khuey][needs review bz][needs sr jst]
Now that the rv: version == Firefox/ version, can we remove one of those as well?
(In reply to comment #4)
> Now that the rv: version == Firefox/ version, can we remove one of those as
> well?
Perhaps, but in a different bug!  I'm super serious about keeping this bug scoped to the one thing we said we were going to do, which is to remove the frozen date.
"Gecko/" is broken syntax. Can we move rv:6.0a1 after the slash?
(In reply to comment #6)
> "Gecko/" is broken syntax.
Not really sure how it is "broken".  It is not like there is a specification to conform to or anything.  This is *exactly* what we said we'd do in the blog post.  (I believe the slash is left in because it is less likely to break things that consume this, although we'll have to see).

> Can we move rv:6.0a1 after the slash?
Perhaps, but in a different bug!  I'm super serious about keeping this bug
scoped to the one thing we said we were going to do, which is to remove the
frozen date.
(In reply to comment #7)
> (In reply to comment #6)
> > "Gecko/" is broken syntax.
> Not really sure how it is "broken".  It is not like there is a specification to
> conform to or anything.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.8

> This is *exactly* what we said we'd do in the blog
> post.  (I believe the slash is left in because it is less likely to break
> things that consume this, although we'll have to see).

Right. I don't think we should do it exactly this way just because we blogged about it.

> > Can we move rv:6.0a1 after the slash?
> Perhaps, but in a different bug!  I'm super serious about keeping this bug
> scoped to the one thing we said we were going to do, which is to remove the
> frozen date.

Except that I'm bringing it up as a solution to a problem with this bug.
(In reply to comment #8)
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.8
I don't believe what we have here is a "product token".  It isn't a version number (although that is only a SHOULD requirement), but rather a build date.  Regardless, this is a decision for the Necko owner or a peer, and the sr.
Comment on attachment 525452 [details] [diff] [review]
v1.0

r=me for build changes.
Attachment #525452 - Flags: review?(khuey) → review+
Whiteboard: [needs review khuey][needs review bz][needs sr jst] → [needs review bz][needs sr jst]
(In reply to comment #9)
> (In reply to comment #8)
> > http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.8
> I don't believe what we have here is a "product token".

Sorry, I thought I was still on the same document that led me to section 3.8. Here's the user-agent section referring to it: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.43
(In reply to comment #11)
> Sorry, I thought I was still on the same document that led me to section 3.8.
> Here's the user-agent section referring to it:
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.43
Right, but I still think we disagree.  I believe that the Gecko token falls under "comments identifying the agent" as opposed to a "product token".  I believe this changed when we froze the buildid, even if the syntax was still a "product token".  Again, I'm going to defer to my reviewer and sr here.
Comments are in the parentheses, afaik. Gecko/date is just a rather poor product token (hence this bug), but still a product token. In fact, among our product tokens, it's the most important one.
As I've said several times, I think we should go whole hog:

User-Agent: Firefox/version (cpu os)

but short of that I have no opinion; especially, I have no idea whether "Gecko/" is a better or worse option, in terms of compatibility, than "Gecko/something" or "Gecko".
... however, I would say that this is a case where what matters is the behavior of actual sniffing code; not what the standard says a user-agent string is supposed to look like.  We're in better shape with web standards than we used to be, but to first order, nobody reads the HTTP spec.
I agree w/Dão -- syntactically you could do "Gecko/something", "Gecko", or
"(Gecko/)", but not "Gecko/".  It would be nice if the "something" were useful,
but if it's not then kill the slash, too.
(In reply to comment #14)
> As I've said several times, I think we should go whole hog:
> 
> User-Agent: Firefox/version (cpu os)
> 
> but short of that I have no opinion; especially, I have no idea whether
> "Gecko/" is a better or worse option, in terms of compatibility, than
> "Gecko/something" or "Gecko".

Throwing a gear in the works is sometimes the best way to change. Developers are (unfortunately) usually quite lazy and will not fix something unless it is broken. If we break this for good, as suggested above, it may finally kick people do stop using user agent sniffing.

Though this should be brought up in the newsgroups as this bug is quite specific and a good first step for Firefox 6.
(In reply to comment #14)
> As I've said several times, I think we should go whole hog:
> 
> User-Agent: Firefox/version (cpu os)
Perhaps, but in a different bug!  I'm super serious about keeping this bug
scoped to the one thing we said we were going to do, which is to remove the
frozen date.

(In reply to comment #15)
> ... however, I would say that this is a case where what matters is the behavior
> of actual sniffing code; not what the standard says a user-agent string is
> supposed to look like.  We're in better shape with web standards than we used
> to be, but to first order, nobody reads the HTTP spec.
This is all very, very true.  I can't find the comment (I spent more time than I'd like to admit looking for it today) where this was discussed, so it's possible it was in the newsgroups or in person with dwitte, but keeping the slash there was going to break parsers less than taking it out.  Without finding that piece of information, it's simply my word right now.

I'm fine with removing the slash, but again, I'm going to defer to my reviewer/sr here.
(In reply to comment #18)
> > User-Agent: Firefox/version (cpu os)
> Perhaps, but in a different bug!  I'm super serious about keeping this bug
> scoped to the one thing we said we were going to do, which is to remove the
> frozen date.

Fair enough!  I only brought it up because I assume Reed cc:ed me on the bug because I argued for the more aggressive change in the newsgroups (and I lost that argument, and I don't want to fight it again).

I've said everything I have to say about removing the frozen date.  I'm fine with either leaving the slash in or taking it out.
(In reply to comment #15)
> ... however, I would say that this is a case where what matters is the behavior
> of actual sniffing code; not what the standard says a user-agent string is
> supposed to look like.  We're in better shape with web standards than we used
> to be, but to first order, nobody reads the HTTP spec.

I wouldn't care if the spec said something stupid. But it makes a lot of sense for a UA string parser to expect a slash to separate two parts a token. Let us please not introduce a new quirk.
Dão, do you know whether any UA string parsers do that?  On the web, indexOf("Gecko/") is very common, but that's about it.  Maybe things look differen on the server-side?

Note that sites that look for the "rv:M.N" part often look for a ')' following it, so I don't think the suggestion in comment 6 is viable.

If we really think that just using "Gecko/" is broken, how about "Gecko/Gecko"?  ;)
(In reply to comment #21)
> If we really think that just using "Gecko/" is broken, how about "Gecko/Gecko"?
>  ;)

Zimbra requires the next character after "Gecko/" to be a digit. So "Gecko/ " and "Gecko/Gecko" break Zimbra, but e.g. "Gecko/0" wouldn't break Zimbra. ("Gecko/0" is also shorter than "Gecko/Gecko".)
How about "Gecko/2"?
How about Gecko/6.0? Obvious choice...
Whether we should freeze the rv: number at 5.0 and change our messaging around that would be up for debate then, I guess, although we could also duplicate the version forever.

(In reply to comment #21)
> Dão, do you know whether any UA string parsers do that?  On the web,
> indexOf("Gecko/") is very common, but that's about it.  Maybe things look
> differen on the server-side?

I have no representative data. I have written my own server-side parser some time ago that happens to identify Gecko by just looking for "Gecko" after filtering out "Webkit", but then it looks for (?<=Gecko/\d{8} )[^ ]+(?=/\d) to differentiate and identify random Gecko browsers, which will break either way.
> How about Gecko/6.0? 

If we need the digit after the '/', then this does seem like the obvious choice, though I'd prefer "Gecko/6", I think....
I've worked on some server-side stats code before that follows RFC2616 for the User-Agent. The complete set of rules used to define the User-Agent header are as follows:

User-Agent      = "User-Agent" ":" 1*( product | comment )
product         = token ["/" product-version]
comment         = "(" *( ctext | quoted-pair | comment ) ")"
token           = 1*<any CHAR except CTLs or separators>
product-version = token
ctext           = <any TEXT excluding "(" and ")">
quoted-pair     = "\" CHAR
CHAR            = <any US-ASCII character (octets 0 - 127)>
CTL             = <any US-ASCII control character (octets 0 - 31) and DEL (127)>
separators      = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <"> | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT
TEXT            = <any OCTET except CTLs, but including LWS>
SP              = <US-ASCII SP, space (32)>
HT              = <US-ASCII HT, horizontal-tab (9)>
LWS             = [CRLF] 1*( SP | HT )
CRLF            = CR LF
CR              = <US-ASCII CR, carriage return (13)>
LF              = <US-ASCII LF, linefeed (10)>

As you can see, "Gecko/" is not valid because product-version is missing, and product cannot include "/" (because "token" exludes "separators").

In the code I saw, I think this would result in a lot of NULLs in the database. I'm not sure what effect this would have on the reporting side of things, but going by the overall code-quality there, I doubt it's prepared for that...

I think replacing the date with the Gecko version would be a suitable first attempt at fixing this bug. The version would then be duplicated because of the rv: section, but any attempt to resolve that should be in a separate bug as there are further compatibility issues there.

So my UA at the moment is:

Mozilla/5.0 (X11; Linux i686; rv:6.0a1) Gecko/20110413 Firefox/6.0a1

As things are, at release it will be:

Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0

But I'm proposing:

Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/6.0 Firefox/6.0



A later step could be:

Mozilla/5.0 (X11; Linux i686) Gecko/6.0 Firefox/6.0

And then perhaps (getting optimistic here...):

Mozilla (X11; Linux i686) Gecko/6.0 Firefox/6.0

Followed by:

Mozilla Firefox/6.0 (X11; Linux i686) Gecko/6.0
(In reply to comment #25)
> If we need the digit after the '/', then this does seem like the obvious
> choice, though I'd prefer "Gecko/6", I think....

Would we keep incrementing the number to double digits? Like:
Mozilla/5.0 (X11; Linux i686; rv:14.0) Gecko/14 Firefox/14.0
?
Now we have - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
Proposal - Mozilla/5.0 (Windows NT 6.1; WOW64) Gecko/2.0 Firefox/4.0
In future - Firefox Mozilla/4.0 (Windows NT 6.1; WOW64) Gecko/2.0
(In reply to comment #28)
> In future - Mozilla Firefox/4.0 (Windows NT 6.1; WOW64) Gecko/2.0
fixed
Hi.  This bug is not the place to discuss where you want the user agent to go (take it to the newsgroup).  This bug is about removing the frozen date from the user agent string and nothing more.  I'm super serious about keeping this bug scoped to the one thing we said we were going to do, which is to remove the frozen date.
I personally believe that we should not make this change in the forseeable future (the next 12 months at least), and we should just leave it at 20100101. Risk pretty high, reward pretty low.
What would be substantially different 12 months from now? I can think of having more beta testers (good) and a slightly higher chance that some particularly stupid sniffers have started depending on 20100101 (not good).
Of all the things in the world, I don't think depending on 20100101 is a big deal. We can keep it around permanently, assuming the risk/reward remains roughly like it does today.
We can discuss the risk/reward, I'm just wondering why you think now is a particularly bad time to do this.

I guess there's also a chance that sniffers will actually stop looking at the date after realizing that it's not updated anymore...
There's no need to sniff Gecko build date and admins need to realize this. Faster removing, the better to change this behavior.
> Would we keep incrementing the number 

We should clearly do Gecko/006 this cycle... ;)
This change should wait for the version 10 problem, which comes soon.
There should be decision *now* how the complete string will look like.
That should be communicated early before version 10 ships.
My favs are
Mozilla/5.0 (Windows NT 6.1) Gecko/rv:10.0 Firefox/10.0   or
Mozilla/5.0 (Windows NT 6.1) Gecko/rv:10.0 Firefox/0010.0  or
Mozilla/5.0 (Windows NT 6.1) Gecko/rv:0010.0 Firefox/10.0
(and sorry for the spam)
A lot of this has been discussed in bug 572661 already which should be reviewed before deciding anything here, and the patch proposed in the bug here is quite similar to attachment 467573 [details] [diff] [review], except for removing the frozen date.

Gecko applications other than Firefox opted for not freezing their UA date, and those are still useful at least for nightly builds to be unambiguous. Rather than removing the support for it entirely, maybe just allow MOZ_UA_BUILDID to be an empty string, in which case only "Gecko" is presented (and "Gecko/" won't be in compliance with RFC2616 as the product-string after "/" can't be empty).

Replacing Gecko/<date> with Gecko/<version> in the UA has been won't-fixed per bug 588909. I'm with Dão's comment #34 to let the current changes sit for a while before making further substantial modifications to the string breaking sites for users (who will usually blame it on the browser, not the site).
This shouldn't create any issues with 99,9% or more sites.
Opera & IE didn't have Gecko string.
Safari and Chrome have "like Gecko" string without build date.

So keeping it and creating time for adopting sniffing the frozen date or not, it a bad idea.
(In reply to comment #37)
> This change should wait for the version 10 problem, which comes soon.
> There should be decision *now* how the complete string will look like.
> That should be communicated early before version 10 ships.
> My favs are
> Mozilla/5.0 (Windows NT 6.1) Gecko/rv:10.0 Firefox/10.0   or
> Mozilla/5.0 (Windows NT 6.1) Gecko/rv:10.0 Firefox/0010.0  or
> Mozilla/5.0 (Windows NT 6.1) Gecko/rv:0010.0 Firefox/10.0
Perhaps, but in a different bug!  I'm super serious about keeping this bug
scoped to the one thing we said we were going to do, which is to remove the
frozen date.
Blocks: 651503
Blocks: 651504
Filed bug 651503 and bug 651504 as alternatives to this bug which avoid producing invalid UAs per comment 26. Comments about either of those proposals should go there to avoid any more super serious comments here.
(In reply to comment #39)
> This shouldn't create any issues with 99,9% or more sites.

...except that one of those supposedly 0,1% of sites was enough (bug 450973) to drop the time stamp and to crop the build date down to a length of eight. ;-) 

> Safari and Chrome have "like Gecko" string without build date.

Yes, and some sniffers actually distinguish those by differentiating between "Gecko" and "Gecko/". So, the underlying question (as with all of these bugs) is whether to break such pages on purpose to force them adapt immediately, or to have more gradual changes like freezing the date (which I agree is ugly) to promote changing bad practices without breaking current methods they may use.

As for my comment #38, if Firefox wants to remove the frozen date for its release builds rather, so be it, but leave the option available for nightly builds (where the actual date matters) and for applications which don't want
to follow that step.
(In reply to comment #42)
> (In reply to comment #39)
> > This shouldn't create any issues with 99,9% or more sites.
> 
> ...except that one of those supposedly 0,1% of sites was enough (bug 450973) to
> drop the time stamp and to crop the build date down to a length of eight. ;-) 

Yep, I remember this times when also you need to browse bank sites with IE or IE user agent. Old times :)
But now it's more like evangelism bug and we should prioritize browser speed by reducing the amount of data and entropy sent out in HTTP requests.

P.S. Can we get review on patch ? ;)
I've seen no argument suggesting that Gecko/ would be better than Gecko/6.0 or Gecko/6. So now that the rival bugs are filed, we can probably close this.
I agree with Dao and dveditz. Just "Gecko/" is simply a HTTP spec violation. There is only "product/version (comment) component/version (comment)". The spec is unambiguous that if you remove version, you must also remove the slash. This patch makes our UA string simply invalid and unparsable.

As others said, I see no reason not to just do "Gecko/2.1" (whatever is the Gecko version) instead. (In fact, the luckily coincidence that "Gecko/2001" and "Gecko/2.0" share the "/2" might break even less sites than "Gecko/6".)

"Gecko/" is about the worst of all options.
(In reply to comment #45)
> As others said, I see no reason not to just do "Gecko/2.1" (whatever is the
> Gecko version) instead. (In fact, the luckily coincidence that "Gecko/2001" and
> "Gecko/2.0" share the "/2" might break even less sites than "Gecko/6".)

I was under the impression that Gecko version numbers are now the same as Firefox's: http://mozilla.github.com/process-releases/draft/development_specifics/#versioning-gecko
OK, nevermind the part in brackets.
Comment on attachment 525452 [details] [diff] [review]
v1.0

I think we should wontfix this as filed and consider moving the whole thing into a comment so we can then use "Gecko/".
Attachment #525452 - Flags: review?(bzbarsky) → review-
So basing on "Gecko Is Gecko" http://geckoisgecko.org/
and "How to properly sniff for a version of Gecko" http://archive.bclary.com/xbProjects-docs/geckoGetRv/

Using
Mozilla/5.0 (Windows NT 6.1; Win64; x64) Gecko/rv:7.0a1 Firefox/7.0a1

instead of how it's now
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:7.0a1) Gecko/20110527 Firefox/7.0a1

or how it will be when we will fix this bug
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:7.0a1) Gecko/ Firefox/7.0a1
(In reply to comment #49)
> or how it will be when we will fix this bug
> Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:7.0a1) Gecko/ Firefox/7.0a1

No, the definition of "comment" that Boris referred to in comment 48 is in comment 26.

I believe something like this would be suitable:

Mozilla/5.0 (X11; Linux i686; Gecko/rv:7.0a1) Firefox/7.0a1

It gets rid of the frozen date and is compatible with the code here: http://archive.bclary.com/lib/js/geckoGetRv.js and also with pages that look for "Gecko/" somewhere in the UA string.
Attachment #525452 - Flags: superreview?(jst)
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → WONTFIX
> Mozilla/5.0 (X11; Linux i686; Gecko/rv:7.0a1) Firefox/7.0a1

And that actually makes sense.

sdwilsh, why did you mark this WONTFIX? And even without any justification?
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
(In reply to comment #50)
> I believe something like this would be suitable:
> 
> Mozilla/5.0 (X11; Linux i686; Gecko/rv:7.0a1) Firefox/7.0a1
Perhaps, but in a different bug!  I'm super serious about keeping this bug scoped to the one thing we said we were going to do, which is to remove the frozen date.

(In reply to comment #51)
> sdwilsh, why did you mark this WONTFIX? And even without any justification?
I marked it as WONTFIX per comment 48.  I didn't feel the need to repeat myself even though I've had to do that a number of times in this bug.  I think what you really want to focus on is bug 651503.

(and please don't suggest we morph this bug; I filed this bug, and I would not be in favor of morphing it)
Status: REOPENED → RESOLVED
Closed: 13 years ago → 13 years ago
Resolution: --- → WONTFIX
Blocks: 660498
(In reply to comment #52)
> (In reply to comment #50)
> > I believe something like this would be suitable:
> > 
> > Mozilla/5.0 (X11; Linux i686; Gecko/rv:7.0a1) Firefox/7.0a1
> Perhaps, but in a different bug!  I'm super serious about keeping this bug
> scoped to the one thing we said we were going to do, which is to remove the
> frozen date.

Thank you for that, but I do realise that. My comment 50 was a response to comment 49. I didn't reopen the bug precisely because I realised your wish for this to be tightly scoped and because I agree with the resolution. I was clarifying Boris's comment 48 with what he presumably meant and not suggesting morphing this bug.

> (In reply to comment #51)
> > sdwilsh, why did you mark this WONTFIX? And even without any justification?
> I marked it as WONTFIX per comment 48.  I didn't feel the need to repeat
> myself even though I've had to do that a number of times in this bug.  I
> think what you really want to focus on is bug 651503.

Bug 651503 is for a different suggestion if you read through the summary.

Bug 660498 (which I just filed) can be used for what Boris suggested in comment 48 (possibly suggested previously in the mountain of bugs related to the UA string).
:sdwilsh, I think that's a tad *too* narrow, see e.g. bug 660498 comment 1, but fair enough, it's your bug.
VERIFIED
Status: RESOLVED → VERIFIED
This fixed by bug 588909
Status: VERIFIED → RESOLVED
Closed: 13 years ago → 12 years ago
Depends on: 588909
Resolution: WONTFIX → FIXED
Whiteboard: [needs review bz][needs sr jst]
This isn't fixed since bug 588909 was backed out in bug 815743. This should be marked WONTFIX.
Resolution: FIXED → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: