651674 - Old style Google homepage / search results shown, due to sniffing for the now removed date in the user-agent (also sniffing for Firefox instead of Gecko)

Status: RESOLVED FIXED

(Tech Evangelism Graveyard :: English Other, defect)

Description

[Daniel C]

Bug 651503 is looking at the possibility of replacing the frozen date in the user-agent with the Gecko version.

For example, going from:

Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0

to:

Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/6.0 Firefox/6.0

That way, there's one less piece of meaningless information being sent out with each HTTP request.

Gecko/ is invalid, Gecko/rv:6.0 is invalid and Gecko on it's own will apparently break some sites. Gecko/6.0 seemed like a good solution, so I made that change in about:config using general.useragent.override expecting nothing to break. Very quickly (it's my homepage) I found a page that broke: the most visited page on the web, Google!

It served up an odd looking page, with the old GIF logo which was broken up into 3 separate images. Searching from the page didn't work either, no results appeared.

This was with http://www.google.co.uk/ but the same is true with http://www.google.com/, except the logo is the newer PNG one. The site still looks, and is, broken.

Clearly, it thought I was using a completely different browser. I played around with the override pref to see what browser it thought I was using. Nothing matched the output that I saw (I'll attach screenshots shortly), but there are loads of different versions of the homepage for different browsers (so their sniffing logic is obviously quite complex).

What appears to happen, is that it simultaneously thinks I'm using a browser that is newer than Firefox 2 and yet older than Netscape 7!

User-agents to try:

Gecko/20020823 Firefox/3.0
Gecko/20020823 Firefox/2.0
Gecko/20020822 Firefox/2.0
Gecko/20020822 Firefox/3.0

The first one displays exactly the same as the standard Firefox 4 release.
The second one displays with a smaller logo, and Google Instant is disabled.
The third one displays with the old GIF logo made from multiple images.
The fourth one is the broken one, with the old GIF logo, but with Google Instant enabled (but not working).

I doubt there's a browser that can't handle the new logo, but can handle Google Instant, so I think they have some of their checks in the wrong scope.

20020823 is the build date of Netscape 7 (https://developer.mozilla.org/en/Gecko_User_Agent_Strings) which was released 6 days later and build on Gecko 1.0.1.

Comment 1

[Daniel C]

Attachment: Homepage 1

Comment 2

[Daniel C]

Attachment: Homepage 2

Comment 3

[Daniel C]

Attachment: Homepage 3

Comment 4

[Daniel C]

Attachment: Homepage 4

Comment 5

[Daniel C]

Attachment: Homepage 5

Comment 6

[Daniel C]

Attachment: Homepage 6

Comment 7

[Daniel C]

So, those are screenshots of the different homepages that get sent for different combinations of Gecko and Firefox strings in the user-agent header. Here is a list of which when you get when you send the following user-agents:

Foo -> 1
Firefox -> 1
Firefox/ -> 2
Firefox/2.0 -> 2
Firefox/3.0 -> 3
Firefox/000000003.0 -> 3
Gecko -> 1
Gecko/ -> 1
Gecko/20080222 -> 4
Gecko/20080223 -> 1
Gecko/0000000020080222 -> 4
Gecko/0000000020080223 -> 1
Gecko/20080222 Firefox/2.0 -> 5
Gecko/20080223 Firefox/3.0 -> 3
Gecko/20080222 Firefox/3.0 -> 6

Rule 1: Firefox/ gets you some extra padding around the top menu.
  Rule 1a: Firefox/x where x < 3 is the same.
  Rule 1b: Firefox/x where x >= 3 gets you Google Instant as well. This works with leading 0s (suggesting that they've got a regexp with something like Firefox\/(\d)* in it.

Rule 2: Gecko followed by a slash, follow by any number that is less than 20080223 (twenty million, eighty thousand, two hundred and twenty-three) will get you an old school GIF logo made up of separate images, old-style buttons and a native-style textbox (which I actually prefer to the custom-themed ones, but hey). This is also true with leading 0s, so something like Gecko\/(\d)+ (at least one digit required this time).

Combining rules 1a & 2 works OK. Combining rules 1b & 2 doesn't.

Conclusion 1: Google is sniffing for Firefox/ to determine whether to send some padding tweaks. It should instead (if it has to) sniff for Gecko or Gecko\/\d+ if you want actual Gecko and not pretenders (or rv:[\d\.]+).

Conclusion 2: Google is sniffing for Firefox >= 3 to determine whether to enable Google Instant. It should instead (if it has to) sniff for a Gecko version >= 1.9.

Conclusion 3: Google is sniffing for Gecko with a build date earlier than 20020823 (Netscape 7) to determine whether to send the new PNG logo or the old broken-up GIF one. It should instead (if it has to) sniff for a Gecko version < 1.0.1. I'm not actually sure what they're looking for here as I thought PNG support was solid well before that. Perhaps it's the styling of the form controls. Perhaps I'll try Netscape 6.2.3 to see if I can work out why...

Comment 8

[Daniel C]

Kev, I believe you're the person with the contacts. This doesn't actually
affect anything currently, but blocks the possibility of replacing the
meaningless date in the user-agent string (20100101) with the Gecko version.
Regardless of that, the way Google are sniffing the user-agent string is bad as
they're singling out Firefox for its capabilities, rather than Gecko. This
means that other applications based on Gecko have to carry around the Firefox
product in their user-agent strings which isn't good for the web and prolongs
the life of bug geckoisgecko :-(

Comment 9

[Daniel C]

That should be bug 334967. I thought Bugzilla worked with aliases like that...

Note that although this was tested with google.co.uk, the same is true of google.com

Comment 10

[Daniel C]

I actually ran Netscape 7 and Netscape 6.2.3 with a spoofed UA to see what the difference was. 6.2.3 (Gecko 0.9.4.1) doesn't cope well with having the buttons styled. Fair enough, I thought, but Google is already sniffing for Netscape6 and in fact Netscape/x where x < 8 and disabling the styling anyway.

A UA of Netscape6 or Netscape/7 will get you Homepage 5 regardless of the date.

So perhaps there's a difference between Mozilla 1.0 and 1.0.1 (which fall on either side of 20020823)? Mozilla 1.0 messes up the buttons when spoofed as 1.0.1, but then 1.0.1 messes them up just the same without any spoofing. I therefore see no reason at all to serve different content based on the date. 

Any code that checks for Firefox versions (or Netscape versions, Mozilla versions etc.) should be changed to look at the rv: string.

Comment 11

[Henri Sivonen (:hsivonen)]

I sent a message to a venue where Google engineers can see it.

Comment 12

[Daniel C]

(In reply to comment #11)
> I sent a message to a venue where Google engineers can see it.

Thanks! I'll attach screenshots now of what I saw with the legacy browsers.

Comment 13

[Daniel C]

Attachment: Netscape 6.2.3 (Gecko 0.9.4.1)

This is what Netscape 6.2.3 (Gecko 0.9.4.1) looks like when spoofed to look like Mozilla 1.0.1 (spoofing to look like Netscape 7 wasn't enough as just Netscape/7 is enough to block the styling and new logo from being used).

Comment 14

[Daniel C]

Attachment: Mozilla 1.0 (Gecko 1.0)

This is what Mozilla 1.0 (Gecko 1.0) looked like when spoofed to look like Mozilla 1.0.1.

Comment 15

[Daniel C]

Attachment: Mozilla 1.0.1 / Netscape 7 (Gecko 1.0.1)

This is what Mozilla 1.0.1 looks like without any spoofing (Netscape 7 was the same). It's slightly different to 1.0 but it's still a bit broken, so why switching on the date 20020823 makes any sense I'm not sure.

Comment 16

[Daniel C]

(In reply to comment #11)
> I sent a message to a venue where Google engineers can see it.

Henri, where is this venue? What's the link so we can track the progress?

Comment 17

[Marek Stępień [:marcoos, inactive]]

> So perhaps there's a difference between Mozilla 1.0 and 1.0.1 (which fall on
> either side of 20020823)?

FWIW, 1.0.1 introduced the "Almost Standards Mode", which works like standards mode, but with quirks mode stuff happening to images inside tables.
https://developer.mozilla.org/en/Gecko%27s_%22Almost_Standards%22_Mode

Comment 18

[Gordon P. Hemsley [:GPHemsley]]

Google announced today that they've redesigned the homepage and search results page.[1]

I don't actually see any changes yet, so they may not be everywhere yet, but when they roll out for you, you might want to recheck to see if they fixed any of this.

[1] http://googleblog.blogspot.com/2011/06/evolving-google-design-and-experience.html

Comment 19

[Daniel C]

(In reply to comment #18)
> Google announced today that they've redesigned the homepage and search
> results page.[1]
> 
> I don't actually see any changes yet, so they may not be everywhere yet, but
> when they roll out for you, you might want to recheck to see if they fixed
> any of this.
> 
> [1]
> http://googleblog.blogspot.com/2011/06/evolving-google-design-and-experience.
> html

The problem still exists.

Comment 20

[(no longer active)]

Henri, did you ever hear back from Google about this?

Comment 21

[Henri Sivonen (:hsivonen)]

(In reply to Ehsan Akhgari [:ehsan] from comment #20)
> Henri, did you ever hear back from Google about this?

Yes. They wanted to know what we are changing to. The communication faded out when we didn't have a definitive target string plan. (Which is, of course, a circular dependency.)

Comment 22

[Chris Lawson (gone)]

Can we poke Google again about this, and get them to fix things for what our strings are *now*, and worry about what they might change to later?

Comment 23

[Daniel C]

Fennec now has Gecko/version and Firefox is looking to match that (see 588909 comment 24).

If that patch gets reviewed and landed then that will presumably give Google more reason to change (looking at comment 21). If everyone could pull any Google strings that they have at that point then hopefully this can be fixed as soon as possible.

Comment 24

[Ed Morley [:emorley]]

Bug 588909 has now landed, causing this to become an issue with today's Nightly onwards (ie https://www.google.com/search?q=foo displays the version shown for older browsers).

Comment 28

[Shane Bundy]

I see the page works and looks fine when spoofing the UA. Has Google been made aware of the issue?

Comment 29

[Ed Morley [:emorley]]

(CC'ing the Google contact in another recent Google tech evang bug)

Mike, if you're not the right person to CC please can you pass this on to whichever team needs to know - thank you :-)

Comment 30

[Boris Zbarsky [:bzbarsky]]

I also mailed my Google contacts.

But it's not clear to me whether we can actually ship the UA string in the shape it's currently in.  :(

Comment 33

[klonos]

...it doesn't only happen for .co.uk

Comment 34

[klonos]

...this is a firefox bug.

Comment 35

[Gordon P. Hemsley [:GPHemsley]]

(In reply to klonos from comment #34)
> ...this is a firefox bug.

The change made to Firefox that caused this bug was done intentionally, in bug 588909. Solving this problem involves tech evangelism with Google, not a change to Firefox.

Comment 36

[Henri Sivonen (:hsivonen)]

Assigning to bz per comment 30.

Comment 40

[Henri Sivonen (:hsivonen)]

I re-pinged the Google folks I discussed this with in April.

Comment 41

[Matthias Versen [:Matti]]

The right way IMHO would be to backout the UA change, get google and zimbra fixed and reapply the patch.
But who cares about the wasted time from our reporters and triagers for duping and later for asking which build ID is used :-(

Comment 42

[Gervase Markham [:gerv]]

(In reply to Matthias Versen (Matti) from comment #41)
> The right way IMHO would be to backout the UA change, get google and zimbra
> fixed and reapply the patch.

We tried that already (at least for Google). Having actual downloadable browsers with a new behaviour will, IMO, make it much more likely that sites will adapt to it. Saying "sometime in the future we are going to make this change" will lead to the response that "well, sometime in the future I'll fix my site - but I have more important things to do right now".

Gerv

Comment 43

[Henri Sivonen (:hsivonen)]

(In reply to Matthias Versen (Matti) from comment #41)
> The right way IMHO would be to backout the UA change, get google and zimbra
> fixed and reapply the patch.

Removing the "a1" part to work around Zimbra should be about as fast as backing out.

I expect Google Search not to get its sniffing changed if we don't have real code on a train headed to release to show.

Asking Google to change their sniffing to anticipate something that we might maybe do sometime is not a recipe for success. Note that Googlers got a heads-up about this in April 2011, but at that time, "what exactly" and "when" were indefinite.

Comment 45

[David Chalkley]

It quite odd that Google for me has been working fine up until only last week when my Nightly (Firefox 13a1) automatically updated to the lasted build and you lot have been experiencing problems since 2011. 
Three things appear stage with my Google (google.com.au): 
1. Double white lines though the Google logo on the homepage (Google.com the logo is located in the middle of the page while Google.com.au has the logo at the top).
2. Double search bars appear on the results page.
3. The search help bar doesn't appear on the left while on the results page.
There is an image included on my bug report 729070 (comment before) on the problem I am facing, but Google works fine on my internet explorer 9 (x64 and x32) and works fine on Firefox 9 (and explorer) on another computer in my house.

Comment 46

[Henri Sivonen (:hsivonen)]

(In reply to David Chalkley from comment #45)
> It quite odd that Google for me has been working fine up until only last
> week when my Nightly (Firefox 13a1) automatically updated to the lasted
> build and you lot have been experiencing problems since 2011. 

We started experimenting with a new kind of Gecko/ token in the UA string in 2011 but the change actually landed in Nightly only recently.

Comment 49

[Daniel]

I've had the exact same experience as described in comment 45, only for google.se. Spoofing the UA-string fixes the problem.

Comment 50

[Marek Stępień [:marcoos, inactive]]

Attachment: Homepage 7

I'm getting slightly different results - it looks like the old Google home page (plus broken logo and misplaced submit buttons), but I do see the new toolbars.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:13.0) Gecko/13.0 Firefox/13.0a1

Comment 51

[David Chalkley]

Thank-you 
Google is now being displayed properly with the latest build:
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:13.0) Gecko/20120221 Firefox/13.0a1 ID:20120221031301 (22/02/2012 - Australian) 
Again thanks for the quick response/patch/fix/update.

Comment 52

[Ben Bucksch (:BenB)]

Henri, if you could activate your Google contact now, that would go a long way to get this UA string settled.

> I expect Google Search not to get its sniffing changed if we don't have real code on a
> train headed to release to show.
> Asking Google to change their sniffing to anticipate something that we might maybe
> do sometime is not a recipe for success.

As you very pointedly said in comment 21, we're at a catch-22. It wasn't just anticipation now. We already did the change, but it was backed out *because* Google didn't change yet. Maybe if you make that dependency clear to them, they can understand. We can't act until they do. There are ways to parse that work both with the old and new string and other browsers, right?

Comment 55

[Mike Graboski]

Sorry for the late reply - I was out of the office since the middle of last week.  

The Google search team is aware of this issue.  It looks like the user agent issue has been reverted in the nightly builds of Firefox.  As soon as this user agent change becomes stable, the Google search team will update its browser rules.

Just to set expectations, it may take 2-4 weeks after this change is stable in Firefox for the browser rules to be updated on our end.

Comment 56

[Dão Gottwald [:dao]]

(In reply to Mike Graboski from comment #55)
> The Google search team is aware of this issue.  It looks like the user agent
> issue has been reverted in the nightly builds of Firefox.  As soon as this
> user agent change becomes stable, the Google search team will update its
> browser rules.

See comment 53 -- we'd prefer Google to be fixed before we move forward with this.
Note that the change is still live in Firefox builds for Android:
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android/
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/

> Just to set expectations, it may take 2-4 weeks after this change is stable in
> Firefox for the browser rules to be updated on our end.

That's fine.

Comment 57

[Mike Graboski]

The Google search team can update its browser rules before the user agent change becomes stable.  However, it seems as if there's some back-and-forth on this thread as to what the final user agent should be.  

Dão, can you point me to the final version of the user agent string that Google should use?

Comment 58

[Gervase Markham [:gerv]]

Hi Mike,

The dust is still settling here; we are working out both who makes the decisions and how, and where we are trying to get to. It's great to have a contact within Google who we can let know about these things. 

Give me a little while to work this out and we'll let you know. Thank you for your patience :-) (A 4 week turnaround is absolutely fine - for any change, the Firefox process will take between 13 and 18 weeks to get from change to release.)

Gerv

Comment 59

[Dão Gottwald [:dao]]

I didn't manage to find the script doing this, but apparently it regards the version segment following Gecko/ as an old date even if it's shorter than 8 digits or contains non-digit characters. I'd suggest attacking it from that angle. It's only a date if it's at least 8 characters long and contains only digits. Should be easy to be forward-compatible here with any change we may make.

Comment 60

[Lukas Blakk [:lsblakk] use ?needinfo]

[Triage Comment]

no need to track since bug 588909 has been backed out

Comment 61

[Daniel C]

(In reply to Dão Gottwald [:dao] from comment #59)
> I didn't manage to find the script doing this, but apparently it regards the
> version segment following Gecko/ as an old date even if it's shorter than 8
> digits or contains non-digit characters. I'd suggest attacking it from that
> angle. It's only a date if it's at least 8 characters long and contains only
> digits. Should be easy to be forward-compatible here with any change we may
> make.

Dão, this is done server-side (at least most of it is). Disabling JavaScript and playing with general.useragent.override will get you different home pages and results pages. As an update to comment 7, I tested the following user agents:

 1. <blank>
 2. Foo
 3. Firefox
 4. Firefox/
 5. Firefox/2
 6. Firefox/3
 7. Firefox/000000003
 8. Gecko
 9. Gecko/
10. Gecko/1
11. Gecko/20020822
12. Gecko/20020823
13. Gecko/0000000020020822
14. Gecko/0000000020020823
15. Gecko/20020822 Firefox/2
16. Gecko/20020822 Firefox/3
17. Gecko/20020823 Firefox/2
18. Gecko/20020824 Firefox/3
19. Gecko/20100101 Firefox/13.0
20. Gecko/13.0 Firefox/13.0

This leaves with pretty much the same conclusions as before.

1. Google sniffs for "Firefox/(\d)*" and if it finds it, adds some padding to the top nav bar. If (\d)* is >= 3 then it shows the new black nav bar and a sign-in button instead of a sign-in link, as well as enabling Google Instant.

2. Google sniffs for "Gecko/(\d)+". If it finds it and (\d)+ >=1 but < 20020823 then the button styling changes a bit. If it finds it and (\d)+ is <1 or (\d)+ is >= 20020823 then the button styling remains the same as if it hadn't found the string.

Therefore, when we have:

Gecko/13.0 Firefox/13.0

because 13.0 < 20020823 in the Gecko token and 13.0 >= 3 in the Firefox token, the browser is treated as simultaneously being Firefox newer than or equal to 3 and Netscape older than Netscape 7 (Netscape/7.0 had rv:1.0.1 and Gecko/20020823).

My suggestion is that the code for detecting a Gecko date < 20020823 only apply when the Gecko version string is 8 characters long or more: change Gecko/(\d)+ to Gecko/(\d){8,}

This would allow for Gecko/13.0 to work.


This change can be applied before the Gecko string is actually finalised as it shouldn't break anything (there have never been any Gecko date strings less than 8 characters as far as I know).

Alternatively, if possible, rip out the code for detecting a Gecko date entirely. Does even the most visited page on the web get hits from Netscape < 7 (other than the hundred or so that I've just done)? If not then this could simplify your code and at the same time allow Gecko to make its planned change to Gecko/13.0.

Thanks.

Comment 63

[Dão Gottwald [:dao]]

This appears to be fixed now. I'm testing with:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/14.0 Firefox/14.0

Mike, can you confirm that this has been taken care of on your side?

Comment 64

[Mike Graboski]

Dão, are you not seeing the old style Google homepage or search results anymore?  Also, can you confirm that this is the final format of the user agent that Firefox will use?  We were actually waiting for the comments in this thread to settle on the final, authoritative user agent string format for us to use.

Thanks,
Mike

Comment 65

[Dão Gottwald [:dao]]

(In reply to Mike Graboski from comment #64)
> Dão, are you not seeing the old style Google homepage or search results
> anymore?

Yes, I'm seeing the modern home and search result pages.

> Also, can you confirm that this is the final format of the user
> agent that Firefox will use?

I'll probably never be able to make such a strong statement before we've actually released a Firefox version with that user agent string. We could always discover that some widely deployed intranet application is broken, requiring us to revert the change at the last minute. That said, the plan is to land the patch that will make the user agent string look exactly like the one I just posted.

Comment 66

[Daniel C]

I also cannot reproduce this anymore using the latest nightly and this UA:

Mozilla/5.0 (X11; Linux x86_64; rv:14.0) Gecko/14.0 Firefox/14.0

I get the modern homepage and results page and there appears to be no difference between using that UA and the Fx 11 UA.

This seems to be fixed. I'll comment in bug 728972 to see if anyone that was hitting that error can reproduce it with the planned UA change.

Comment 67

[Henri Sivonen (:hsivonen)]

Marking FIXED rather than WFM, because there was actual communication with Googlers leading to this result.