1018724 - Tabzilla lang bar broken

Status: RESOLVED FIXED

(Websites :: Tabzilla, defect)

Description

[Pascal Chevrel:pascalc (PTO until April 26)]

Reported by our Basque localizer, the lang bar is proposing him to switch to Hungarian while he is on the Basque home page. I can also see weird things with my browser in French, if I go to the Basque page, I am proposed to go to the Spanish version of the site (and I don't send Spanish in my  accept-lang headers), if I go to a page not in french I am not being proposed to go to a page in French.

I don't think that the main locale detection on the site is broken, I am correctly redirected to my language using Firefox, Chrome and Curl. I see that in the last days commits there were changes to the lang bar detection logic so I suspect that it may be related to that:

https://github.com/mozilla/bedrock/commit/b57021ea385bf2df31cc7f3a41d650c30b95897d

Caching issue maybe?

Comment 1

[Pascal Chevrel:pascalc (PTO until April 26)]

Another example:
https://www.mozilla.org/oc/

Tabzilla proposes me to visit the Polish version

Comment 3

[Kohei Yoshino [:kohei]]

Hmmm.. @vary_on_headers('Accept-Language') is not working? Let me check.

Comment 4

[Kohei Yoshino [:kohei]]

It's working on local and dev/allizom servers but it's NOT working on production/CDN. I can only see Vary: Accept-Encoding.

pmac, is this something we need an IT bug?

Comment 5

[Josh Mize [:jgmize]]

:kohei I see "Vary: Accept-Language, Accept-Encoding" in the headers testing with curl against production for both http and https, against multiple locales and also without a locale in the url. One example, based on comment #1: 

$ curl -I https://www.mozilla.org/oc/tabzilla/tabzilla.js                                                                                      
HTTP/1.1 200 OK
Server: Apache
Vary: Accept-Language, Accept-Encoding
X-Backend-Server: bedrock1.webapp.scl3.mozilla.com
Cache-Control: max-age=43200
Content-Type: text/javascript
Date: Sun, 01 Jun 2014 20:54:47 GMT
Expires: Mon, 02 Jun 2014 08:54:47 GMT
Transfer-Encoding: chunked
X-Robots-Tag: noodp
Connection: Keep-Alive
X-Frame-Options: DENY
Last-Modified: Thu, 29 May 2014 13:15:29 GMT
X-Cache-Info: caching


I also verified that there weren't any differences across data centers based on the X-Backend-Server header. :kohei, can you provide a specific example?

Comment 6

[Kohei Yoshino [:kohei]]

I run curl several times and here are the results:


$ curl -I https://mozorg.cdn.mozilla.net/oc/tabzilla/tabzilla.js?build=b57021e
HTTP/1.1 200 OK
Content-Encoding: gzip
Accept-Ranges: bytes
Cache-Control: max-age=43200
Content-Type: text/javascript
Date: Sun, 01 Jun 2014 22:01:09 GMT
Expires: Mon, 02 Jun 2014 10:01:09 GMT
Last-Modified: Thu, 29 May 2014 13:15:29 GMT
Server: ECAcc (mdw/12BD)
X-Backend-Server: bedrock3.webapp.phx1.mozilla.com
X-Backend-Server: bedrock2.webapp.scl3.mozilla.com
X-Cache: HIT
X-Cache-Info: not cacheable; response code not cacheable
X-Frame-Options: DENY
X-Robots-Tag: noodp
Content-Length: 9846


$ curl -I https://mozorg.cdn.mozilla.net/oc/tabzilla/tabzilla.js
HTTP/1.1 200 OK
Cache-Control: max-age=43200
Content-Type: text/javascript
Date: Sun, 01 Jun 2014 22:00:43 GMT
Expires: Mon, 02 Jun 2014 10:00:43 GMT
Last-Modified: Thu, 29 May 2014 13:15:29 GMT
Server: Apache
Vary: Accept-Language,Accept-Encoding
X-Backend-Server: bedrock4.webapp.scl3.mozilla.com
X-Backend-Server: bedrock4.webapp.phx1.mozilla.com
X-Cache-Info: caching
X-Cache-Info: cached
X-Frame-Options: DENY
X-Robots-Tag: noodp
Connection: close


$ curl -I https://mozorg.cdn.mozilla.net/oc/tabzilla/tabzilla.js
HTTP/1.1 200 OK
Content-Encoding: gzip
Accept-Ranges: bytes
Cache-Control: max-age=43200
Content-Type: text/javascript
Date: Sun, 01 Jun 2014 22:04:47 GMT
Expires: Mon, 02 Jun 2014 10:04:47 GMT
Last-Modified: Thu, 29 May 2014 13:15:29 GMT
Server: ECAcc (mdw/12BD)
X-Backend-Server: bedrock5.webapp.phx1.mozilla.com
X-Backend-Server: bedrock2.webapp.phx1.mozilla.com
X-Cache: HIT
X-Cache-Info: caching
X-Cache-Info: caching
X-Frame-Options: DENY
X-Robots-Tag: noodp
Content-Length: 9842


$ curl -I https://mozorg.cdn.mozilla.net/oc/tabzilla/tabzilla.js?build=dev
HTTP/1.1 200 OK
Cache-Control: max-age=43200
Content-Type: text/javascript
Date: Sun, 01 Jun 2014 22:07:39 GMT
Expires: Mon, 02 Jun 2014 10:07:39 GMT
Last-Modified: Thu, 29 May 2014 13:15:29 GMT
Server: Apache
Vary: Accept-Language,Accept-Encoding
X-Backend-Server: bedrock3.webapp.phx1.mozilla.com
X-Backend-Server: bedrock1.webapp.phx1.mozilla.com
X-Cache-Info: caching
X-Cache-Info: cached
X-Frame-Options: DENY
X-Robots-Tag: noodp
Connection: close

Comment 7

[Josh Mize [:jgmize]]

Thanks :kohei, I think I see the problem now. The production servers that I was hitting on www.mozilla.org appear to be consistently setting what I believe to be the correct vary headers, but the cdn does not appear to be respecting them. It's not just that you aren't seeing the vary headers on cache hits, but also that if I send a request with a random value for the "Accept-Language" header, I'm still getting cache hits, which does not appear to be correct to me:

$ curl -I -H "Accept-Language: fjlkdsajflkdsa" https://mozorg.cdn.mozilla.net/oc/tabzilla/tabzilla.js
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=43200
Content-Type: text/javascript
Date: Sun, 01 Jun 2014 22:40:11 GMT
Expires: Mon, 02 Jun 2014 10:40:11 GMT
Last-Modified: Thu, 29 May 2014 13:15:29 GMT
Server: ECAcc (ftw/FAA9)
X-Backend-Server: bedrock3.webapp.phx1.mozilla.com
X-Backend-Server: bedrock4.webapp.scl3.mozilla.com
X-Cache: HIT
X-Cache-Info: caching
X-Cache-Info: caching
X-Frame-Options: DENY
X-Robots-Tag: noodp
Content-Length: 29460

:cturra, can you shed any light on this?

Comment 8

[Paul [:pmac] McLanahan]

I was just talking to some people recently about how many CDN providers don't respect or propagate some headers. Hopefully we can make the CDN behave, otherwise we might have to have a separate view on mozorg we can use to check the real accept-lang header.

Comment 9

[Jake Maul [:jakem]]

FYI, X-Cache-Info is from our load balancers, not the CDN (Edgecast). "caching" means this is a new entry, and it's placing it into the cache. "cached" means the hit was served *from* cache. There is also a "not cacheable; <reason>".

X-Hit *is* from the CDNs... so that hitting is very o.O. We'll poke around the CDN config and probably ask Edgecast about it. I confirmed that the origin is sending "Vary: Accept-Language,Accept-Encoding", so that behavior is definitely wrong.

Comment 10

[Jake Maul [:jakem]]

I don't see any config for this. There is a rules engine, but I'm not seeing how to use it to do this. We'll have to open a bug with Edgecast.

Comment 11

[Kohei Yoshino [:kohei]]

Should we use www.mozilla.org instead of mozorg.cdn.mozilla.net while waiting for the fix?

https://github.com/mozilla/bedrock/blob/master/bedrock/base/templates/base-resp.html#L150

Comment 12

[Kohei Yoshino [:kohei]]

Attachment: pull request to workaround the issue

Comment 13

[Chris Turra [:cturra]]

clearing feedback now that :jakem has responded.

Comment 14

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/505ab41ebcc6cbb0e5b9432b56fc16e3eef6dcc3
Bug 1018724 - Tabzilla lang bar broken

A workaround for the wrong language detection due to caches, until the
header issue is fixed by CDN.

https://github.com/mozilla/bedrock/commit/0056d87baba4177d8b01aba2319ddd12f780297a
Merge pull request #2069 from kyoshino/bug-1018724-wrong-tabzilla-lang

Bug 1018724 - Tabzilla lang bar broken

Comment 15

[Kohei Yoshino [:kohei]]

I can now see Vary: Accept-Language when I do curl -I https://mozorg.cdn.mozilla.net/{LANG}/tabzilla/tabzilla.js. Has Edgecast fixed the issue?

Comment 16

[Chris More [:cmore]]

I have two people that are having the false positive language mismatch errors constantly and it happens with various different languages. Their browsers are set to en-US and they are visiting an en-US page. I had them try to to start a private browsing session and it still happened, but now with a different language prompt. This could be affecting conversion rate of our download page since anything that confuses the user could affect downloads.

Comment 17

[Kohei Yoshino [:kohei]]

Attachment: pull request to prevent tabzilla.js from being cached by proxies

I guess it's due to proxy servers in companies. I sent a PR to fix it...

Comment 18

[Kohei Yoshino [:kohei]]

(In reply to Kohei Yoshino [:kohei] from comment #15)
> Has Edgecast fixed the issue?

Hmm, no, it's still broken :(

Comment 19

[Paul [:pmac] McLanahan]

Seems like some of the issue is that we've made all of tabzilla a bit less cacheable by including the value of Accept-Language in it. What if we had a separate AJAX view that returned the value of some select headers (or just accept-language) that we could set `@never_cache` on for use when the langbar is enabled? I have a feeling such a facility will come in handy in the future as well.

Comment 20

[Paul [:pmac] McLanahan]

Gareth, Josh, and I met briefly and decided that the best course is as follows:

1. Temporarily disable the transbar and re-enable the CDN for Tabzilla.
2. Implement the simple header AJAX view solution I mentioned in comment #19 then re-enable.

I'll have a PR for step 1 submitted shortly.

Comment 21

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/98a6a96a742aec56ae166f9b3bafba409a5bf322
Bug 1018724: Temporarily disable translation bar.

We'll figure out the cache issues and enable again very soon.

https://github.com/mozilla/bedrock/commit/700f8ecd6af3b715bb0fb8628002ee290b1d7a2f
Merge pull request #2087 from pmclanahan/temporarily-disable-transbar-1018724

Bug 1018724: Temporarily disable translation bar.

Comment 22

[Kohei Yoshino [:kohei]]

Attachment: pull request to use async call to detect user language

Thank you :pmac, :garethc, :jgmize for your help. And sorry to trouble you.

By the way, one of the reason why {{ accept_languages }} was embedded in tabzilla.js was making it easy for other scripts to use the detected languages, including optimize.ly (see Bug 975020). In this PR I have introduced a language detection queue for them and mentioned it in the Tabzilla document.

Comment 23

[Jake Maul [:jakem]]

(In reply to Kohei Yoshino [:kohei] from comment #15)
> I can now see Vary: Accept-Language when I do curl -I
> https://mozorg.cdn.mozilla.net/{LANG}/tabzilla/tabzilla.js. Has Edgecast
> fixed the issue?

I don't see this... I see a "Vary: Accept-Encoding", but no Accept-Language. As far as I know they have not changed anything. This fell off my radar last week, and I only just now opened the ticket with them. Sorry. :(

Leaving my NEEDINFO so I can find this when I get a response from them.

Comment 24

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/3cc4fdb5973e7819f84b5f9c0e01e6d4831db141
Use async call to detect user language in Tabzilla, Bug 1018724

https://github.com/mozilla/bedrock/commit/ca88cb54e93321c7f51ebbd81a0c7e199d185309
Merge pull request #2088 from kyoshino/bug-1018724-tabzilla-userlang

Use async call to detect user language in Tabzilla, Bug 1018724

Comment 25

[Ibai Oihanguren]

I have seen that now it works as it should (at least for Basque).
I've visited Spanish and English versions of the web, and I have been asked if I wanted to view the page in Basque.
When I visit the Basque page, no message is shown.

Comment 26

[Kohei Yoshino [:kohei]]

Ibai: Thanks for verification! Closing then.

Comment 27

[Jake Maul [:jakem]]

Clearing my needinfo... we can still do this, but it was going to result in a bit of a fee to get it set up. If the current solution gets the job done, then I'm happy to leave it alone.