975020 - Translation Bar and non-locale redirects don't work as expected in some cases; use the user's accept language list to fix it

Status: RESOLVED FIXED

(Websites :: Tabzilla, enhancement)

Description

[Kohei Yoshino [:kohei]]

As tested by :cmore and :agibson, navigator.language is not reliable on some non-Firefox browsers where the property just shows the application's language. We need to expose the user's HTTP Accept-Language header on template to

1. utilize the info for optimize.ly
2. improve the Translation Bar on Tabzilla

Comment 1

[Kohei Yoshino [:kohei]]

Attachment: pull request

Comment 2

[Alex Gibson [:agibson]]

There has been some discussion in the PR and we have two choices for moving forward with this change:

1.) We use only the first (preferred) language in 'Accept Language' and show the translation bar only if this first language is matched. For example, if my languages are ['fj', 'fr', 'en-US'] and I visit to '/de' page, the trans bar would not show since 'fj' is not available. (This matches is the current behavior for the translation bar, since navigator.language is only ever a single value).

2.) We use the 'best match' language in 'Accept Language' and show the translation bar if one of these languages is matched. For example, if my languages are ['fj', 'fr', 'en-US'] and I visit to '/de' page, the trans bar would offer me to view the page in 'fr', since 'fj' is not available.

Personally, I'm edging toward 2.) as it matches more closely toward how I would expect a preferred language list to behave. But I would like to get other people's opinions here also so that we are all on the same boat, so to speak.

Any strong opinions here, Chris?

Comment 3

[Chris More [:cmore]]

(In reply to Alex Gibson [:agibson] from comment #2)
> There has been some discussion in the PR and we have two choices for moving
> forward with this change:
> 
> 1.) We use only the first (preferred) language in 'Accept Language' and show
> the translation bar only if this first language is matched. For example, if
> my languages are ['fj', 'fr', 'en-US'] and I visit to '/de' page, the trans
> bar would not show since 'fj' is not available. (This matches is the current
> behavior for the translation bar, since navigator.language is only ever a
> single value).
> 
> 2.) We use the 'best match' language in 'Accept Language' and show the
> translation bar if one of these languages is matched. For example, if my
> languages are ['fj', 'fr', 'en-US'] and I visit to '/de' page, the trans bar
> would offer me to view the page in 'fr', since 'fj' is not available.
> 
> Personally, I'm edging toward 2.) as it matches more closely toward how I
> would expect a preferred language list to behave. But I would like to get
> other people's opinions here also so that we are all on the same boat, so to
> speak.
> 
> Any strong opinions here, Chris?

I'm leaning toward case #2 too. One thing I thought was working on the website, but it not is the following scenario.

1) Set your browser to en-GB.
2) Visit https://www.mozilla.org/firefox/new/
3) You get redirected to /en-US/firefox/new/ since en-GB is not a bedrock lang.
4) See the mismatch prompt up top
5) Click yes on the prompt.
6) Get redirected to /en-GB/firefox/new/ and then back to /en-US/firefox/new/
7) See the mismatch prompt again and click yes
8) Repeat steps 3-7 forever

We shouldn't be prompting a lang mismatch if it is not an available lang on bedrock. It seems like case #2 would handle this use-case better.

Comment 4

[Chris More [:cmore]]

If we need to do a separate bug for my comment 3 scenario, let's do it. This issue made a recent A/B test hypothesis look false until we realized these are not really mismatches. After removing these false positives, the hypothesis was true. During the analysis, the en-GB/en-US false mismatches are converting at a much lower rate than other false positive mismatches.

Comment 5

[Kohei Yoshino [:kohei]]

With regard to the en-GB issue, looks like the Translation Bar itself does the right thing. There is <link rel="alternate" hreflang="en-GB" href="http://www.mozilla.org/en-GB/firefox/new/" title="English (British)"> as well as <option lang="en-GB" value="en-GB">English (British)</option>. 

I'll file a new bug to deactivate the en-GB locale on /firefox/new/:
http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/locales/en-GB/firefox/new.lang?view=markup#l1

Comment 6

[Kohei Yoshino [:kohei]]

Oh, the en-GB should be activated on /firefox/new/ so visitors from UK can download the en-GB version of Firefox. We have to investigate why https://www.mozilla.org/en-GB/firefox/new/ is redirected to https://www.mozilla.org/en-US/firefox/new/. It's not a bug in the Translation Bar code anyway.

Comment 7

[Chris More [:cmore]]

(In reply to Kohei Yoshino [:kohei] from comment #6)
> Oh, the en-GB should be activated on /firefox/new/ so visitors from UK can
> download the en-GB version of Firefox. We have to investigate why
> https://www.mozilla.org/en-GB/firefox/new/ is redirected to
> https://www.mozilla.org/en-US/firefox/new/. It's not a bug in the
> Translation Bar code anyway.

Good catch! I checked /firefox/all/ and Firefox is in en-GB. 

I don't see a difference between the en-US strings vs en-GB here: http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/locales/en-GB/firefox/new.lang?view=markup#l1

Pascal: Do you know why we redirect en-GB to en-US on Bedrock when we have en-GB strings and when we do have an en-GB download? Why have a browser build available that can't be downloaded on the primary Firefox download page?

It looks like it is done via bug 672498 and we may want to look to revert that redirect.

Comment 8

[Kohei Yoshino [:kohei]]

Filed Bug 980212.

Comment 9

[Kohei Yoshino [:kohei]]

And copied :cmore's comment 7 to Bug 980212 :)

Comment 10

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Chris More [:cmore] from comment #7)
> (In reply to Kohei Yoshino [:kohei] from comment #6)
> > Oh, the en-GB should be activated on /firefox/new/ so visitors from UK can
> > download the en-GB version of Firefox. We have to investigate why
> > https://www.mozilla.org/en-GB/firefox/new/ is redirected to
> > https://www.mozilla.org/en-US/firefox/new/. It's not a bug in the
> > Translation Bar code anyway.
> 
> Good catch! I checked /firefox/all/ and Firefox is in en-GB. 
> 
> I don't see a difference between the en-US strings vs en-GB here:
> http://viewvc.svn.mozilla.org/vc/projects/mozilla.com/trunk/locales/en-GB/
> firefox/new.lang?view=markup#l1
> 
> Pascal: Do you know why we redirect en-GB to en-US on Bedrock when we have
> en-GB strings and when we do have an en-GB download? Why have a browser
> build available that can't be downloaded on the primary Firefox download
> page?

1 yes,  we should have an en-GB download page back
2 Bedrock has changed the behaviour of the en-US download button which no longer does client-side detection and thus no longer provides an en-GB build for en-GB visitors of an en-US download page.

> 
> It looks like it is done via bug 672498 and we may want to look to revert
> that redirect.

No, this bug redirected specific unmaintained pages but not the download pages and that was pre-bedrock on the php site. the bug is more likely to be in this line on Bedrock:
https://github.com/mozilla/bedrock/blob/master/etc/httpd/global.conf#L212

en-GB and en-ZA are not included in the redirect for Firefox download page, I can make a PR for that shortly.

Comment 11

[Pascal Chevrel:pascalc (PTO until April 26)]

I made a pull request in bug 980212 to have the download pages going through bedrock and avoid the redirect to en-US.

Comment 12

[Kohei Yoshino [:kohei]]

So the en-GB issue has been fixed in Bug 980212. I have found another issue. Here's my comment on GitHub:

> By the way I was thinking a different case. For example, if I have
> ['ko', 'fr', 'en-US'] and visit http://www.mozilla.org/firefox/os/,
> I'm redirected to en-US instead of fr because get_best_language()
> returns ko but that page doesn't have the ko translation. I should
> be redirected to fr here because it's the second preferred language.

> The same holds for the Translation Bar and that's why we might have
> to have an array instead of a single value.

The redirector is considering the user's primary accept language only and that's the problem here. Needs a check for the secondary, tertiary... accept languages also, and transbar should behave in the same way.

Changing the summary again...

Comment 13

[Kohei Yoshino [:kohei]]

Attachment: pull request

Comment 14

[Kohei Yoshino [:kohei]]

Let's use navigator.languages later.

Comment 15

[Zibi Braniecki [:zbraniecki][:gandalf]]

you may also be interested in language negotiation code that we use in l20n, which is based on ECMA 402 internal LookupAvailableLocales variant - https://github.com/l20n/l20n.js/blob/master/lib/l20n/intl.js#L434

We submitted the method for standardization in the second edition of ECMA 402 - http://wiki.ecmascript.org/doku.php?id=globalization:prioritizelocales

Comment 16

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/b0b6f32175588a645045e2e208c5e0518d45a3f7
Fix Bug 975020 - Translation Bar and non-locale redirects don't work as expected in some cases; use the user's accept language list to fix it

https://github.com/mozilla/bedrock/commit/b57021ea385bf2df31cc7f3a41d650c30b95897d
Merge pull request #1771 from kyoshino/bug-975020-user-lang

Fix Bug 975020 - Translation Bar and non-locale redirects don't work as expected in some cases; use the user's accept language list to fix it