160370 - if japanese is added to languages, it takes precedence

Status: RESOLVED INVALID

(Core :: Internationalization, defect)

Description

[dennis bloodnok]

set my navigator|languages preferences as follows: english (en-us), english
(en-gb), japanese (ja). when surfing to a site hosted by an apache webserver
running multiview, the browser defaults to the japanese version, regardless. 

this problem was originally noticed several builds ago (0.9.8 perhaps?) and was
discovered trying to access http://httpd.apache.org/docs-2.0/. (the example url
above has its own problems - namely the wrong encoding.) the behaviour is
consistent on linux and windoze builds. it remains consistent regardless what
order the language preferences are set. 

this may be an apache bug, but unlikely. workaround is to remove language
preference. mozilla will then present japanese pages only if that's the only
option available, which is sufficient. but it sorta makes one wonder what the
purpose of the languages preference is...

Comment 1

[Boris Zbarsky [:bzbarsky]]

Could you please do the following?

1)  Change the preferences so that this problem is reproduced
2)  Quit mozilla
3)  export NSPR_LOG_MODULES=nsHttp:5
4)  export NSPR_LOG_FILE=/tmp/httplog.txt
5)  mozilla http://httpd.apache.org/docs-2.0/ (to reproduce the problem)
6)  Quit mozilla
7)  attach the log file to this bug

Comment 2

[Yuying Long]

Attachment: httplog file

Steps to reproduce:
1. Load page: http://httpd.appache.org/docs-2.0 with a new profile - the
default accept languages are: English[en_us] and English[en], the page is
displayed as English.
2. Add Japanese at the end of accept languages list, and load the page again,
the page still display in English.
3. Switch the order of accept languages order to [en-us], [ja], [en], then load
the page, the page is displayed in Japanese.
4. Change accept languages order to original: [en-us], [en], [ja], load the
page again, page is in English (however, the font face is slightly
differently).

Note: this log file only for step 3 because of the file size too big.

Comment 3

[Yuying Long]

Confirming this on netscape 07-31 branch build, this not only on linux client
but for all platforms.

Comment 4

[Boris Zbarsky [:bzbarsky]]

From that log:

Accept-Language: en-us, ja;q=0.66, en;q=0.33

is what we send.  That means "Give me 'en' with a weight of 0.33, 'ja' with a
weight of 0.66 or 'en-us' with a weight of 1", where higher weights mean we
prefer that language.  This header interacts with the server's internal settings
to determine the language that is actually returned.

In this case, it looks like the server does not recognize the en-us (or en-gb)
language and se has the choice of 'ja' or 'en', with 'ja' having higher weight...

So should we always make sure that 'en' and 'en-*' are next to each other in the
list or something?  Because the networking side of things looks correct here....

Comment 5

[Yuying Long]

One more thing which might be from server, but I just wondering:
if the accept languages has two: [en], [ja], then when I load the page, will
give me an English contains display page but with charset iso-2022-jp (the ascii
characters in ja font)?

Comment 6

[dennis bloodnok]

a friend (unemployed q/a person) tested this on mac osx and got the same
results. however, being a q/a sort, he also tested with en defined in the list.
in that case, he got the results expected. i tried the same and also got correct
behaviour on windows. however, the case of en-us, en-gb, etc. remains
counter-intuitive.

Comment 7

[Roy Yokoyama]

i'll accept for now; but I need to know what is the best course of action for this.
It looks like we are sending the right 'Accept-Language'; but the Apache
server doesn't recognize 'en-us'.

cc momoi-san for his input.

Comment 8

[dennis bloodnok]

discovered yet another quirk with this setup. note i've currently got en (no
en-gb or en-us) and ja defined. this may be a google specific issue: i went to
register with google's automatic url exclusion system,
http://services.google.com:8882/urlconsole/controller?cmd=reload&lastcmd=login
with the above languages defined. google served a japanese page. only by
removing ja from my language definitions could i get it to serve an english
page. this was the case on both windows (1.5) and linux (1.6b).

as to the en-gb/en-us issue, if apache is at fault here then their developers
need to be informed of it. that's my 2 cents anyway.

Comment 9

[Jungshik Shin]

So, in case of google, with 'en, ja', 'en' is not ignored (because it doesn't
match 'en-us') and 'ja' is honored, right? I've gotta check the spec on the
language  negotiation

Comment 10

[Simon Montagu :smontagu]

Everything described here seems to follow the specs[1]. Specifically, "Accept-Language: en-us, en-gb, ja" does not match content tagged as "en". We already have bug 165834 on either automatically inserting the generic language-range in the header or alternatively making the UI clearer.

[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4