161337 - pull accept-language header from system prefs

Status: RESOLVED FIXED

(Camino Graveyard :: General, enhancement)

Description

[François Frisch]

We have two instalations of Chimera 0.4 . 

number1 is on a dual-1GHz OSX 10.1.5. A previous version of Chimera was on the
machine and was overwriten by 0.4.
number2 is on dual 500Mhz OSX10.1.5. A previous version of Chimera had been
installed (probably older than the one that was on number1) but had been deleted
some time ago. I installed Chimera 0.4 a few days ago. 

The problem is that on number2 Chimera is not sending any accept-language
header. It is sending "accept-language = ("en-us, en;q=0.50");" on number1. This
caused our localized WebObjects app to throw an exception (something we have to
fix on our side). However this could be a problem with other sites. I'm not sure
if accept-language is a compulsory header...

Here is a full printout of the headers.

number1:

httpVersion=HTTP/1.1 headers={user-agent = (Mozilla/5.0 (Macintosh; U; PPC Mac
OS X; en-US; rv:1.0.0) Gecko/20020724); keep-alive = (300); host =
(www.foobar.com:8080); accept =
(text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1);
accept-encoding = (gzip, deflate, compress;q=0.9); accept-language = ("en-us,
en;q=0.50"); connection = (keep-alive); } content-length=0 cookies=null
userInfo=null>) method=GET uri=/cgi-bin/WebObjects/ElectronicCommerce.woa
defaultFormValueEncoding=ISO8859_1  formValueEncodingDetectionEnabled=NO
formValueEncoding=ISO8859_1 formValues={}

number2:

httpVersion=HTTP/1.1 headers={user-agent = (Mozilla/5.0 (Macintosh; U; PPC Mac
OS X; rv:1.0.0) Gecko/20020724); keep-alive = (300); host =
(www.foobar.com:8080); accept =
(text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1);
accept-encoding = (gzip, deflate, compress;q=0.9); connection = (keep-alive); }
content-length=0 cookies=null userInfo=null>) method=GET
uri=/cgi-bin/WebObjects/ElectronicCommerce.woa
defaultFormValueEncoding=ISO8859_1  formValueEncodingDetectionEnabled=NO
formValueEncoding=ISO8859_1 formValues={}

Comment 1

[Steve Smart]

*** Bug 161343 has been marked as a duplicate of this bug. ***

Comment 2

[saari (gone)]

have you tried blowing away your profile? Sounds like something we may have
added inbetween versions. Pink, bryner, ideas?

Comment 3

[François Frisch]

I tried deleting the chimera directory in ~/Library/Application Support/ , no 
change. Is there anything else you might need? 
How does Chimera get its list of languages? 

Comment 4

[Greg K.]

Is there anything I can use to view the accept-language header being sent?

Comment 5

[François Frisch]

I use the printout of my WebObjects application but you could use 
http://www.ElfQrin.com/binfo.shtml

Comment 6

[kaldari]

According to the specs, Chimera is actually doing the right thing now...

From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html:

"As intelligibility is highly dependent on the individual user, it is
recommended that client applications make the choice of linguistic preference
available to the user. If the choice is not made available, then the
Accept-Language header field MUST NOT be given in the request."

Check out bug 182610 for adding a preference to choose accept-language header.

Comment 7

[François Frisch]

The accept-language should be changed by using the international system
preferences pane. This is what I think is the expected behavior by users.
Mac OS X provides a built in selection of preferred languages that is what
should be used, don't reinvent the wheel.

Comment 8

[kaldari]

Problem is, I doubt the Mac OS Language setting uses HTTP language tags (defined
in RFC 1766). If they do, I would be happy to reopen this bug, but otherwise
there is no guarantee that a particular language chosen in the System prefs will
correspond to a valid language tag.

Comment 9

[François Frisch]

Ya Ok I had another look at the preference pane and  bug 182610. The
international pane is only language so even if it was using http language tags
it doesn't have the necessary country. let alone the variant.

Comment 10

[kaldari]

Correction: RFC 1766 has been depricated by RFC 3066. Also, RFC 3066 is based on
ISO-639 (two-letter and three-letter language codes) and ISO-3166 (two-letter
country codes).

BTW, I came across a page at unicode.org that shows a table of which Mac
language codes correspond to which ISO language codes:

http://www.unicode.org/unicode/onlinedat/languages.html

I have no idea if this table is valid for Mac OS X, but if so, it may be
possible to pull the language tags from the system prefs after all (as I believe
the country and variant codes are optional). I'll go ahead and reopen the bug
and leave it up to the developers to decide. Ideally, I would still prefer a
separate preference (bug 182610) for the following reasons:

* Apple's language pref is for "application menus and dialogs".

* The W3C spec specifies that the choice should be available in the client
application. It does not mention OS preferences being applicable.

* People who develop web pages or web applications in several languages (such as
myself) would love to have a way to test their pages without having to change
the language for the entire OS.

That said, if bug 182610 is deemed to be bloat, this bug may be a decent
work-around, as having no language code at all is less than ideal. Of course,
I'm still not sure if this bug is even doable. Perhaps one of the developers
could give us a better idea.

Comment 11

[kaldari]

Changing summary and severity to reflect repurposing of this bug per comments
7-10. I probably should have just opened a new bug, but I'm a lazy bastard :)

Previous summary: "accept-language header missing"
New suumary: "pull accept-language header from system prefs"

Comment 12

[Stephane Moureau]

You "must" have a different dialog, it has nothing to do with the Sys Pref.
In a lot of countries, users are obliged to use a program in one language, no
translated version available, that does not mean they want to see the web pages
in that language.
In Belgium we have 3 languages, sites are using that very often, an exception
apple.be :-)

Comment 13

[Steve Smart]

RE: comment 12, the system preference lets you rank the order of languages which
you want to use, so you should put your preferred language at the top, and the
system will fall back to other languages if a localization is not available. You
shouldn't set a language at the top just because it's the one that more programs
have available. I don't see why these wouldn't be exactly the same settings to
use for web browsing; it should be a list of all supported languages in the
order the user is most comfortable with them.

The real problem is that the system prefs don't contain all the possible HTTP
languages. They probably do contain the ones 90% of our users will use. An
alternate solution would be to use the international language settings to create
the default settings, which then only the other 10% of users would have to change.

Comment 14

[Stephane Moureau]

Steve, that's a one way solution. You think program then page, you can have also
the contrary. Think about it you'll see you are obliged to have both, programs
via SysPref and Page.

Comment 15

[Steve Smart]

The user will always set up their program language preferences before their page
language preferences because you set up the system preferences language during
the Mac OS X installation process. But I still agree that we need a seperate
setting since the system preference doesn't cover all the languages available
for web pages, I just think we should use system prefs to set the default the
first time a user starts Chimera.

If a user's native language is available under system prefs, then it will
already have been set as the first preferred language during system
installation, and by setting it as their default accept-language, we're saving
them a trip to the Chimera preferences.

Comment 16

[kaldari]

*** Bug 187169 has been marked as a duplicate of this bug. ***

Comment 17

[Josh Aas]

Some more commentary on this bug would be helpful. Any new perspectives?

Comment 18

[Josh Aas]

Can somebody compare our current accept-language header behavior to that of
Firefox on Mac and Windows?

Comment 19

[Josh Aas]

Oh yeah - also a comparison with Safari would be nice.

Comment 20

[Bruce Davidson]

Okay, a quick summary of the different browsers behaviour.

Camino: Claim to understand US English fluently and all other English dialects
only half as well. (Accept-language= "en-us,en;q=0.5"). No way of changing this
without resorting to the source code. (If about:config worked the user could
alter the intl.lanugages option, see bug 209070).

Safari: Take the list of languages in the "International" pane of Systems
Preference. Claim to understand the first language (English in my case)
fluently. Claim to understand the rest of the checked languages less and less
well (docking 1/n for each entry further down the list of n languages). For each
language (apart from English, not sure it this is because its special, or
because its my first language) claim to understand the primary dialect (eg.
"fr-fr") slightly better (~0.05 units) better than the generic (e.g. "fr").
(Accept-language= en, ja;q=0.29, es;q=0.88, de-de;q=0.82, de;q=0.76,
nl-nl;q=0.71, nl;q=0.65, it-it;q=0.59, it;q=0.53, da-dk;q=0.47, da;q=0.41,
ja-jp;q=0.35, fr;q=0.94, ko-kr;q=0.24, ko;q=0.18, no-no;q=0.12, no;q=0.06)
This is fine expect that the international pane has far too many languages
ticked by default (sorry Norwegians, but you'll have to make do with my (very
limited) Danish). 

Mozilla and Firefox: Take the languages in order from their own preferences
dialog and divide up the unit space equally between them. (Accept language =
en-gb,en;q=0.8,en-us;q=0.6,fr;q=0.4,de;q=0.2)

Basically Camino is completely broken for those whose first language isn't
English, and broken (but useable) for all those of us who consider English
rather than American our preferred language. I'm not sure how many people will
have edited their international preferences pane, certainly mine isn't as
refined a list as I have set up in Mozilla at work). However it has the
advantage of being what's used by the other big Mac browser, and it would avoid
the need to write lots of UI. I think we should probably pick up the information
from system preferences.

Comment 21

[Bruce Davidson]

Okay, I think what we need to do is:

1. Pull the list of languages from system preferences (see
http://developer.apple.com/documentation/Cocoa/Conceptual/Internationalization/Concepts/InternatSupport.html)
for details of how to do this. Note that this page implies you get back a name
for the language rather than an ISO code, but other pages imply that only ISO
codes are supported. The code for WebCore might make this clearer).

2. For the top language in the list we need to work out what dialect we want to
declare. We can do this by examining the user's region.
(http://developer.apple.com/documentation/CoreFoundation/Conceptual/CFLocales/Articles/CFLocaleWorking.html#//apple_ref/doc/uid/20002241/BBCEFFGC)

3. Set up the intl.languages option in Gecko with something link
L1-XX, L1;q=0.95, L2;q=(n-1)/n, L3;q=(n-2)/n, ... Ln;q=1/n

Where L1 is the lanuage at the top of the list and XX is the region we
determined, and L2...Ln are all teh other languages in the list. (I don't think
we can sensibly determine the region for these, and I don't like the logic that
Safari uses of "assume its the same as the language".)

Comment 22

[Bruce Davidson]

cc'ing Ludo as its probably of interest to people interested in localising the
browser! Are they currently recoding the language-accept header as part of
localisation?

Comment 23

[Bruce Davidson]

Taking this bug.

Comment 24

[Bruce Davidson]

Attachment: Patch

This patch implements the algorithm I outlined in my earlier post. Seems to
generate a reasonable accept-language header. Something in necko (?) is
truncating the numbers to only 1 dp, so the numbers generated are slightly
different from Safari.

I'm going to ask Ludo to do the first review of this one. It would benefit from
testing by people who's primary language isn't English.

Comment 25

[Bruce Davidson]

This patch unfortunately causes us to effectively ignore any setting for
intl.accept_languages the user may have in their user.js file, in favour of our
computed value. This is a bit nasty, but I can't see a way round this. If we
only compute the value if the pref is empty then we will always end up using the
values we compute the first time round, even if the user subsequently edits the
languages list in System Preferences.

Is there any way to determine if a preference was set in user.js or prefs.js?

Also, I've realised the necko isn't doing rounding to 1dp I mentioned. Its
actually computing its own q values. IMHO its wrong for necko to do this, and we
should still compute q values in case necko ever changes its behaviour to accept
q values from the pref.

Comment 26

[Jasper]

We need some good test cases for this bug, becuase I really don't know how o
test this to see if it works correctly.

Comment 27

[Mina]

This page seems to detect correctly the languages accepted:
http://www.christian-gerner.de/computer/status406-lang.php

Comment 28

[Ludovic Hirlimann [:Usul]]

Comment on attachment 167885 [details] [diff] [review]
Patch

>Index: src/preferences/PreferenceManager.mm
>===================================================================
>RCS file: /cvsroot/mozilla/camino/src/preferences/PreferenceManager.mm,v
[snip]
> 
>     [self configureProxies];
>+		
>+		// Now work out the user's preferred accept-language headers

You are using tabs here, and you should not : see
http://www.mozilla.org/hacking/mozilla-style-guide.html#Visual for your code
formating within mozilla

>+#if DEBUG

Remove debug code please.

Patch applies correctly. I have a few comments however on the way the patch
works :
I hadn't cared to modify my language list in the System preference hence my
list of accepted language was long - this is unecessary we should limit this to
say 5 languages (I'm giving here a number, we should discuss the value for this
limit).

and sorry for the delay bruce.

Comment 29

[Wevah (Nate Weaver)]

Something I ran into while testing this was that english was represented on my
machine not as 'en' but as 'English', which isn't a valid Accept-Language token.
I didn't see anything in the patch taking this into account (I'm guessing what I
experienced isn't the case on all systems).

Comment 30

[Wevah (Nate Weaver)]

Also, I am curious as to why the line

#import <Foundation/NSUserDefaults.h>

is in the patch when importing <Cocoa/Cocoa.h> already imports the
NSUserDefaults class.

Comment 31

[Bruce Davidson]

(memo to self): The System Preference's lanuage list already includes some
regional variants. However these are currently returned in the form "en_GB", and
will need to have the underscore changed to a hyphen. We should probably also
check that we don't send the same language twice with our "guess the country"
approach for your primary language.

Comment 32

[Bruce Davidson]

Attachment: Revised patch

Updated patch addressing Ludo's review comments, Wevah's review comment. Also
includes protection against human readable locale names that Wevah encountered
on systems upgraded from 10.1 and deals with languages from the system
preferences pane that include a dialect indicator.

Requesting review.

Comment 33

[Bruce Davidson]

Attachment: Update patch

Updated patch that returns nil rather than empty string for invalid locales for
slightly cleaner code. (Thanks to Wevah).

Comment 34

[Bruce Davidson]

Attachment: Updated patch with nil checks in "house style"

Changing the null checks to if ( x ) rather than if ( x != nil ) to match rest
of code.

Comment 35

[Wevah (Nate Weaver)]

As far as some languages coming through as their English names (as was my case
until I fiddled), you might take a look at
/System/Library/PrivateFrameworks/IntlPreferences.framework/Versions/A/Resources/EnglishToISO.strings
for a nice conversion table.

Comment 36

[Ludovic Hirlimann [:Usul]]

Comment on attachment 171685 [details] [diff] [review]
Updated patch with nil checks in "house style"

>+      r = [NSString stringWithString:language];
>+
>+    [language release];
>+    return r;
>+}

>+    NSUserDefaults *defs = [NSUserDefaults standardUserDefaults];
>+    NSArray *languages = [defs objectForKey:@"AppleLanguages"];
>+    NSString* primaryLocale = [defs objectForKey:@"AppleLocale"];
>+		
>+    // Set up the user's primary language using the locale (so we get dialect info)
>+    NSMutableArray* acceptableLanguages = [[NSMutableArray alloc] init];
>+		if ( primaryLocale != nil ) {

the if should be aligned.

>+      NSString* primaryLanguage = [PreferenceManager convertLocaleToHTTPLanguage:primaryLocale];
>+      if ( primaryLanguage )
>+        [acceptableLanguages addObject:primaryLanguage];
>+    }
>+
>+    // Now set up all the other languages the user understands
>+    // (from System Preferences | International). Strip duplicates because the
>+    // user may have their primary locale set up as a language too (e.g. British English)
>+    for ( unsigned i = 0; i < [languages count]; ++i ) {
>+      NSString* language = [PreferenceManager convertLocaleToHTTPLanguage:[languages objectAtIndex:i]];
>+      if ( language && ![acceptableLanguages containsObject:language] )
>+        [acceptableLanguages addObject:language];
>+    }
>+
>+    // Now set up the accept-language header itself
>+    // Note that necko will determine quality factors itself
>+    NSMutableString* acceptLangHeader = [[NSMutableString alloc] init];
>+    for ( unsigned i = 0; i < [acceptableLanguages count]; ++i ) {
>+      if ( i > 0 )
>+        [acceptLangHeader appendString:@","];
>+
>+      [acceptLangHeader appendString:[acceptableLanguages objectAtIndex:i]];
>+    }
>+    [acceptableLanguages release];
Shoudldn't you call RemoveAllObjects before releasing the Array ?
>+
>+    if ( [acceptLangHeader length] > 0 )
>+      [self setPref:"intl.accept_languages" toString:acceptLangHeader];
>+		
>+    NSLog( [self getStringPref:"intl.accept_languages" withSuccess:NULL] );
should be removed

Comment 37

[Bruce Davidson]

Attachment: Updated patch

> Shoudldn't you call RemoveAllObjects before releasing the Array ?

Releasing an NS[Mutable]Array releases any objects still in the array. There's
therefore no need to do so explicitely by removing these objects before calling
release.

The other review comments (alignment and a stray NSLog that had somehow
survived the cull) are fixed in this new patch.

Comment 38

[Ludovic Hirlimann [:Usul]]

Comment on attachment 172020 [details] [diff] [review]
Updated patch

r+

Comment 39

[Julian Pellico]

Comment on attachment 172020 [details] [diff] [review]
Updated patch

changing self to r not sr

Comment 40

[Julian Pellico]

Comment on attachment 172020 [details] [diff] [review]
Updated patch

Nit: In places like
+    for ( unsigned i = 0; i < [languages count]; ++i ) {
I'd liek to see 'unsigned int' rather than 'unsigned', but I'll let pink decide
whether it matters.

Looks fine, r+

Comment 41

[Mike Pinkerton (not reading bugmail)]

> +    NSArray* localeParts = [anAppleLocale componentsSeparatedByString:@"_"];
> +			
> +    [language appendString:[localeParts objectAtIndex:0]];

are we guaranteeed that |localeParts| will have an object at item 0? What if
|anAppleLocale| is nil?

> ++ (NSString*)convertLocaleToHTTPLanguage: (NSString*)anAppleLocale

nit, please use |inAppleLocale| for the param name.

> +    NSMutableString* acceptLangHeader = [[NSMutableString alloc] init];
...
> +    [acceptLangHeader release];

for things that are alloc'd/released in the same scope, I generally prefer that
you autorelase on the alloc/init line. It prevents someone from accidentally
adding a return between the alloc/release (introducing a leak) and is more
consistent with what we do in the rest of the app.

Comment 42

[Bruce Davidson]

Attachment: Updated patch addressing pink's comments

Added nil check in the conversion locale subroutine to protect against nil
(redundant in current code, but better safe than sorry).

Renamed subroutine argument.

Changed several local variables to be autoreleased.

Comment 43

[Bruce Davidson]

Attachment: Updated patch

Updated patch to address nit with NSMutableString initialisation that Mike
raised on IRC. Verbal sr=pink received on IRC with this change.

Comment 44

[Mike Pinkerton (not reading bugmail)]

Comment on attachment 172293 [details] [diff] [review]
Updated patch

one more nit ;)

+    NSMutableArray* acceptableLanguages = [[[NSMutableArray alloc] init]
autorelease];

can just be

... = [NSMutableArray array];

Comment 45

[Bruce Davidson]

Attachment: Absolutely the last patch - hopefully!

Comment 46

[Mike Pinkerton (not reading bugmail)]

Comment on attachment 172367 [details] [diff] [review]
Absolutely the last patch - hopefully!

sr=pink

Comment 47

[Josh Aas]

landed on trunk

Comment 48

[Mike Pinkerton (not reading bugmail)]

landed on branch