234033 - Remove "X-Accept-Language" header

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect)

Description

[Harold Beumer]

User-Agent:       
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6) Gecko/20040206

The Mozilla Mail and Mozilla Thunderbird mail clients generate an e-mail header
"X-Accept-Language: en-us, en".

When using the Mozilla application suite, the language settings can be changed
using the BROWSER's language options.

When using Mozilla Thunderbird, the UI does not allow one to change the language
settings.

Either the language settings should be made a configurable option or the mail
header line "X-Accept-Language: en-us, en" should be omitted. I'm not sure
what's the use of this header line in an e-mail message. I'd prefer it to be
omitted.



Reproducible: Always
Steps to Reproduce:
1.
2.
3.

Comment 1

[Hermann Schwab]

comment 0 tells this bug is in Thunderbird only, 
Product -> Thunderbird

Comment 2

[Scott MacGregor]

I'd happily take it out if someone can do some RFC research to figure out what
the story is.

Comment 3

[Harold Beumer]

I did some RFC research on this topic.

http://www.cs.tut.fi/~jkorpela/headers.html indicates it to be non standard and
Netscape specific.

The official source: RFC 2822 does NOT define this header X-Accept_Language. It
really is a HTTP header, not an e-mail header.

RFC 2822 allows any other (optional) headers than those defined in the standard
(RFC 2822 - section 3.6.8), but states: "For the purposes of this standard, any
optional field is uninterpreted". So "X-Accept-Language" is meaningless as far
as RFC 2822 is concerned.

Comment 4

[Harold Beumer]

I think the best option would be to remove this header.
This means the bug is not Thunderbird only but MailNews core.
Product -> MailNews

Comment 5

[Hermann Schwab]

If you file this as a Mail&News bug, it is of Severity Enhancement, as it is
working in Mozilla, not broken.
I also don´t see that you gave reason why this should be removed, besides
'Mozilla should not have what my Thunderbird don´t have'
Mozilla contains a lot of non-standard things, and there is reason for some of them.

Comment 6

[Wouter Van Hemel]

I would like to hear the reason for this non-standard header in either
Mozilla-Mail or Mozilla-Thunderbird.

Reasons why I want to see it removed:
(1) it's useless and non-standard, AFAIK
(2) extra line added in all emails (bandwidth, diskspace)
(3) privacy issues
(4) it's not set to the right language in most cases anyway

PS: The 'OS' field should be changed to 'all' (I am on Linux, so I guess it
counts for all operating systems).

Comment 7

[timeless]

have you read about what the X- prefix means?

Comment 8

[Wouter Van Hemel]

I know what the 'X-' in headers means, yes. It means they are useless. ;)

I just would like to know the reason a non-standard header is included by
default, especially when it appears to be of such little use. And the value is
even set wrong, and I can't adjust (if I would even wish to do so). Arguably,
Mozilla should not include or disclose any information in headers that's not
required to make email work (or the user does not wish to reveal). I hope this
will disappear or be granted a preference (perhaps with other headers such as
X-Mailer).

Comment 9

[Harold Beumer]

Comment #6 confirms this bug.
Could someone please change the status of this bug to NEW or ASSIGNED

Comment 10

[Jens Bannmann]

Confirming bug.

Harold, if you want the functionality removed from the back-end, change the
Summary, Product/Component and Severity fields accordingly.

I could imagine this header field might be useful to indicate which languages
the sender understands, so the recipient can choose in which language he wants
to reply. However, as this is a non-standard-header only set by Netscape and
Mozilla, and not even displayed by these applications in standard settings (=
"all headers" mode off), it really is useless IMO.

Besides, we can safely assume that most mail is sent in a language both the
sender and the recipient understand, and that the recipient will likely write
the reply in that same language...

Comment 11

[Jens Bannmann]

(In reply to comment #2)
> I'd happily take it out if someone can do some RFC research to figure out what
> the story is.

Scott, I found the story behind it: It's a proprietary feature which is supposed
to tell Netscape Messaging Server in which language it should send automated
replies.

The code was apparently taken over from the original Netscape code released in
1998, and never changed since then (links to classic and current file below):
http://lxr.mozilla.org/classic/source/lib/mailto/msgsend.cpp#2946
http://lxr.mozilla.org/mozilla/source/mailnews/compose/src/nsMsgCompUtils.cpp#530

More information about the feature itself was available at
developer.netscape.com, which apparently was taken down in March 2004. However,
you can still access a wayback.org cached copy of the appropriate chapter in the
"Netscape Messaging Server 4.1 Manual" via http://tinyurl.com/7y7vg .

Comment 12

[Jens Bannmann]

Morphing bug according to comment 2.

Comment 13

[Jens Bannmann]

Attachment: no longer add the X-Accept-Language header (checked in)

David, do you have time to review this? Just 8 lines removed in one file.

Comment 14

[Vidar Haarr (not reading bugmail)]

Couldn't you just remove nsMsgI18NGetAcceptLanguage() alltogether ?
The only other place it's used is in 
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1208>,
it seems, and it only checks if it's set to "en" or not.

Comment 15

[Jens Bannmann]

(In reply to comment #14)
> Couldn't you just remove nsMsgI18NGetAcceptLanguage() alltogether ?
> The only other place it's used is in 
>
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1208>,
> it seems, and it only checks if it's set to "en" or not.

I hesitated to touch that code - first, it isn't exactly in the scope of this
bug, second, it looks like that would turn off folding for en-US builds, and I'm
not sure if this is desired...

Comment 16

[Jens Bannmann]

Comment on attachment 170087 [details] [diff] [review]
no longer add the X-Accept-Language header (checked in)

Does this need SR from someone else? If not, please SR it yourself.

Comment 17

[Vidar Haarr (not reading bugmail)]

(In reply to comment #15)
> I hesitated to touch that code - first, it isn't exactly in the scope of this
> bug, second, it looks like that would turn off folding for en-US builds, and I'm
> not sure if this is desired...

I say it is the scope of this bug, since the code is useless once this bug is fixed.

The only place where logic will be touched is this place:
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1224>
if ((charset && *charset && PL_strcasecmp(charset, "us-ascii") != 0) || 
    (language && *language && PL_strcasecmp(language, "en") != 0 &&
     PL_strcasecmp(language, "en-us") != 0))
More specifically, the last part of the if:
    (language && *language && PL_strcasecmp(language, "en") != 0 &&
     PL_strcasecmp(language, "en-us") != 0))
Which will never be true any more (except if other messenger applications use
the header?), since the actual header is removed.

Comment 18

[Jens Bannmann]

(In reply to comment #17)
> I say it is the scope of this bug, since the code is useless once this bug is
fixed.
> 
> The only place where logic will be touched is this place:
>
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1224>

Umm... but isn't this RFC2231ParmFolding function called at places that do not
have anything to do with X-Accept-Language? There's even a pref named
"mail.strictly_mime.parm_folding" involved...
http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#968
http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1040

Mailinglist posting related to this stuff:
http://www.landfield.com/usefor/2002/Sep/0478.html

Comment 19

[Jens Bannmann]

(In reply to comment #17)
> Which will never be true any more (except if other messenger applications use
> the header?), since the actual header is removed.

We remove the part where that header is sent, but we do not remove the
intl.accept_languages pref that nsMsgI18NGetAcceptLanguage() retrieves, so the
function isn't exactly useless. Whether that whole folding-depends-on-language
thing is required or not would depend on RFCs and perhaps
other-MUA-compatibility, but I don't really want to make that decision...

Comment 20

[Vidar Haarr (not reading bugmail)]

(In reply to comment #18)
> Umm... but isn't this RFC2231ParmFolding function called at places that do not
> have anything to do with X-Accept-Language? There's even a pref named
> "mail.strictly_mime.parm_folding" involved...

Yes, but those two callers just pass in nsMsgI18NGetAcceptLanguage():
<http://lxr.mozilla.org/seamonkey/source/mailnews/base/util/nsMsgI18N.cpp#540>
Which checks the pref intl.accept_languages for "en" or "en-us" (once we're in
RFC2231ParmFolding).

I'm not entirely sure, but I think intl.accept_languages always contain en-us or en.

In any case, the language parameter can be removed, and the
nsMsgI18NGetAcceptLanguage() retrieval can be moved inside RFC2231ParmFolding(...).

> Mailinglist posting related to this stuff:
> http://www.landfield.com/usefor/2002/Sep/0478.html

As far as I can see, mail.strictly_mime.parm_folding isn't involved this late in
the process, but is checked before RFC2231ParmFolding is called, so it wouldn't
be touched by removing this code.

Comment 21

[Jens Bannmann]

(In reply to comment #20)
> I'm not entirely sure, but I think intl.accept_languages always contain en-us
or en.

No, it reflects the settings in Prefs>Navigator>Languages. On my system, it's
set to "de-de, de, en-us, en", but people *could* remove english via the UI.

> In any case, the language parameter can be removed, and the
> nsMsgI18NGetAcceptLanguage() retrieval can be moved inside
RFC2231ParmFolding(...).

Agreed. Updated patch coming up...

> As far as I can see, mail.strictly_mime.parm_folding isn't involved this late in
> the process, but is checked before RFC2231ParmFolding is called, so it wouldn't
> be touched by removing this code.

RFC2231ParmFolding() is only invoked when that pref is set to 2, see the if
conditions here:
http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#961
http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1033

Comment 22

[Vidar Haarr (not reading bugmail)]

(In reply to comment #21)
> No, it reflects the settings in Prefs>Navigator>Languages. On my system, it's
> set to "de-de, de, en-us, en", but people *could* remove english via the UI.

Yes, so if people remove it, the second part of the if .. will fail. If it's
there, it will return true, but why?

Why should it return true for en-us, but not for nb-NO? I guess we need bienvenu
or mscott to answer that.

> RFC2231ParmFolding() is only invoked when that pref is set to 2, see the if
> conditions here:

Yes, but I haven't proposed to touch that pref or its logic at all, so I don't
know why we're discussing it.

Comment 23

[Jens Bannmann]

I only mentioned the pref to (In reply to comment #22)
> Yes, but I haven't proposed to touch that pref or its logic at all, so I don't
> know why we're discussing it.

I brought it up only to make clear that the retrieval of intl.accept_languages
is obviously important somehow, and we can't just remove it - comment 14 and
comment 17 seemed to propose that (though I probably misunderstood that part).

Comment 24

[Vidar Haarr (not reading bugmail)]

From what I can see, ducarroz checked this in as part of bug 86089, although
with no explanation as to why we need to check for "en" or "en-us".
sspitzer sr'ed this change, so I'm CC'ing him (even though it says he doesn't
read bugmail). I'm guessing varada is out of the game, but I'm CC'ing him just
in case.

(In reply to comment #23)
mail.strictly_mime.parm_folding is not related to this at all, lets stop
discussing it. I never proposed to remove anything related to that pref.

Comment 25

[Jens Bannmann]

Reassigning due to lack of time on my side.

Comment 26

[Vidar Haarr (not reading bugmail)]

dmose: What we're wondering is why, in:
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1224>
We check for "en" or "en-us". The code is called when we're adding an attachment
to a message and output its filename.
In <https://bugzilla.mozilla.org/show_bug.cgi?id=52428#c27>, Momoi summarizes
that we use the "system encoding", but what if intl.accept_languages only
contains for example "nb" and "nb-no" and not en/en-US (which should really
yield the same result, since they're both iso-8859-1 ?

Or am I completely missing something?

Comment 27

[David :Bienvenu]

Comment on attachment 170087 [details] [diff] [review]
no longer add the X-Accept-Language header (checked in)

since Scott already said he was fine with removing this, pending rfc
investigation, since done, I'll sr it as well.

Comment 28

[Asa Dotzler [:asa]]

not going to block on this but it's still got time to land for alpha6 if someone
can get it in today. 

Comment 29

[Vidar Haarr (not reading bugmail)]

Bienvenu: If you're OK with attachment 170087 [details] [diff] [review] as-is, could you check it in?
Could you read comment 26 as well? It would be nice to clean the patch up
further if it's possible.

Comment 30

[Katsuhiko Momoi]

I'm sorry to be so late in writing this comment, but originally this header was
probably used to match the language of the response msg when the other party
requests a return msg. Before removing this code, can you possibly check to see
if if there is any code dependency on this header, e.g. return request reaply
msg language?  One logic might be something like:

** Return the msg in Japanese if your L10n language is "ja" and if the
requesting party's mail is in Japanese encoding (i.e. iso-2022-jp) and if the
msg contains X-Accept-Language header value of "ja".

As you can see, the logic is somewhat convoluted and we may not be doing
anything about this now. 

Comment 31

[Vidar Haarr (not reading bugmail)]

(In reply to comment #30)
> ** Return the msg in Japanese if your L10n language is "ja" and if the
> requesting party's mail is in Japanese encoding (i.e. iso-2022-jp) and if the
> msg contains X-Accept-Language header value of "ja".

I'm not sure what you mean, but the code in question (pointed to in comment #26)
does not differentiate on language; it only checks if the language is "en-us" or
not - which is why we're (I'm :) wondering what's going on.
Also, this only applies to the encoding of filenames for attachments, as far as
I can see from the code - not the message as a whole..

Comment 32

[Katsuhiko Momoi]

(In reply to comment #31)
> (In reply to comment #30)
> > ** Return the msg in Japanese if your L10n language is "ja" and if the
> > requesting party's mail is in Japanese encoding (i.e. iso-2022-jp) and if the
> > msg contains X-Accept-Language header value of "ja".
> 
> I'm not sure what you mean, but the code in question (pointed to in comment #26)
> does not differentiate on language; it only checks if the language is "en-us" or
> not - which is why we're (I'm :) wondering what's going on.
> Also, this only applies to the encoding of filenames for attachments, as far as
> I can see from the code - not the message as a whole..


I don't believe that Bug #52428 used the Accept-Language Header info in any
significant way. You can get system encoding by quering the OS itself. 
What I remember is something like this:

1. You're a Japanese localized mail client. Your MDN msgs are stored in 2
languages -- English and Japanese. 

2. The question is: when do we sent back the MDN in Japanese, i.e. 
the MDN in the localized language. 

3. For Communictor for sure, we sent back the Japanese MDN only if 1) you have
localized MDN for Japanese AND 2) the msg you received has "ja" as the most
favored Accept-Language as listed in the X-Accept-Language Header. I'm not
certain if we carried that over into Mozilla Mail. 

What I am not certain is if the Mozilla MDN is designed this way with regard to
the language choice. And as far as I know, that was the only practical use
scenario for this X-header. If the requester had set the browser Accept Language
to include "ja" as the most favored language, it's a fair assumption that the
recipient can read Japanese MDN.

Comment 33

[Vidar Haarr (not reading bugmail)]

Moimoi: I don't think this issue is related to MDN at all - or even messages in
general; it's just about encoding filenames. Please read comment #26.

Comment 34

[Jens Bannmann]

(In reply to comment #30)
> I'm sorry to be so late in writing this comment, but originally this header was
> probably used to match the language of the response msg when the other party
> requests a return msg. Before removing this code, can you possibly check to see
> if if there is any code dependency on this header, e.g. return request reaply
> msg language?

Well, as far as http://lxr.mozilla.org/seamonkey/search?string=x-accept-language
reports, this header is never *read*, only *set*. Btw: Interpreting this header
on incoming messages would not make sense anyway, as no other well-known MUA
sends it.

As I wrote in comment 11, it is only used for Netscape mail servers (see
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#529>).

Comment 35

[Katsuhiko Momoi]

> 
> Well, as far as http://lxr.mozilla.org/seamonkey/search?string=x-accept-language
> reports, this header is never *read*, only *set*. Btw: Interpreting this header
> on incoming messages would not make sense anyway, as no other well-known MUA
> sends it.

You are right about other MUA's not using this. I verified this feature for
Communicator 4.5x, however, and it worked as I described -- an additional
convenience feature available only to Netscape users. 
From all the discussions that preceded, it is clear that Mozilla has not used it. 
> 
> As I wrote in comment 11, it is only used for Netscape mail servers (see
>
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#529>).

Yes. Thanks for reminding me about it. I think this was the original reason why
this header was created. An additional covenience features for Netscape Mail
Server users -- only available through Netscape Mail Program. And we applied the
same sort of consideration for MDN language also. But this feature was not
carried over to Mozilla. 

To Vidar, I read comment #26 and the bug that it refers back to. I am surprised
the Accept-Language value is consulted at all for this. It seems strange to
consult that value for encoding file names of attachments. What I mean by
"system encoding" is a query to the OS. 

Comment 36

[Vidar Haarr (not reading bugmail)]

(In reply to comment #35)
> To Vidar, I read comment #26 and the bug that it refers back to. I am surprised
> the Accept-Language value is consulted at all for this. It seems strange to
> consult that value for encoding file names of attachments. What I mean by
> "system encoding" is a query to the OS.

Yes, exactly; I find it strange, too. Had it been consulting a value that
represented an encoding, then I would probably not think more of it, but since
it explicitly asks for "en" or "en-us", that has me wondering:
1. If that code even works/is used ?
2. If it does, does it only work for english countries ?

So that's what I was asking in comment 26, Momoi (and why I CC'ed you): Can we
remove that check? And if not; why not?

Comment 37

[Doug Wright]

*** Bug 247570 has been marked as a duplicate of this bug. ***

Comment 38

[Katsuhiko Momoi]

(In reply to comment #36)

> Yes, exactly; I find it strange, too. Had it been consulting a value that
> represented an encoding, then I would probably not think more of it, but since
> it explicitly asks for "en" or "en-us", that has me wondering:
> 1. If that code even works/is used ?
> 2. If it does, does it only work for english countries ?
> 
> So that's what I was asking in comment 26, Momoi (and why I CC'ed you): Can we
> remove that check? And if not; why not?

Upon further check, RFC 2231 allows both encoding and language info to be set. 

http://www.faqs.org/rfcs/rfc2231.html

See this exmple contained in there:

     Content-Type: application/x-stuff;
     title*=us-ascii'en-us'This%20is%20%2A%2A%2Afun%2A%2A%2A

Note in particular "us-ascii'en-us' ..."
and the specification in 

7.  Modifications to MIME ABNF

of this RFC.

I don't understand the coding details in the code you're discussing but there is
a reason for checking the language-value. 
Please answer this question:

Is the language value in the code supplied by the value in the X-Accpet-Language
header? After all, the language value is used by X-Accept-Language and
Accept-Language headers but the lang value itselsf exists independent of these
headers.

Comment 39

[Vidar Haarr (not reading bugmail)]

(In reply to comment #38)
> Is the language value in the code supplied by the value in the X-Accpet-Language
> header? After all, the language value is used by X-Accept-Language and
> Accept-Language headers but the lang value itselsf exists independent of these
> headers.

The only two callers of RFC2231ParmFolding(...) are these two:
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#967>
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1039>
As you can see, these two both send in nsMsgI18NGetAcceptLanguage() as "language":
<http://lxr.mozilla.org/seamonkey/source/mailnews/base/util/nsMsgI18N.cpp#540>
Which only operates on the pref intl.accept_languages, so the X-Accept-Language
header is not used at all.

Comment 40

[Vidar Haarr (not reading bugmail)]

Attachment: removes nsMsgI18NGetAcceptLanguage from RFC2231ParmFolding

bienvenu: This patch removes what has been discussed above.
However, we're still not sure if it should be removed or not - could you please
read comment 26 ?

Comment 41

[Jonas Bofjall]

This discussion has died out but the header is still there. What is blocking the 
removal of it? (I read #26 but do not see the problem.)

Comment 42

[Vidar Haarr (not reading bugmail)]

The header is not there? It was removed by attachment 170087 [details] [diff] [review]. The only notable
mention of "X-Accept-Language" in the entire source tree exists in
<http://lxr.mozilla.org/mozilla/source/mailnews/extensions/mdn/src/nsMsgMdnGenerator.cpp#82>.
The #define is unused, however, and could be removed.

The bug has morphed into discussing removing something else, and we still need
someone who understands the code - like Bienvenu - to say whether it is safe to
remove it or not, and why.

Comment 43

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to comment #26)
> dmose: What we're wondering is why, in:
>
<http://lxr.mozilla.org/seamonkey/source/mailnews/compose/src/nsMsgCompUtils.cpp#1224>
> We check for "en" or "en-us". The code is called when we're adding an attachment
> to a message and output its filename.

Is that really the right lxr link?  When I look at the file, I see no reference
to "en-us" at all.

Comment 44

[Vidar Haarr (not reading bugmail)]

Right.

In between the discussion and your answer, the whole thing has been resolved by
the patch in bug 193439.

Filed bug 307807 about the leftovers.

-> FIXED.

Comment 45

[Harold Beumer]

(In reply to comment #44)

I'm glad to see this bug has been resolved. When will I be able to see the
result in Mozilla Thunderbird as my current version 1.06 still adds the header
"X-Accept-Language: en-us, en" to all my outgoing e-mails. Will Thunderbird 1.07
solve the problem?

Comment 46

[Vidar Haarr (not reading bugmail)]

(In reply to comment #45)
> I'm glad to see this bug has been resolved. When will I be able to see the
> result in Mozilla Thunderbird as my current version 1.06 still adds the header
> "X-Accept-Language: en-us, en" to all my outgoing e-mails. Will Thunderbird 1.07
> solve the problem?

This patch was never checked in to branch (1.0.x), only trunk. Thunderbird 1.5
should have the fix.

Comment 47

[Jean-Marc Desperrier]

I'm coming a bit late, but I would like to rise an issue about this bug.

Accept-Language is now an standardized header for mail.
RFC 3282 defines it, listing in it's use in "MIME body parts" (&3. The Accept-Language header) and RFC 4021 includes it in the list of registered mail headers (&2.1.27.  Header Field: Accept-Language).

So even if Mozilla doesn't use it as input, it makes sense to include it in outputs message. It takes only 5 lines of code to implement it, so removing it doesn't remove any significant bloat. nsMsgI18NGetAcceptLanguage is a bit longer, but couldn't it be mutualized with browser code that does the same ? (in fact it's all the MIME code that should be mutualized, if only to stop correcting MIME bugs twice).

Comment 48

[Tristan Miller]

(In reply to Jean-Marc Desperrier from comment #47)
> I'm coming a bit late, but I would like to rise an issue about this bug.
> 
> Accept-Language is now an standardized header for mail.
> RFC 3282 defines it, listing in it's use in "MIME body parts" (&3. The
> Accept-Language header) and RFC 4021 includes it in the list of registered
> mail headers (&2.1.27.  Header Field: Accept-Language).
> 
> So even if Mozilla doesn't use it as input, it makes sense to include it in
> outputs message.

I've just raised this as a separate issue (Bug 1081980).