142092 - Support content-language header/tag for multipart/alternative message display

Status: NEW

(MailNews Core :: MIME, defect)

Description

[nhottanscp]

http://www.ietf.org/rfc/rfc1766.txt

The RFC mentions that multipart/alternative message can specify language.

> 4.  Use of Content-Language with Multipart/Alternative
> 
>    When using the Multipart/Alternative body part of MIME, it is
>    possible to have the body parts giving the same information content
>    in different languages. In this case, one should put a Content-
>    Language header on each of the body parts, and a summary Content-
>    Language header onto the Multipart/Alternative itself.

Comment 1

[nhottanscp]

http://lxr.mozilla.org/seamonkey/source/mailnews/mime/src/mimemult.cpp

Comment 2

[Léa Gris]

Attachment: A complete testcase message in its original eml format

With mozilla 1.5 the last MIME block is displayed. With the test case message
it ownly display the french part.

I expect mozilla-mail to gain some language preferences or use the language
preferences from the navigator if needed and apply the same rules in resolving
the preferable language to display in a multi language multipart/alternatife
MIME message as does Apache server in multiview when provided a Accept-language
header.

Comment 3

[Rafael Cordones Marcos]

Attachment: multipart/alternative message with content-language tags

The testcase message contains an error. The first part contains the English
message but is tagged with "Content-Language: fr". I've added an attachment
with a corrected version.

Comment 4

[Rafael Cordones Marcos]

Here's a preliminary patch! This is my first patch to Mozilla so excuse my 
"newbieness". 
 
We use the language preferences available to the user at "Edit" -> 
"Preferences" -> "Navigator" -> "Languages". These preferences are then stored 
in the "prefs.js" file under the key "intl.accept_languages". The value 
for this key is a string of ISO language tags like, for instance, "es-es, 
es, en, en-us, sv". 
 
* Rationale 
 
Mozilla, when confronted with a "multipart/alternative" 
message, treats each part sequentially. For each part 
it checks if it can display it, if so, then it keeps it 
cached in a buffer and checks the next part. If the 
next part can also be displayed then Mozilla discards 
the previous cached part and saves the new part for 
possible display. When it has processed the last part 
it then displays the cached part. So, given a  
"multipart/alternative" message with several parts tagged 
with "Content-Language" information, Mozilla will display 
allways the last part (see the code of function "create_child" 
 in mailnews/mime/src/mimemalt.cpp). Furthermore, the 
MIME parsing code, instead of creating an object for 
the multipart/alternative message and populate it with 
all the objects representing each of the the parts of 
the multipart/alternative message, creates only a 
child object for the part that will eventually be 
displayed. 
 
Instead of parsing all the parts and then make the decission 
of which of them to display according the the "Content-Language" 
info it contained, we devised a sequential algorithm that 
would compute in one pass, the appropriate part to show. 
Furthermore, the implementation of this algorithms will require 
very few modification in the MIME parsing code. 
 
We proceed as follows: 
 
1. For each part, we obtain the "Content-Language" 
  information which will consist of an RFC 3066 
  language tag like "en" or "es", for instance. If the part 
  has no "Content-Language" tagging let Mozilla proceed 
  as usual and cancels this algorithm. 
 
2. If the part does contains "Content-Language" 
  information, then calculate the position the tag 
  has in the "intl.accept_languages" string. The first 
  language tag on the list gets position number "1" and 
  so on. We reserve a position of "0" to indicate that 
  the language tag is not in the list. 
 
3. If this part has a language tag with a lower 
  position than previous parts then we save (cache) it 
  as a candidate for presentation. Go to Step 2. 
 
4. Show the last cached part. 
 
For example, if the user language preferences are "es, 
en, sv" and we are evaluating a part that has "en" in the " 
Content-Language" header field, then the position for 
this part is 2. If the part had "es" the position would 
be 1, the first preference. 
 
* Implementation 
 
We modified the following source files: 
 
 - mailnews/mime/public/nsMailHeaders.h  
   which defines an interface to the standard header fields. 
 
  We added the definition of "Content-Language" header 
  field and the "differences" parameter to indicate which 
  is the header field that differentiates each of the 
  parts (see Section 4 in RFC 1766). 
 
 - mailnews/mime/src/mimeobj.h  
   which defines the MimeObject class which is the base  
   class for the objects representing all other MIME types. 
 
  We added the definition of the "content_language" 
  attribute to contain the contents of the "Content-Language" 
  header field if it exists. 
 
 - mailnews/mime/src/mimeobj.cpp  
   Contains the code common to all MIME messages. 
 
  We added the functionality to parse the "Content-Language" 
  header field if it exists and to update the "content_language" 
  attribute. 
 
 - mailnews/mime/src/mimemalt.h  
   Defines the MimeMultipartAlternative class and the 
   multipart/alternative MIME container, which displays 
   only one (the `best') of a set of enclosed 46 documents. 
 
  We added the declaration of the "compute_language_position",  
  "part_with_content_language" functions and the "differences" 
  attribute. 
 
 - mailnews/mime/src/mimemalt.cpp  
   Contains the code to parse multipart/alternative messages. 
 
  We modified the function "create_child" in order to 
  check not only that the part that we are currently 
  parsing is displayable but also if it contains "Content-Language" 
  information. To carry on this task we had to create 
  to auxiliary functions: "part_with_content_language" 
  wich checks wether a part has a "Content-Language" 
  header field and "compute_language_position" which 
  implements the algorithms presented in the previous 
  section in order to calculate the position a part has 
  in the language preferences set by a user. 
 
* Limitations and Further Work 
 
 - No tests have been performed with 
   multipart/alternative messages in which parts have 
   different charsets (see related Mozilla bug #127612). 
 
 - The subject line displayed by Mozilla is the one 
   appearing in the outer encapsulating 
   multipart/alternative headers, not the subject line 
   belonging to the chosen part. 
 
 - There should be a possibility for the user to be 
   aware of the other alternative parts and to be able 
   to select them (see Mozilla bug #130119). 
 

Comment 5

[Rafael Cordones Marcos]

Attachment: Add support for multipart/alternative with Content-Language tagged parts

Comment 6

[Rafael Cordones Marcos]

Attachment: A multipart/alternative message with three parts in en, es, sv

A test message containing three parts, each in a different language. 
The message does not containg the "differences" parameter to the Content-Type
header field (see Section 4 in RFC 1766).

Comment 7

[Rafael Cordones Marcos]

Attachment: Add support for multipart/alternative with Content-Language tagged parts

This fixes a bug in the previous patch. If a message contained a part in a
language that was not in the list of preferred languages, it was this part that
was shown.

Comment 8

[Jean-Francois Ducarroz]

Comment on attachment 155687 [details] [diff] [review]
Add support for multipart/alternative with Content-Language tagged parts

First let me say good job, this is a great feature to have.

couple coding issues:

1) /* If there is a cached part and current part has Content-Language haders */
typo: haders -> headers

2) There is way too much debug printf, please remove them before checkin.

3) you could cache the accept_languages pref in the MimeMultipartAlternative
structure to prevent to have to fetch it for each part.

4) if MimeObject_initialize, we check 3 time that obj->headers is not null,
should be done only one time. Please change that while you are working on that
function.


Algorithm main issue:

your algorithm is not compliant with RFC 1766 section 4.1. You correctly
extract the "differences" content-type's parameter but you never used it! I can
understand you choose the automatic detection way but that will certainly cause
trouble when we will have to deal with a message which mix both mode!

Comment 9

[Rafael Cordones Marcos]

Attachment: Support for Content-Language tagged parts in multipart/alternative messages

Changes in 0.4
--------------

 PLEASE NOTE: cd into the directory 'mailnews/mime' before patching!

 - Removed unnecessary DEBUG pritfs.

 - Corrected some typos in comments.

 - Cached the contents of the 'intl.accept_languages' variable:

   * mimemalt.h: added char *lang_prefs     

   * mimemalt.cpp: added initalization code at	 
     MimeMultipartAlternative_initialize

   * mimemalt.cpp: added cleaning code at MimeMultipartAlternative_finalize
     
   * mimemalt.cpp: changed signature of 
		   MimeMultipartAlternative_compute_language_position in
		   order to pass a reference to the language preferences.
     
   * mimemalt.cpp: modified MimeMultipartAlternative_compute_language_position 

		   to access the cached information.
 
 - Added an outer "if" to function MimeObject_initialize in mimeobj.cpp 
   so we check only once that obj->headers is not null.
   
 - Algorithm main issue:
 
   "your algorithm is not compliant with RFC 1766 section 4.1. You correctly
   extract the "differences" content-type's parameter but you never used it!"
 
   Yes, I know I extract the "differences" parameter and I do not use it. 
   I just forgot to document why. I am following J. Postel's robustness
   principle here: "be conservative in what you do, be liberal in what you
   accept from others."
   
   My point is that if we base the detection algorithm *only* onthe presence
   of the "differences" parameter then it's never going to be used. Because
   nobody is currently using this parameter in emails!
   
   I do not understand what you mean with: "I can understand you choose 
   the automatic detection way but that will certainly cause trouble when we
   will have to deal with a message which mix both mode!"
   QUESTION: Do you mean a multipart message wich mixes HTML and plain text
   parts tagged *also* with Content-Language tags?
   
   QUESTION: I remember having seen a Bug about the displaying of the 
   Subject header. But I cannot find it now! :-) Otherwise I will create 
   a new bug called "Correct display of Subject line in multipart/alternative  

   messages" and we can mark it as a duplicate of the other when we find it!

Comment 10

[Rafael Cordones Marcos]

The note for the previous patch was not complete. It should say: "cd into the 
directory 'mailnews/mime' before patching with 'patch -p1 < PATCH'. 

Comment 11

[Frank Tang]

both naoki and I are off mozilla for about 20 monthes. If these bugs are still
here, the real status is 'wont fix'. If you want to reopen it, please find a
better owner who really looking at the bug database now. 

Comment 12

[Travis Chase]

Mass Reassign Please excuse the spam

Comment 13

[Travis Chase]

Mass Re-opening Bugs Frank Tang Closed on Wensday March 02 for no reason, all
the spam is his fault feel free to tar and feather him

Comment 14

[Travis Chase]

Reassigning Franks old bugs to Jungshik Shin for triage - Sorry for spam

Comment 15

[Wayne Mery (:wsmwk)]

maybe this could be a new love interest for Thomas

Comment 16

[David :Bienvenu]

Comment on attachment 155687 [details] [diff] [review]
Add support for multipart/alternative with Content-Language tagged parts

clearing review request on obsolete patch

Comment 17

[David :Bienvenu]

Comment on attachment 162846 [details] [diff] [review]
Support for Content-Language tagged parts in multipart/alternative messages

putting on my review queue to keep track of it.

Comment 18

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #17)
> Comment on attachment 162846 [details] [diff] [review]
> Support for Content-Language tagged parts in multipart/alternative messages
> 
> putting on my review queue to keep track of it.

FYI bienvenu, you'd need to update the patch 'cause it's already bitrotted.

$ patch -p0 --dry-run < ~/Desktop/p142092.diff 
patching file mime/public/nsMailHeaders.h
Hunk #1 FAILED at 49.
Hunk #2 FAILED at 98.
2 out of 2 hunks FAILED -- saving rejects to file mime/public/nsMailHeaders.h.rej
patching file mime/src/mimemalt.cpp
Hunk #1 succeeded at 59 (offset -1 lines).
Hunk #2 succeeded at 83 (offset -1 lines).
Hunk #3 succeeded at 139 (offset -1 lines).
Hunk #5 succeeded at 444 (offset -3 lines).
patching file mime/src/mimemalt.h
Hunk #1 FAILED at 57.
1 out of 1 hunk FAILED -- saving rejects to file mime/src/mimemalt.h.rej
patching file mime/src/mimeobj.cpp
Hunk #1 FAILED at 96.
Hunk #2 FAILED at 180.
2 out of 2 hunks FAILED -- saving rejects to file mime/src/mimeobj.cpp.rej
patching file mime/src/mimeobj.h
Hunk #1 FAILED at 177.
1 out of 1 hunk FAILED -- saving rejects to file mime/src/mimeobj.h.rej

Comment 19

[Gordon P. Hemsley [:GPHemsley]]

Despite this bug being for MailNews, I am going to assume that BCP 47 support in Firefox would relate to this, so I'm giving it the appropriate dependencies.

Comment 20

[Wayne Mery (:wsmwk)]

Gordon, would you be able to update the patch?

Comment 21

[David :Bienvenu]

Comment on attachment 162846 [details] [diff] [review]
Support for Content-Language tagged parts in multipart/alternative messages

clearing review request until we can get a patch that applies against the current trunk.

Comment 22

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.