Closed Bug 307023 Opened 19 years ago Closed 15 years ago

Re-editing quoted-printable HTML message (Open Draft, Forward Inline, Edit-as-New) shows corruption of tag attributes (<img> <body> etc)

Categories

(MailNews Core :: Composition, defect, P2)

Product:
MailNews Core
Component:
Composition
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED
Milestone:
Thunderbird 3.0b3

People

(Reporter: st, Assigned: rkent)

References

Details

Attachments

(9 files, 1 obsolete file)

quoted-printable mail triggering the bug
71.87 KB, text/plain
Details
8bit mail not triggering the bug
71.85 KB, text/plain
Details
mbox with four sample messages
10.82 KB, text/plain
Details
another (minimal) example showing the problem (mbox format)
3.50 KB, text/plain
Details
html-message, single-part, signed
7.74 KB, message/rfc822
Details
html-message, multipart/related, signed
12.41 KB, message/rfc822
Details
picture of re-edit html-message, multipart/related, signed
97.32 KB, image/png
Details
proposed fix
2.03 KB, patch
standard8
: review+
neil
: superreview+
Details | Diff | Splinter Review
force no decoding in multipart related output
1.12 KB, patch
Bienvenu
: review+
Bienvenu
: superreview+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.7.10) Gecko/20050717 Firefox/1.0.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.7.5) Gecko/20041206 Thunderbird/1.0 (20041206)

When trying to (inline) forward the attached testcase mail, images are not
displayed in the HTML compose window. Double-clicking on an image's placeholder
the image properties window appears. The graphic's address is stated as follows:

ailbox:///C|/Dokumente% ...
/Local%20Folders/Inbox?number2360653&part=1.1.2&filename=image001.gif"

Thunderbird seems to have misaligned its copy of the URL, dropping the first
character (m for mailbox). Opening the extended attributes in the property
window, the same problem is visible for HTML-attribute src (see above) and name
(displayed as mage001.gif")

When trying to send the broken mail, Thunderbird reports that it cannot send it
and I should 'Enter a password for mage001.gif" at (null)'.

Trying to forward the mail as attachment gets the mail out, but Thunderbird
doesn't add the images.

Everything works fine if the mail answered and not forwarded, all of the mail's
content is quoted perfectly, including the images. The original mail is
displayed correctly in Thunderbird's main window, too.

Note that the bug appears only with the attached testcase. I have no problems
inline forwarding HTML messages with graphics composed by Thunderbird. The
T-Online mailer (T-Online = kind of German AOL, largest online service in
Germany) used to compose the testcase is known as bad software so the source of
the problem might be a violation of some MIME RFC. Specifically, the
Content-Disposition header of the image parts misses the disposition keyword:

Content-Disposition: ; filename="image001.gif"

Manually fixing this to

Content-Disposition: inline; filename="image001.gif"

doesn't help, though.

Reproducible: Always

Steps to Reproduce:




I am unsure if this is a duplicate of bug 2903 since that one seems to be about
loosing images when forwarding an HTML mail as plain text. This is _not_ the
case with my bug, I'm forwarding as HTML. I'm also not having problems with any
mail, just with the testcase (well, until now).
Found the bug:

Thunderbird doesn't properly decode quoted-printable messages, the HTML composer
looses one character at the beginning of HTML attribute strings. No idea why
this happens only when forwarding (answering is fine).

Some source examples and the resulting attributes from the extended image
property window:

<img name=3D"image.gif" src=3D"cid:part0" width=3D"321">
name  = mage.gif"
src   = ailbox:... (see below)
width = 21

<img name=3Dimage.gif src=3D"cid:part0" width=3D321>
name    = age.gif
src     = ilbox:... (see below)
width21 = 

Apart from that, it's not possible to safely edit the end of the src attribute.
The complete data in there is
ailbox:///C|/Dokumente%20und%20Einstellungen/u/Anwendungsdaten/Thunderbird/Profiles/default/jke1rd4j.slt/Mail/Local%20Folders/Inbox?number7259846&part=1.1.2&filename=image.gif""
, however, the last " is not displayed at first, it'll only appear if you enter
another character at the end. Second, the cursor is sometimes located in the
middle of characters and new characters appear some characters left of the cursor.

If Content-Encoding is changed to 7bit or 8bit, everything's fine. That's the
reason why my Thunderbird is able to forward its own HTML mails - it sends as
7bit by default.
Summary: Cannot forward HTML message with inline images → Fails to forward images in HTML message if encoding is quoted-printable
..and I hope you like the image :)
Version: unspecified → 1.0
This bug might be related to bug 287394.

Sorry for the mess, last one was in parts quoted-printable, parts 8bit..
Attachment #194827 - Attachment is obsolete: true
Could you try this in the latest builds and see if you're still seeing this?
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-trunk/
Yes, same problem using version 1.6a1 (20050903) (and 1.0.6 (20050716)).
For your convenience, the exact same message just with 8bit encoding. Content
differs only in six bytes (apart from the Content-Encoding header), three times
=3D replaced with = .

This one can be forwarded flawlessly.
Testing with 1.6a1-1201, Win2K.

The q-p source of the message's <img> tag is:
  <img name=3D"image.gif" src=3D"cid:image" width=3D"321">

On forwarding, this ends up in the buffer as:
  <img moz-do-not-send="true" name="mage.gif&quot;"
   src="ailbox:{path}?number%3F122&amp;part=1.1.2&amp;filename=image.gif%22"
   width="21">    /* where {path} is the path into my mail folders */

The 'name', 'src' and 'width' attributes have all been corrupted.  (I believe the moz-do-not-send attribute is the result of a recent fix, and that will prevent the image from being forwarded anyway, but that's not specific to this bug.)

The MIME structure of the sample message is pretty straightforward:
 Top: Content-Type: Multipart/Alternative
   Part 1: Content-Type: text/plain; C-T-E: quoted-printable
   Part 2: Content-Type: multipart/related
     Part 2.1:  Content-Type: text/html; C-T-E: quoted-printable
     Part 2.2:  Content-Type: Image/gif; C-T-E: base64

The problem is seen even with a more stripped down version:
 Top: Content-Type: multipart/related
   Part 1:  Content-Type: text/html; C-T-E: quoted-printable
   Part 2:  Content-Type: Image/gif; C-T-E: base64

However, I had another HTML/quoted-printable test message where the HTML was not including the image (and not showing the corruption); its structure was:
 Top: Content-Type: multipart/related
   Part 1:  Content-Type: multipart/alternative
     Part 1.1: Content-Type: text/plain; C-T-E: quoted-printable
     Part 1.2: Content-Type: text/html; C-T-E: quoted-printable
   Part 2:  Content-Type: image/gif; C-T-E: base64

If I tweaked this message to mirror the structure of the test message, it exhibits the same problem -- and, it also shows the same corruption of the attributes of <a> elements, as in bug 287394.

Unsurprisingly, the same symptom is seen with Seamonkey 1.0.  Resummarizing, moving to Core.
Assignee: mscott → nobody
Status: UNCONFIRMED → NEW
Component: Message Compose Window → MailNews: Composition
Ever confirmed: true
OS: Windows XP → All
Product: Thunderbird → Core
Hardware: PC → All
Summary: Fails to forward images in HTML message if encoding is quoted-printable → Forward-inline of quoted-printable HTML message shows corruption of tag attributes
Version: 1.0 → Trunk
This attachment is an mbox; copy it into a mail folder directory and start TB (or Seamonkey).  The four messages exhibit the four cases discussed in 
comment 8.

See also bug 173012 -- this problem apparently applies to cases where the body is Base64 as well as for quoted-printable.
*** Bug 287394 has been marked as a duplicate of this bug. ***
this attachment in mbox format shows also the bug.
the 1st messages is the original one, the 2nd one the forwarded one.
In the 2nd one the http-link is corrupted. No images or base64 stuff endocding are needed here to reproduce the bug. (Tested with TB 1.5 under German Windows XP)
(In reply to comment #11)
> Created an attachment (id=209197) [edit]
> another (minmal) example showing the problem (mbox format)
> 
> this attachment in mbox format shows also the bug.

As included, I don't see any evidence of corruption, except that the second message (the forward) appears to be incomplete.
Comment on attachment 209197 [details]
another (minimal) example showing the problem (mbox format)

From - Fri Jan 20 06:56:51 2006
Return-Path: <newsletter-fail@8-seasons.com>
Delivered-To: GMX delivery to test@thunderbird.net
Received: (qmail invoked by alias); 19 Jan 2006 17:44:17 -0000
Received: from h023.thunderbird.net (EHLO tiny.thunderbird.net) [195.135.159.23]
  by mx0.thunderbird.net (mx083) with SMTP; 19 Jan 2006 18:44:17 +0100
Received: from tiny.thunderbird.net (tiny.thunderbird.net [195.135.159.23])
	by tiny.thunderbird.net (Postfix) with ESMTP id CC28718D29;
	Thu, 19 Jan 2006 17:54:14 +0100 (CET)
Content-Type: multipart/related;
    boundary="----------=_1137689650-32022-0";
    charset="iso-8859-1"
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.411 (Entity 5.404)
From: news@thunderbird.net
To: news@thunderbird.net
Reply-To: news@thunderbird.net
Subject: thunderbird mail forward test
Message-Id: <20060119165414.CC28718D29@tiny.thunderbird.net>
Date: Thu, 19 Jan 2006 17:54:14 +0100 (CET)

This is a multi-part message in MIME format...

------------=_1137689650-32022-0
Content-Type: text/html; charset="iso-8859-1"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

<html>
<body>
<p>
<a href=3D"http://www.mozilla.org">test</a>
</p>
</body>
</html>

------------=_1137689650-32022-0


------------=_1137689650-32022-0--
From - Sat Jan 21 22:59:47 2006
X-Account-Key: account2
X-UIDL: e27d2f415834622dcf4a01576ad011d2
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Return-Path: <bernhard.baier@gmx.net>
X-Flags: 0000
Delivered-To: GMX delivery to bernhard.baier@gmx.net
Received: (qmail invoked by alias); 21 Jan 2006 21:59:27 -0000
Received: from p549837E7.dip0.t-ipconnect.de (EHLO [192.168.178.20]) [84.152.55.231]
  by mail.gmx.net (mp017) with SMTP; 21 Jan 2006 22:59:27 +0100
X-Authenticated: #751631
Message-ID: <43D2AECD.7040809@gmx.net>
Date: Sat, 21 Jan 2006 22:59:41 +0100
From: Bernhard Baier <bernhard.baier@gmx.net>
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To: Bernhard Baier <bernhard.baier@gmx.net>
Subject: [Fwd: thunderbird mail forward test]
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: 7bit
X-GMX-Antivirus: -1 (not scanned, may not use virus scanner)
X-GMX-Antispam: 0 (Mail was not recognized as spam)
X-GMX-UID: 6MugY3QMeSEvY24bYHQhaXN1IGRvb0Be

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
<br>
-------- Original-Nachricht --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
 cellspacing="0">
  <tbody>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">Betreff: </th>
      <td>thunderbird mail forward test</td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">Datum: </th>
      <td>Thu, 19 Jan 2006 17:54:14 +0100 (CET)</td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">Von: </th>
      <td><a class="moz-txt-link-abbreviated" href="mailto:news@thunderbird.net">news@thunderbird.net</a></td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">Antwort an: </th>
      <td><a class="moz-txt-link-abbreviated" href="mailto:news@thunderbird.net">news@thunderbird.net</a></td>
    </tr>
    <tr>
      <th align="right" nowrap="nowrap" valign="baseline">An: </th>
      <td><a class="moz-txt-link-abbreviated" href="mailto:news@thunderbird.net">news@thunderbird.net</a></td>
    </tr>
  </tbody>
</table>
<br>
<br>
<p>
<a href="ttp://www.mozilla.org%22">test</a>
</p>
</body>
</html>
Attachment #209197 - Attachment description: another (minmal) example showing the problem (mbox format) → another (minimal) example showing the problem (mbox format)
sorry, I just wanted to update my attachment (date 2006-01-21 06:14 PST), but instead it was inserted as comment - sorry for that (I'm new to Bugzilla)!
@Mike Cowperthwaite: It's true, in my attachment the really interesting part was missing: In the 2nd mail (i. e. the forward one) as you see in comment #13 the http-link is crippled:
"ttp://www.mozilla.org%22"
In the meantime I made some closer investigations of this bug by stepping through the code.
I found out that the following conditions must be given that the bug occurs:
1. The MIME content type must be multipart/related (i. e. multipart/alternative or multipart/mixed don't show the error)
2. The message body must have any "non-trivial" endcoding i. e. quoted printable, base64, uuencode (*not* 7bit, 8bit etc.)
4. The message must be forwarded (reply doesn't show the error)

Then (only für the multipart/related case) a temporary file "nsmail.htm" will be created holding the (decoded) message body (file mailnews/mime/src/mimedrft.cpp, function mime_decompose_file_init_fn)

Now a strange thing happens: The decoder function will be called *twice* on the the buffer holding the message body until it will be written to the file "nsmail.htm". Then this file is used as the source for the composer window.
This explains easily the effects with broken links and image references e. g. with the encoding "quoted printable".
Example:
Original<a href=3D"http://www.mozilla.org">test</a>
after 1st decoding: <a href="http://www.mozilla.org">test</a>
after 2nd decoding: <a href=ttp://www.mozilla.org">test</a>
(Comment: for "quoted printable" the '=' character is interpreted as an escape character)


One could also check this effect with a "multipart/related" test message using the encoding "base64" for the message body. Then the message body is totally unreadable after the 2nd decode pass.

Although I found out these facts fairly easy I don't really understand until now *why" the decoder will be called twice - and how to fix this.

To be more concrete let's use as an example again the encoding "quoted printable" and let's then have a look on the source code:
The decoding is then done in mimeenc.cpp, function mime_decode_qp_buffer. The 1st time this function will be called in order to decode the full message in one step. Before leaving the function mime_decode_qp_buffer the function "MimeInlineText_parse_decoded_buffer" will be called via the function pointer stored in data->write_buffer. Let's then break on the function "mime_decode_qp_buffer" and then look on the call stack ("deepest" function 1st):

mime_decode_qp_buffer
MimeDecoderWrite
mime_decompose_file_output_fn
real_write
flush_tag
mime_multipart_related_output_fn
MimeOptions_write
MimeObject_write
MimeInlineTextHTML_parse_line
MimeInlineText_rotate_convert_and_parse_line
convert_and_send_buffer
mime_LineBuffer
MimeInlineText_parse_decoded_buffer
mime_decode_qp_buffer

So the function mime_decode_qp_buffer is called *recursively* again!

What is now happening between the 2 calls of "mime_decode_qp_buffer"?
E. g. in the function "mime_multipart_related_output_fn" the HTML message body is parsed again in order to search for "cid:" references in IMG tags which are used in "multipart/related" MIME messages. If such a reference will be found the HTML code will be patched in order to point to the decoded image (leading later to the broken "ailbox:///C|/Docume.../Local%20Folders/Inbox?n" references)

This is now the time to have a general look on the design of MIME multipart handling in the Mozilla suite.

This is what I would expect in a clean design:
Step 1: Decode (recursively) all kind of MIME multipart messages with a generic decoder and store all the decoded parts somewhere (e. g. in a temporary directory)
Step 2: Perform all actions on the decoded multipart tree which are specific for the particular mulitpart (sub) type. (E. g. for multipart/related patch all the cid: references in IMG tags [although I personally don't find this a very elegant design - the resolution of such references could also be done in other ways - it seems that also the author of mimemrel.cpp was very unsure about this - see as a hint his/her comments at the beginning of the source file)] )

What is actually done in the Mozilla suite (as I claim to understand it after several debugging sessions - I wasn't able to find any design documentation ...):
Step 1: Create / fill a MIME data structure holding also function pointers for the MIME type specific actions.
Step 2: Recursively decode the multipart messages and perform all multipart type specific actions by calling the function pointers created in step 1)

Here I think Mozilla's design tries to do too many things in one (recursivly nested) step gaining no real benefit but for the price of a code which is very difficult to comprehend and maintain.
Also I don't see in Mozilla's design a clear separation between the generic multipart procedures and the ones which are specific for a particular multipart type. As a consequence much of the code dealing with multipart related/alternative/mixed etc. messages seems to be doubled - also not the best basis for further maintaince...

Although I think I described the preconditions of this bug sufficiently (see above) and although I had several (long) debug sessions I'm even not able to present a "quick hack" to fix the bug - not to speak about a clean fix with no side effects. 

My personal conclusion is:
- A complete redesign of the MIME (multipart) handling in Mozilla/Thunderbird would be the best.
- In the current state it will be very difficult to provide a fix for the bug where it is clear that it has no side effects (of course I would be happy if there is anybody in the Mozilla team or somewhere else in the world who can convince me of the opposite;-)

and ... I'm speaking here about a really annyoing bug and not a little "flaw". I guess about 70% percent of all (my) HTML messages are affected by this Mozilla bug. 

Has anybody any hint how to proceed here? (Does it e. g. make sense to mark is blocking for future releases of Thunderbird?)

Best regards
Bernhard
xref bug 287517, and also attachment 169181 [details] from bug 196180 (which is 
actually unrelated to that bug).
Bernhard, thx for the diagnosis.  I agree that a complete redesign of the mime code is in order, but it's unlikely to happen any time soon unless someone has a few months to spend on it. I'll try to see if I can come up with a fix.
Assignee: nobody → bienvenu
This problem also occurs for Edit As New; and, it affects the 'background' 
attribute of the <body>, which can lead to additional problems (not possible to save or send the message at all, until the image URL is fixed or cleared).
See bug 331711.
*** Bug 331711 has been marked as a duplicate of this bug. ***
(In reply to comment #18)
> This problem also occurs for Edit As New; and, it affects the 'background' 
> attribute of the <body>, which can lead to additional problems (not possible
> to save or send the message at all, until the image URL is fixed or cleared).
> See bug 331711.

Also re-opening drafts that have been saved with quoted-printable; this seems more important than the Forward and Edit as New cases, as it's ruining the user's own editing.

And it's not just 'background' that gets munged, but the color attributes are broken as well.

Steps to Reproduce:
1) Turn on the option "For message containing 8bit characters, use 
   'quoted-printable'..."
2) Create a new HTML message.  
3) Set the encoding to ISO-8859-15 or UTF-8 -- for some reason, using 
   ISO-8859-1 saves the supra-ASCII characters as HTML entities rather than 
   q-p bytes.  
4) Enter some text with supra-ASCII characters.
5) Insert an image, and a smiley.
6) Save the message as a draft and close the window.
7) Reopen the draft.

Actual results:
Background color is wrong; image is missing; any smileys are shown as plain text.

Expected results:
None of the above: message should look as it did when I clicked Save.
Severity: normal → major
Summary: Forward-inline of quoted-printable HTML message shows corruption of tag attributes → Re-editing quoted-printable HTML message (Open Draft, Forward Inline, Edit-as-New) shows corruption of tag attributes
Blocks: 348468
Depends on: 224733
Summary: Re-editing quoted-printable HTML message (Open Draft, Forward Inline, Edit-as-New) shows corruption of tag attributes → Re-editing quoted-printable HTML message (Open Draft, Forward Inline, Edit-as-New) shows corruption of tag attributes (<img> <body> etc)
QA Contact: composition
OS: Win98;  Win2000
Thunderbird 1.5.0.9

Using "Edit as new " option on an e-mail that is in "Unsent Messages" folder; much of the formatting information is lost.  One that is easy to see is the wrap.  The new e-mail seems to have fixed width and wraps at fixed boundary; not like the editor when "Write" is used that wraps at window width.

This problem also exists in Netscape 7.x
Bug 403595 has a good example attached, also showing that only some attributes are corrupted. Color specifications #xxxxxx do not seem to be affected. Text arguments appear to be always truncated at the first character and &quot; are frequently appended to those, which was also reported in bug 287394.

The last paragraph in bug 428179 reports that the problem was also seen in replies, which were considered to be not affected (comment #15).

Further potential dupes:
 - bug 394322 reports empty composition with Simple HTML display;
 - bug 401922 awaits response from reporter to confirm symptoms.

Clearly a usability issue.
Flags: wanted-thunderbird3.0a2?
Flags: wanted-thunderbird3.0a2?
Flags: wanted-thunderbird3.0a2+
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3?
Product: Core → MailNews Core
Flags: wanted-thunderbird3.0a2+
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3?
Flags: blocking-thunderbird3+
Priority: -- → P2
Target Milestone: --- → Thunderbird 3.0b2
This is not just a "usability issue", but seriously trashes S/MIME signing of mulitpart/related messages. See bug 459682.
This is a HTML-message without an image, signed using S/MIME. Re-editing this message work okay.
Attachment #344861 - Attachment description: html-message, singel-part, signed → html-message, single-part, signed
This is a HTML-message with an included image (resulting in multipart/related), signed using S/MIME. Re-editing this message trashes the message (see next attachment).
This picture shows the trashed message and it's html body (using the add-on html-edit). The red marks show the places where the html was trashed. This leaves the impression that quo-pri is decoded twice. eg.:

   alt="" src="imap:...

becomes

   alt="src="map:...
bug 459682 is closely related to this one. Here is a short summary of bug 459682:
A HTML-message
- containing umlauts (8-bit characters)
- containing an image included into the message
- send/saved with S/MIME-signing is switched on
gets trashed.

comment #28, comment #29 and comment #30 related to this.
Opening the message of attachment 344863 [details] using KMail work. Thus this bug is an decoding problem.
I tried to find some hint for the bug and discovered a function flush_tag at
<http://mxr.mozilla.org/comm-central/source/mailnews/mime/src/mimemrel.cpp#797>.
This function rewrites the cid: urls contained in a multipart/related message.
Maybe there is a one-off-error in there. Unfortunately I'm to less used to C to debug this.
I think the problem is that when we're outputting things like <a href=http://www.mozilla.org"/a>, we get confused and think that we're still decoding quoted printable, so we eat the =h. MimeDecoderWrite thinks the data is encoded, but we've already decoded it...We may need to push and pop the data encoding when writing out these tags...

>	mime.dll!mime_decompose_file_output_fn(const char * buf=0x0aa28350, int size=0x00000009, void * stream_closure=0x0661b730)  Line 1976 + 0x14 bytes	C++
 	mime.dll!real_write(MimeMultipartRelated * relobj=0x0ab3e640, const char * buf=0x0aa28350, int size=0x00000009)  Line 746 + 0x20 bytes	C++
 	mime.dll!flush_tag(MimeMultipartRelated * relobj=0x0ab3e640)  Line 828 + 0x14 bytes	C++
 	mime.dll!mime_multipart_related_output_fn(const char * buf=0x0aa710ee, int size=0x00000042, void * stream_closure=0x0ab3e640)  Line 969 + 0x9 bytes	C++
 	mime.dll!MimeOptions_write(MimeDisplayOptions * opt=0x0ac639d0, const char * data=0x0aa710d8, int length=0x00000058, int user_visible_p=0x00000001)  Line 1700 + 0x14 bytes	C++
 	mime.dll!MimeObject_write(MimeObject * obj=0x07ce73e0, const char * output=0x0aa710d8, int length=0x00000058, int user_visible_p=0x00000001)  Line 1732 + 0x18 bytes	C++
 	mime.dll!MimeInlineText_parse_decoded_buffer(const char * buf=0x0aa710d8, int size=0x00000058, MimeObject * obj=0x07ce73e0)  Line 346 + 0x13 bytes	C++
 	mime.dll!mime_decode_qp_buffer(MimeDecoderData * data=0x07d84fc0, const char * buffer=0x0aa710d8, int length=0x00000000)  Line 199 + 0x1a bytes	C++
 	mime.dll!MimeDecoderWrite(MimeDecoderData * data=0x07d84fc0, const char * buffer=0x0aa710d8, int size=0x0000005c)  Line 838 + 0x11 bytes	C++
 	mime.dll!MimeLeaf_parse_buffer(const char * buffer=0x0aa710d8, int size=0x0000005c, MimeObject * obj=0x07ce73e0)  Line 174 + 0x14 bytes	C++
Attached patch proposed fix — — Splinter Review 
I guess nothing in libmime should shock me anymore, but this was pretty broken, and broken for a very long time.

The problem was that we were decoding the text for body parts twice, as the previous stack shows. So if your quoted printable body had an '=' after decoding, that '=' would get eaten in the second decoding. So the fix is not to run body parts through a second decoder, but still decode external parts, which we will then put in the compose window, re-encode, and send.

This (only :-)) affects edit message as new, edit draft, and forward inline. Since forward inline is now the default, and quoted printable seems to be getting used more rather than less, I think this bug is more important than it may have once been.

I tried a few tests, including attachments 209197 and 194828, and forwarding inline messages with binary attachments, and so far everything seems ok.
Attachment #346954 - Flags: superreview?(neil)
Attachment #346954 - Flags: review?(bugzilla)
Attachment #346954 - Flags: superreview?(neil) → superreview+
Attachment #346954 - Flags: review?(bugzilla) → review+
fix checked in - changeset:   1064:65665add33af
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Depends on: 464969
V with: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1b2pre) Gecko/20081116 Shredder/3.0b1pre

Thanks, David.
Status: RESOLVED → VERIFIED
Depends on: 465586
this fix is going to be backed out because of the regressions it caused, e.g.,  Bug 465586. Sorry about this - libmime is not easy :-( there's an other potential fix in bug 465586 but for beta 1, we're just going to live with this bug and not the regressions.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
moving to b2
Target Milestone: Thunderbird 3.0b1 → Thunderbird 3.0b2
moving to b3 - as painful as it is, we've shipped several products with this bug.
Target Milestone: Thunderbird 3.0b2 → Thunderbird 3.0b3
Just to confirm it is still standing in the latest Shredder nightlies (b2pre) and to ask if there is any workaround.

@ David: You propose a fix in comment #36. Does it work? How can I apply this? Do I need to compile Shredder myself? If yes, how do I merge your fix to the code? Please point to instructions if too busy to explain.

Thank you in advance
klonos, it fixes some issues, but causes new issues/bugs, so I'm not sure it's worth applying and building your own version of shredder with.
If anyone is looking for a challenge, fixing libmime to deal with this is a nice one.
Status: REOPENED → NEW
Keywords: helpwanted
Assignee: bienvenu → kent
Status: NEW → ASSIGNED
Keywords: helpwanted
Here's my shot at this. The buffers used for output in multipart/related are decoded upstream where needed, so I force the output to not decode. Not sure if I can't just turn them off then, but my fear is that if the message is processed in chunks, something upstream may still need to know that the original part is encoded. Anyway, this works on all of the samples I could find, including attachment 194828 [details], attachment 205259 [details], and attachment 288468 [details] from bug 403595.
Attachment #371933 - Flags: superreview?(bienvenu)
Attachment #371933 - Flags: review?(bienvenu)
I know I tried something like this very early on, but perhaps not exactly like this. I'll try this patch out - thanks for looking at this!

unit tests for the mime conversion process are a bit challenging, though I think Andrew might have done some for gloda.
The gloda JSMimeEmitter/mimemsg stuff could help if you can run the 'forward' logic in an automated fashion that could then be re-streamed for gloda's mime emitter.  So, 'forward' to a draft message on disk.  Then stream that through the JS Mime emitter where you can verify the resulting MIME structure and body payload is correct.  (Because the various libmime modes of operation are mutually exclusive, and the JS emitter depends on a specific mode with a specific option, it needs its own run.)

http://mxr.mozilla.org/comm-central/source/mailnews/test/resources/messageGenerator.js can also help in creating the initial message programatically, if so desired.

Of course, if you can do all that, you probably (more) easily just have an input file on disk that you pass through the forwarding stage and verify that its output matches another file on disk.

Maybe nsMsgComposeService::LoadDraftOrTemplate could be made scriptable without requiring a call to nsMsgComposeService::OpenComposeWindow, and that would work?

cc'ing sid because I think he's done a lot more pragmatic testing work in this area...
Whiteboard: [needs r/sr bienvenu]
Attachment #371933 - Flags: superreview?(bienvenu)
Attachment #371933 - Flags: superreview+
Attachment #371933 - Flags: review?(bienvenu)
Attachment #371933 - Flags: review+
Comment on attachment 371933 [details] [diff] [review]
force no decoding in multipart related output

this works for the various tests I could find - let's try it out in the real world. Thx, Kent!
fix checked in.
Status: ASSIGNED → RESOLVED
Closed: 16 years ago → 15 years ago
Resolution: --- → FIXED
Whiteboard: [needs r/sr bienvenu]
(In reply to comment #55)
> this works for the various tests I could find - let's try it out in the real
> world. Thx, Kent!
Could you please provide the download link.  Thank you.
Verified that the test case in attachment 205259 [details] is fixed, on WinXP/64 with 
(32-bit) Gecko/20090425 Shredder/3.0b3pre

I haven't looked at the other attachments nor any of the dupes.  If reporters of bugs duped to this aren't seeing the problem fixed with a current 3.x build, they should reopen those bugs.  If reporters to this bug still see the problem with their test cases, then it's another bug.

Comment 25 suggested bug 394322 as a dupe, but that symptom persists.
Sorry: really marking Verified now.
Status: RESOLVED → VERIFIED
Trying to test/verify in other platforms and latest 3.1 nightlies.

So what is/are the test case(s) now? Do I follow the steps in comment #0 to repro the bug or what?
saving one of the test cases (e.g. https://bugzilla.mozilla.org/attachment.cgi?id=209197 ) as a mailbox, and doing a forward inline or edit message as new on the message should let you recreate the problem and verify the fix.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: