821250 - Blank email in TB with certain attachments (when equal-sign(=) or underscore(_) is used in boundary, dbmail IMAP server returns "RFC2047 encoded boundary string" as boundary parameter data to "fetch BODYSTRUCTURE" request)

Status: RESOLVED INVALID

(Thunderbird :: Mail Window Front End, defect)

Description

[David Froehlich]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0
Build ID: 20121129050502

Steps to reproduce:

Receive a multipart mail containing some text and an attachment. When opening the mail in TB, I just see an empty message. Other imap clients like roundcube webmail for instance show the mail and all attachments properly. The mail itself is there in TB, Ctrl-U shows me the whole message including text and attachments. However I am neither able to read the text nor to open any attachment. When I click on the mail, sender and subject are shown but the mail window itself is empty. Also the pin icon (attachment) in the message list disappears as soon as I open the mail for the first time. I had a look at various similiar bug reports but none of them helped me. The problem occurs with TB 3, 10 and 17. When I save the mail as .eml, and then open the .eml in TB, it works and I can see the message including attachments. I already rebuilt the .msf without success. But, after having saved the mail to harddisk as eml, then I can also view this particular mail directly in thunderbird. When I then delete the ImapMail directory in my TB profile, the mail shows up empty again. Looks like some kind of cache issue but I did not enable any offline cache at all. I have also attached a complete wireshark capture of the imap connection (without auth.) while trying to open the mail. In the first 10 seconds I just opened the mail which showed me the empty mail as described. After 20 seconds you can see what happens when I press Ctrl-U. Any ideas, maybe something concerning mime header, imap bodystructure etc. confuses TB?


Actual results:

The mail is shown empty.


Expected results:

The mail should be displayed properly.

Comment 1

[David Froehlich]

Attachment: raw .eml file

Comment 2

[David Froehlich]

Attachment: wireshark capture

Comment 3

[David Froehlich]

I could isolate the issue to the mime_parts_on_demand settings. By default this option is enabled which makes sense and of course is how I want to use my imap connection. However when I set mime_parts_on_demand to false, then all my mails are being displayed properly. When I open the mail, I notice that the whole message is being downloaded (progress bar) and then the html body of the mail is shown, I can open the attachment, and everything is fine. Unfortunately the whole message is downloaded including all attachments, even if I am not going to open them. So my intention is to enable mime_parts_on_demand again, but as soon as I do this, the message body disappears again. So can you give me a hint, what TB prevents from properly displaying the attached mail with mime_parts_on_demand enabled?

Comment 4

[David Froehlich]

Attachment: screenshot

Comment 5

[David Froehlich]

Attachment: thunderbird debug log

Comment 6

[WADA:World Anti-bad-Duping Agency]

I couldn't reproduce your problem with your attached .eml, Yahoo! IMAP server, Tb 17.0.1 on Win-XP, with Offline-use=Off folder, with mime_parts_on_demand=true(default setting) and Memory Cache/Disk Cache enabled(default setting), with some addons for debugging purposes.
Same IMAP log as yours was obtained for fetch body[1] of first text/html part in multipart/mixed mail.

(Q1) Do you see your problem on the mail in newly created offline-use=off folder and offline-use=on folder?
(1) Create new IMAP folder of offline-use=on(Folder Properties/Sychronization).
    Copy the mail to this folder.
(2) Create new IMAP folder of offline-use=off(Folder Properties/Sychronization).
    Copy the mail to this folder.

(Q2) Do you see difference among View/Message Body As, Original HTML, Simple HTML Plain Text?

(Q3) Do you see your problem with Tb's safe mode(thunderbird.exe -safe-mode, with all addon disabled)?

(Q4) Do you use special setting such as "Disable Memory Cace", "Disable Disk Cache"?

(Q5) Do you see you problem on any similar mail? (multipart/mixed, first part=text/html, multiple not-inline-displayable attachment parts) Or problm on specific mail only? 

(Q7) When did your problem start to occur on the mail? After upgrade of Tb?

(Q8) Is "Filter move by message filter" relevant to the mail?

(Q9) Specific IMAP server only problem?
     (Free Gmail account and free Yahoo! mail accunt can use IMAP)

Comment 7

[WADA:World Anti-bad-Duping Agency]

(Q10) If anti-virus software quarantines Disk Cache file written by Tb(delete the file), such phenomenon may occur. Do you enable virus scanner for Tb's Disk Cache directory? Is log for quarantine seen?

Comment 8

[David Froehlich]

Thanks for your hints, hi went through all of them, with these results:

(Q1)
(1) no
(2) yes
It works as soon as I download the complete message, i.e. when using offline folder synchronisation or when disabling mime_parts_on_demand.

(Q2) no, I do not see any body in all three modes.

(Q3) Same result when starting TB in safe mode.

(Q4) I am using 50 MB cache with compress when saving 100 KB here (TB 3), on another TB 17 machine I am using 1000 MB cache with compress when 20 MB. Different settings and different TB versions with the same result, but alway at least 50 MB cache.

(Q5) Yes, I have more mails with the same problem. I just found another which unfortunately is an invoice from my ISP which I of course do not want to publish. But I can attach a cutted version without the actual content so you can see the mime headers etc.

(Q7) No upgrade at all. At home and at one of my customers I am still using TB 3 for compatibility reasons (but upgrade to 17esr is planned). On another notebook I have a brand new installied TB 17esr with the same issue. The mailserver is maintained by me which is a dbmail imap server. The last update of dbmail was months ago, the TB problem started some weeks ago.

(Q8) I do not use any TB filter at all.

(Q9) Maybe. At the moment this might be the only fact that is shared among all my TB installations, they all use the same dbmail server. But in my opinion it would be to easy just to blame the dbmail server. It worked for a very long time and I do not have any problems with other imap clients. And if there is a problem with the imap server, I would expect something to see in the imap debug log.

(Q10) I am using various antivurs software, mostly Sophos Endpoing Protection but also Microsoft Security Essential on some machines. All of them do not show any hints regarding quarantine. Disabling them does not make any difference either.

Comment 9

[David Froehlich]

Attachment: another cutted .eml which results in empty body

Comment 10

[WADA:World Anti-bad-Duping Agency]

I checked by test account of your IMAP server, and I could see your problem.
Following is a part of data in Disk Cache.
>(file name)
> C:\Documents and Settings\<id>\Local Settings\Application Data\Thunderbird\Profiles\ueihzgrp.Test\Cache\D\D0\38325d01
>   2012/12/17  12:34            34,081 38325d01
>(part of file content)
> Content-Type: multipart/mixed;
> 	boundary="----=_NextPart_949116202830194211954781"
> 
> --=?us-ascii?q?----=3D=5FNextPart=5F94911620283019421195478?= =?us-ascii?q?1?=
> Content-Type: text/html; charset=UTF-8
> Content-Transfer-Encoding: quoted-printable
Boundary line for text/html part is broken. I think this is reason why nothing is displayed. 

This data comes from fetch BODY[HEADER], fetch BODYSTRUCTURE, and fetch BODY[1], when Offline-use=Off and mime_parts_on_demand=true is applied.
In contrast to it, fetch BODY[] is used by Tb when Offline-use=On and View/Message Source.
And following is seen in your IMAP log.
> 2280[b06b100]: f475400:imap.edv-froehlich.de:S-eBay:SendData: 15 UID fetch 635610 (BODYSTRUCTURE)
> 2280[b06b100]: f475400:imap.edv-froehlich.de:S-eBay:CreateNewLineFromSocket: * 9 FETCH (
>(snip)
> "mixed" ("boundary" "=?us-ascii?q?----=3D=5FNextPart=5F94911620283019421195478?= =?us-ascii?q?1?=") NIL NIL NIL))

It looks comibination of "quoted-printable encoding" and "RFC2047 encoding". 
Do you see your problem on mail of non-quoted-printable text/html part?
(Content-Transfer-Encoding: 7bits or 8bits, and Content-Transfer-Encoding: base64)
Is your IMAP server correctly configured?

Comment 11

[David Froehlich]

> I think this is reason why nothing is displayed. 

I can see, that the boundary is _different_ (plain vs. RFC 2047). But can you actually say that it is _broken_? Shouldn't decoding the quoted printable string result in what is specified as boundary="..." above? I just used an online RFC 2047 decoder which gave me the same result:

--=?us-ascii?q?----=3D=5FNextPart=5F94911620283019421195478?= =?us-ascii?q?1?=

equals

------=_NextPart_949116202830194211954781

So everything should be fine - or not?

I can also confirm that I can reproduce this behaviour only with RFC2047 and QP encoding. When calling "fetch bodystructure" dbmail tries to encode all strings with RFC2047 and QP as soon as there is at least one equal-sign (=) or underscore (_). With other mails which only use alphanumeric chars and minus signs (-) I do not have this problem; probably because there is not RFC2047 and QP encoding done.

Unfortunately I cannot configure this behaviour on my mailserver and I could not find the specific line in the sourcecode of dbmail which invokes the encoding.

We can summarize that this only happens with RFC2047 QP encoded boundaries. This encoding is not really necessary but according to RFC2047 it should be okay. So might this indicate a bug in TB which prevents TB from properly decoding the RFC2047 string?

I have also attached some lines from the debug log of dbmail. It shows that dbmail internally handles the mail (and the boundary) without QP encoding. The QP encoding seems to be added just right before the bodystructure response is built. Unfortunately I could not isolate the specific line yet. From what I have seen in the sourcecode, the RFC2047 encoding is done quite generally (i.e. for all parts of the bodystructure response). So there is no easy way to skip RFC2047 just for the boundary strings.

Comment 12

[David Froehlich]

Attachment: dbmail mailserver debug log

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to David Froehlich from comment #11)
> So might this indicate a bug in TB which prevents TB from properly decoding the RFC2047 string?

How can it be Tb's bug?
How can the fetch BODYSTRUCTURE response by your IMAP server be proper?
Please read http://tools.ietf.org/html/rfc2046#section-5.1.1 for syntax of "boundary delimiter" and definition of "boundary".
And, please read http://tools.ietf.org/html/rfc2047 for "where RFC2047 encoding can be applied".

Comment 14

[David Froehlich]

That's the interesting question I could not answer for sure. Is it allowed to use RFC2047 encoding within the BODYSTRUCTURE response or not? After reading quite a lot I agree with you that it is not needed to encode the boundary string at all because it is only allowed to use certain 7-bit chars; so no need for encoding. I guess I will have to force dbmail not to encode the boundary string in the BODYSTRUCTURE response somehow. Do you know whether RFC2047 is completely forbidden in the BODYSTRUCTURE response? Other content like filenames for instance might contain non 7bit chars so I guess we need some kind of encoding here. However the biggest problem at the moment is the improperly formed boundary which prevents TB from showing the message at all. So I will try to find a way and how to fix this on the mail server. I will post a note in a few days with my results and whether this actually was the one and only cause.

Comment 15

[David Froehlich]

It works! Thanks a million for your hint regarding the improperly formatted boundary string in the BODYSTRUCTURE response. I guess I would have not found this little but important detail wihtin the large debug logfile on my own.

After crawling through the dbmail sourcecode I could finally isolate the function which calls g_mime_utils_header_encode_text from the gmime library. Unfortunately this call is placed in a generic function to retrieve a parameter list from a hash map. Since this function is called from various other functions I added a conditional clause which skips the RFC2047 encoding when processing a key named "boundary" from a hashmap. I hope this avoids any side effects. At least the result of my change is that the BODYSTRUCTURE response now contains the boundary string as it is without any further encoding. And now thunderbird is able to display all mails which did not work before - even when using the default behaviour i.e. mime_parts_on_demand turned on.

Refer dbmail bug 996: http://www.dbmail.org/mantis/view.php?id=996

So we can note that it indeed was a server bug. I have already filed a bug report in dbmail's mantis including my current workaround. I hope the developer can transfer my workaround to a real patch making dbmail more RFC compliant. My scenario with thunderbird, dbmail, lightning and eGroupWare now works just perfect.

I will now mark this bug report as resolved as it actually was no TB bug.

Comment 16

[WADA:World Anti-bad-Duping Agency]

I hope dbmail's bug will be solved soon.
David Froehlich, remove test account of your IMAP server what I know, or at least change password, please.

Comment 17

[David Froehlich]

I completely removed the test account yesterday so it shouldn't be accessible anymore.