244741 - sometimes when I select a message I cant see the body of the imap message

Status: RESOLVED DUPLICATE of bug 246415

(MailNews Core :: Networking: IMAP, defect)

Description

[Henrik Gemal]

I'm using nightly build and see some:
"failed creating protocol instance"
in my IMAP logs.

I'm also seeing that sometimes when I select a message I cant see the body of
the message. I only see the header fields. It seems like Mozilla chokes on the
message.

The problem is that if Mozilla chokes on a message it will still put it in some
kind of IMAP cache so selecing another mesage and reselecting it wont work.

I can mail you the IMAP log if you want. I dont want to attach it here, because
of privacy.

Comment 1

[Keith Hopkins]

(In reply to comment #0)
> I'm using nightly build and see some:
> "failed creating protocol instance"
> in my IMAP logs.

I'm seeing this also on a MS-Exchange server via IMAP during filtering.
(researching for 233041/253368/235047). 

Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7226.0

Mozilla 1.7
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040616

Comment 2

[Matej Cepl]

Attachment: Mozilla Thunderbird log with the thing happening

Yeap, the same here. It is official version of Mozilla Thunderbird 0.7.3 with
Windows XP (Home editioin if it matters).

Comment 3

[David :Bienvenu]

"failed creating protocol instance" is not a problem in and of itself - it just
means that we had to wait for a url to finish before we could re-use a
connection. I understand that Henrik is seeing an actual bug, but are either of
you seeing a real problem, or just that message in the log?

Henrik, have you tried a recent trunk build? There was a problem where empty
messages could get in the memory cache, but I believe I fixed that... 

Comment 4

[Matej Cepl]

Well, I see a real _HUGE_ problem. Even when trying five times or more, Thunderbird today 
morning did not show ANY messages in any of my IMAP folders (BTW, whenever I switched off TB 
with Ctrl-Q I could still see a process thunderbird.exe in Task Manager and unless I kill it there I 
could not open new instance of TB). After some five unsuccessfull attempts (there is nothing more 
frustrating than when I couldn't read my emails in the morning after breakfast -- kind of email 
addiction, you see), I gave up and read some emails through telnet to my university computer.

Matej

Comment 5

[David :Bienvenu]

Matej, could you generate a protocol log of that happening? Are you an offline
user, who does things offline and then goes back online?

Comment 6

[Matej Cepl]

Yes I am and I have already attached my log to my comment #2

Comment 7

[David :Bienvenu]

Matej, your log in comment 2 shows nothing wrong (as I said earlier, "failed
creating protocol instance" is not an error). I meant a log of what you
described in comment #4

Comment 8

[heslo04]

It happens sometimes on Windows2k, too... and sometimes while saving
attachement, they have 0 size. Repeated action corrects the problem, but it
happens quite often.

Comment 9

[heslo04]

It happens sometimes on Windows2k, too... and sometimes while saving
attachement, they have 0 size. Repeated action corrects the problem, but it
happens quite often.

Comment 10

[Phil Lacy]

Attachment: sample email

I removed header parts and text parts for the most part.

The message doesn't display as plain or html in latest shredder b4pre.

IMAP gmail.

I can provide other particulars of email headers if needed, but the removed items seemed inconsequential.

Comment 11

[Phil Lacy]

(In reply to comment #10)
> Created an attachment (id=390507) [details]
however if I copy to local store, it will display there, just not the imap folder.

Comment 12

[Phil Lacy]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5pre) Gecko/20091024 Shredder/3.0pre

Not sure but seems to be large messages with attachments. The message shows the icons for the attachments but spins for infinity with 'Loading message...' and shows blank content area for message. Clicking out and back in the message removes the status note but still shows blank message with attachment footer section still ok.
with synchronize set to store local folders it is ok but once the synchronize is off and the msg file is deleted it does same. Same with rebuilding index.

Comment 13

[Phil Lacy]

Bug 524852 may be related.

Comment 14

[Phil Lacy]

some more clues.

I have one persistent offending email that shows blank body and then lists in the attachment footer. icons for 2 of it's embedded images a horiz space and the two attachments, When I edit as new and save the new to imap draft, the original now displays ok.

consist of part 1.1 1.2 1.3 which should be the images and part 2 and 3 which should be the attachments.

The source view shows everything in email even when the email display is messed up so it is downloading just not cached right.

clearing the cache returns to the blank messed up display version.

Comment 15

[Phil Lacy]

David if you can give me some me some more breakpoints to check.
I get this in the logs but I don't know where to check to see if the body.peek data is being picked up or not.

1352[4507e90]: ReadNextLine [stream=45c9fc0 nb=21 needmore=0]
1352[4507e90]: 45c08d8:imap.gmail.com:S-INBOX:CreateNewLineFromSocket:  BODY[2.MIME] {357}
1352[4507e90]: ReadNextLine [stream=45c9fc0 nb=78 needmore=0]

Comment 16

[David :Bienvenu]

sounds to me more like the message is mal-formed and not messed up in the offline store - if you copy it and another message in the same imap folder to a local folder (copying two messages makes us bypass the offline store & caches), and look at it in the local folder, does it display correctly?

Comment 17

[Phil Lacy]

I can copy the single file to a local folder and it will display ok. Copy 2 messages to local and also displays ok.
clear cache and local files still ok and imap file still bad.
I was in process of checking message structure. I could use a link or brief explanation how the boundries define the bodystructure parts

Comment 18

[Phil Lacy]

(In reply to comment #16)
> sounds to me more like the message is mal-formed and not messed up in the

ok so if it is copied then the whole message is sent to local folder as is and we are decoding it internally. Whereas when fetched on IMAP we must be asking for a part and gmail must be interpetting it as a different part? Again, it would help to see what we get when we send the fetch command to gmail, but I don't know where that code resides.

Comment 19

[David :Bienvenu]

If it's the protocol we're sending, you should be able to get a protocol log of a bad session, right? The fetch commands are in nsImapProtocol.cpp

Comment 20

[Phil Lacy]

something is stopping a good download. When I check 'display attachments inline' it downloads the message again and it displays ok. but the download spinner in status bar keeps spinning. I can't seem to get it to break when the display attachments inline is checked.
Also this message has two different 'image.jpg' and one is not referenced in the message even though it is a subpart to part 1 which is the body.

Comment 21

[Phil Lacy]

(In reply to comment #19)
> If it's the protocol we're sending, you should be able to get a protocol log of
> a bad session, right? The fetch commands are in nsImapProtocol.cpp
I got a pretty fair outline of the bad messge, the log and my manual parse of the message which seems to match our fetch of body parts.

Comment 22

[Phil Lacy]

when in display attachments inline is off and the display is wrong. I see us not downloading the sub-parts that define the message display.
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Multipart: 1.1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-XHeader: 1.1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1.1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Filling: 1.1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Leaf: 1.2
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-XHeader: 1.2
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1.2
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Filling: 1.2
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 1
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Leaf: 1.3
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-XHeader: 1.3
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1.3
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Filling: 1.3
2752[4b2b910]: 4b22568:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary-Last: 1

Whereas with DAInline on we are downloading and generating the display stuff. 1.1.1 and 1.1.2
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 0
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Multipart: 1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Multipart: 1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Leaf: 1.1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1.1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Part-Inline: 1.1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SendData: 17 UID fetch 35 (BODY.PEEK[1.1.1])
<snip>
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Boundary: 1.1
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Leaf: 1.1.2
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-MIMEHeader: 1.1.2
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SHELL:GENERATE-Part-Inline: 1.1.2
2204[4146d28]: 413c9d8:imap.gmail.com:S-INBOX:SendData: 18 UID fetch 35 (BODY.PEEK[1.1.2])
etc.

Comment 23

[David :Bienvenu]

nsImapBodyShell.cpp might be involved in figuring out which parts to download. You could look at that code to figure out why we're not fetching the right parts.

Comment 24

[Phil Lacy]

Attachment: causes multipart section to fetch inline when grandparent is not message

I am hacking this per my comment in the patch. But I don't know the condition or email case that the original code is trying check. For 1.1 the grandparent wouldn't be 0 but that is the partnumber it is checking and what seems to be what is returned from imap server. There may be a parsing error somewhere or maybe this is correct. I haven't looked where that section is that typed as IMAP_BODY_MESSAGE_RFC822.

Comment 25

[Phil Lacy]

Comment on attachment 409549 [details] [diff] [review]
causes multipart section to fetch inline when grandparent is not message

need to correct some experimenting

Comment 26

[Phil Lacy]

Comment on attachment 409549 [details] [diff] [review]
causes multipart section to fetch inline when grandparent is not message


>+    if (GetType() == IMAP_BODY_MULTIPART)
>+      return PR_TRUE;
Originally I put this part in to hack it to work but forgot to take it out. However I need to check further because now the attachments are displaying ok but there is no message body so the 1.1.1 and 1.1.2 are not being fetched in the patch without this code.

Comment 27

[Phil Lacy]

Attachment: hacks this bug but not final due to unknown sideeffects

This patch is more informative than suggestive. The hack causes this bug to go away but it is unknown regarding other email formats that need the check of the grandparent.
See comments in code and other patch regarding findings of cause of this bug.

Comment 28

[Phil Lacy]

Comment on attachment 409562 [details] [diff] [review]
hacks this bug but not final due to unknown sideeffects

in trying to help clarify it further

>     // If "Show Attachments as Links" is on, and
>     // the parent of this multipart is not a message,
>     // then it's not inline.

this comment doesn't mention the other test and should have read:

     // If "Show Attachments as Links" is on, and
     // the parent of this multipart is not a message,
+    // and if the parent is multipart and grandparent is 
+    // not a message
     // then it's not inline.

which is the problem in my case.
0 message
1 multipart
1.1 will not be inline and therefore misses 1.1.1 and 1.1.2

Comment 29

[Phil Lacy]

http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=nsIMAPBodyShell.cpp&branch=1.50&root=/cvsroot&subdir=mozilla/mailnews/imap/src&command=DIFF_FRAMESET&rev1=1.17&rev2=1.18

There's where I see it change to present code but not sure why except notes re. working on loading multiparts on demand.

Here's the referenced bug.
https://bugzilla.mozilla.org/show_bug.cgi?id=32767

Comment 30

[Phil Lacy]

Attachment: fix this problem and appears to satisfy original code intent

My other comments are obsolete. I set out my problem and intent of original code in comments. Also reformatted the tests to expand them a little. These can be removed if the patch is ok, or not, or made into bug comment.

This takes care of my problem with MS Exchange email. Any other test-emails we have to check this code?

Comment 31

[Phil Lacy]

Comment on attachment 413783 [details] [diff] [review]
fix this problem and appears to satisfy original code intent

WIP

Comment 32

[Phil Lacy]

Attachment: alternative text needs fetching

this fetches multipart/alternative and then is able to fetch the plain or html text subtypes depending on user-selected body view.

This needs another bug to pick up the plain text for non supported text subtypes when html view is selected. Hopefully we can land this patch first to correct a few more of the download-on-demand errors.

There may be an alternative of plain and rich-text and we have html selected in which case we need to pick up the plain.

Comment 33

[David :Bienvenu]

I tried tweaking your test message a bit so that the last alternative part was text/html, just so I could see something in the message body, and even with your patch, I got "this body part will be downloaded". So I don't think the patch is quite right. Either that or I messed up my tests somehow...

Comment 34

[Phil Lacy]

(In reply to comment #33)

I don't get that message anymore, I do have Bug 246415 applied but I don't think that affects your problem.

When testing this I remember having to clear out the cache.  I'm pretty sure the only sure way I did that was to delete the cache files here:

%userprofile%\Local Settings\Application Data\Thunderbird\Profiles\[profile]\Cache

Comment 35

[David :Bienvenu]

Comment on attachment 417328 [details] [diff] [review]
alternative text needs fetching

The else here is not needed:

         return PR_TRUE;	// we're downloading it inline
       }
+      else if ((!PL_strcasecmp(m_parentPart->GetBodySubType(), "alternative") ||

because we returned above.

this:

+        if ((!PL_strcasecmp(m_bodySubType, "plain") && preferPlainText)
+            || (!PL_strcasecmp(m_bodySubType, "html") && !preferPlainText))
+        {
+          return PR_TRUE;
+        }
+        else
+          return PR_FALSE;

can just be return of the if clause.

again, the last else is not needed:

+        else
+          return PR_FALSE;
+      }
       else

because we returned before.

Other than those nits, this does seem to be an improvement, thx!

Comment 36

[Phil Lacy]

Attachment: does some reformatting and repair nits.

I got rid of the else's but also did some reformatting hopefully to make it clearer.

Comment 37

[Phil Lacy]

Comment on attachment 421758 [details] [diff] [review]
does some reformatting and repair nits.

needs work,
David, why can't we have a member variable for content-disposition to see if it is an attachment and not fetch is if not set to download?

Comment 38

[David :Bienvenu]

Attachment: message not loading correctly for me

Phil, I'm having trouble with this message...

Comment 39

[Phil Lacy]

good message I had one like it and it seemed to go lost in my experimenting.

The reason it's good is because we are fetching the proper parts 1.1.1 when plain is set and 1.1.2 when html. And we do fetch the data.

I was tracking it down and glad I got a good email for testing now.  It's in the libmime(I'm guessing for now).

The Bodyshell seems to be working good though. I got a log. I'll post it.

Comment 40

[Phil Lacy]

Attachment: log of message

if I put valid html, I get the html then a divider line and then "this part downloaded on demand"

Without the html fix up I get the divider line and blank. But after I viewed the source apparently it cached the text part and now it shows up below the divider line.  But the part is not being fetched in the log.

Comment 41

[David :Bienvenu]

Scary that the uploaded file wasn't right; I know it was right on my system. I was worried because I was pretty sure that it worked with the previous versions of your patch, but not the latest ones...

Comment 42

[David :Bienvenu]

Comment on attachment 421758 [details] [diff] [review]
does some reformatting and repair nits.

minusing because of problem with test message (if I understood you correctly, you were able to reproduce the issue, despite the munged attached message)

Comment 43

[Phil Lacy]

Attachment: Your test eml but formatted correctly

Comment 44

[Phil Lacy]

Not exactly. When formatted correctly the bug is fixed here. That is the body is displayed html or plain as selected.

The Download on demand message is for a part that mime is inserting inline. I'll upload a good message and let you see what I mean. I don't know if that is correct display but it would be different bug if not. It's like this:
"
html or plain text message.

---------------------------------------------------------
This part will be downloaded on demand
"

Comment 45

[Phil Lacy]

note: bug is not a dupe but patch is duped to bug 246415