1461565 - Mail-list digest display and attachments list are flawed, when no offline store configured, and attachments are not inline

Status: NEW

(MailNews Core :: Backend, defect)

Description

[gene smith]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID: 20180323154952

Steps to reproduce:

Setup thunderbird imap folder with no offline storage and with attachments not displayed inline. Import this email file (drag/drop) into the folder: https://bugzilla.mozilla.org/attachment.cgi?id=8974557 and try to read this email.
This email consists of a top level body and 6 emails that are attachments and one signature attachment.


Actual results:

After some perfectly readable information, the email body shows 2 lines of improperly decoded "gibberish".
After revealing the attachment links at the bottom, each of the 6 attachment (should be 7) has the same name ForwardedMessage.eml instead of the email subject. When attachments opened, some of the resulting email displays show improperly decoded gibberish characters. The last attachment shows header info and also contains two "will be downloaded on demand" strings. So effectively, only 5 of the expect 6 email attachments are readable.


Expected results:

With no offline store and attachments not inline, should cleanly display email body and list the correct number of attachments (7) and list the attachments with file names corresponding to the email subject. This email displays correctly when the folder is setup to have offline store and when the attachments are displayed inline. 
The last (7th) attachment is a signature.asc file and not an email and with offline store, can be opened properly as text.
There should be no gibberish characters or header info in the main email body or the displayed email attachments when no offline store and attachments as links.

Note: The example eml file linked to above comes from bug 1460104.

Comment 1

[gene smith]

I'm not sure this is actually "dataloss" since the email can still be read for the most part if you set attachments to display inline, although there is still some gibberish and the last eml attachment shows header data for the eml and mime header for the signature data, but the email text is there. Only get 100% correct display if you have offline storage it seems.

I want to ask the reporter of bug 1460104 (Bill M.) to see if he also sees what I am seeing. He doesn't use offline storage but typically displays emails with attachments inline. He provided the sample eml file that I have been testing with. Will set NI flag for him.

Comment 2

[gene smith]

Bill, To test this I would need for you to set those "prefs" I had you set back to default. But since on yahoo it will totally break this email, probably no need to test it (unless you have a non-yahoo server you could try this on). Thanks anyhow.

Comment 3

[Wayne Mery (:wsmwk)]

"corrupted" mislead me

Comment 4

[Bill Mattison]

I'd like to see the bad display.  Could you either e-mail a screen capture to me or attach a screen capture to this bug?  I looked at a few more Fedora Users list digests, and I see that sometimes attachments have attachments (recursion?!).  Could that be a factor in this bug?

Comment 5

[gene smith]

Attachment: Showing corrupted chars in email

This shows what I mean by "gibberish" in my description. To see this you must disable offline store (right click folder properties under sync tab) then under 1st property tab repair the folder or shutdown tb, delete the mbox file for the folder and restart tb. Also, the attachment mode is set to not inline.

Comment 6

[gene smith]

There are actually two gibberish lines in the email, sorry the 2nd one got cut off. Also, I don't think attachments to attachments is the problem.

Comment 7

[Bill Mattison]

(responding to Comment 5)
> To see this you
> must disable offline store (right click folder properties under sync tab)
> then under 1st property tab repair the folder or shutdown tb, delete the
> mbox file for the folder and restart tb.

In my Thunderbird, offline storage is already disabled for all e-mail accounts and all folders.  So I can completely skip this part.  Correct?

> Also, the attachment mode is set to
> not inline.

I:
1. unchecked View -> Display Attachments Inline;
2. exited Thunderbird; and
3. then clicked the link to the message in this bug.
That launched Thunderbird and displayed the message.  I saw nothing wrong.  I exited Thunderbird.  I then
4. double clicked my private copy of the that ".eml" file in my "Files" file browser (in Gnome, in Fedora).  Exact same result.

Why are we seeing different results?

(responding to Comment 6)
> I don't think attachments to attachments is the problem.
If you need a Fedora Users list digest e-mail without nested attachments so you can actually try it, let me know.  I can provide one.  I can also provide examples in which nested attachments are not in the last message.  I have one where the ".asc" is not attached to the last message in the digest.  That's good evidence that in the digest you have, the ".asc" attachment is attached to the last message in the digest rather than an attachment to the digest as a whole.  I don't know if that's relevant.

Comment 8

[gene smith]

(In reply to Bill Mattison from comment #7)
> (responding to Comment 5)
> > To see this you
> > must disable offline store (right click folder properties under sync tab)
> > then under 1st property tab repair the folder or shutdown tb, delete the
> > mbox file for the folder and restart tb.
> 
> In my Thunderbird, offline storage is already disabled for all e-mail
> accounts and all folders.  So I can completely skip this part.  Correct?

Yes, should be OK. However, it sounds like you are either clicking on the link for the message in bugzilla or clicking on the link in your file manager to open the email. I see mostly ok when I do that too. To really test it you need to open the email from a real imap account. If it's not still on your yahoo account, you can drag the eml file from your file manager to the appropriate imap folder in tb and drop it in. This will do an imap "append" of the email to the folder (mailbox in imap parlance) so it should be visible in the target folder of the server. Then try opening it from the imap folder in tb once it is in place on the server.

I'm assuming the drag/drop of the eml file works in gnome (it works for me in kde). Also, I haven't tried it yet on my yahoo account, just on two other imap accounts.

Comment 9

[Bill Mattison]

(In reply to gene smith from comment #8)

> [...] it sounds like you are either clicking on the
> link for the message in bugzilla or clicking on the link in your file
> manager to open the email.

Correct - both - two separate tests.

> [...] To really
> test it you need to open the email from a real imap account. If it's not
> still on your yahoo account, you can drag the eml file from your file
> manager to the appropriate imap folder in tb and drop it in. [...] Then try opening it
> from the imap folder in tb once it is in place on the server.

ok.  I
1. dragged the ".eml" file from the "Files" manager display into a folder of the yahoo account used for Mozilla Bugzilla communication;
2. unchecked View -> Display Attachments Inline;
3. exited Thunderbird;
4. re-launched Thunderbird and logged in to the appropriate e-mail account; and
5. went into the appropriate folder and selected the test message.
I saw nothing wrong.

All my accounts in Thunderbird are yahoo imap accounts, so I can't test non-imap or non-yahoo.

Comment 10

[gene smith]

Sorry, as I mentioned in comment 2 above, putting this eml on yahoo won't test it since in order to get past the INVALID error you have to either enable offline storage or set the special prefs described in bug 1460104. These prefs cause the whole message to be stored and accessed from cache (sort of a temporary memory based offline store) so the email will appear OK.

I dragged the message to a gmail folder and with no offline store and attachments not inline, I see the same gibberish as in the png attached above. Also, the email attachments refuse to open and report that they are empty. Also, a FETCHed error pops up in tb when the containing gmail folder is accessed. So, gmail is actually worse than the two server I previously tried, openwave and dovecot, and on par with yahoo.

I'm thinking that this bug is actually just a duplicate or just another aspect of bug 1460104. Tb is probably confused by the bodystructure of the message.

Comment 11

[Bill Mattison]

(In reply to gene smith from comment #10)
> Sorry, as I mentioned in comment 2 above, putting this eml on yahoo won't
> test it since in order to get past the INVALID error you have to either
> enable offline storage or set the special prefs described in bug 1460104.
> These prefs cause the whole message to be stored and accessed from cache
> (sort of a temporary memory based offline store) so the email will appear OK.

I'd like to try it.  To first check that I understand you correctly, I need to
* set mail.server.default.fetch_by_chunks back to true, and
* set mail.server.default.mime_parts_on_demand back to true.
I should leave View -> Display Attachments Inline unchecked, and leave offline storage disabled.
Then exit, re-launch, and log in.  The gibberish should show up when I display the message.  Have I got it right?

Comment 12

[gene smith]

The procedure is right but since the example eml is the one you said in bug 1460104 would not display at all and produces a pop-up error [UNAVAILABLE], I don't think there is any use trying it on your yahoo server account, but go ahead and see what happens. I get errors and no display when I try it on my yahoo and gmail.

Comment 13

[Bill Mattison]

I did the experiment.
* 12 messages displayed fine; only one had a ".asc" attachment, and it was not the last in the message.
* 9 messages appeared empty and had pop-up errors; they all had ".asc" attachments, though not necessarily last in the message.  The message attached to the other bug was in this category.
* 2 messages showed gibberish; they both had ".asc" attachments, one at the end and one not at the end; the gibberish in both cases is where the ".asc" would show if it were displayed in-line.

I hope that helps.

By the way, the "ForwardedMessage.eml" may be the result of the way the Fedora infrastructure is putting digest messages together.  There is a problem with postings submitted through yahoo e-mail and/or the "fedora HYPERKITTY", and these "ForwardedMessage.eml" occurrences may be related to some patch to get around the problem.

Comment 14

[gene smith]

The "ForwardedMessage.eml" thing seems to be caused by tb. It is reported in the 15 y.o. Bug 227720. I also found that you can clear the "gibberish" if you change to view attachments inline and then return to not inline. This only fixes the one message you are looking at and probably defeats the purpose of not inline since all the attachments get downloaded.