770888 - Imap: For messages that were filtered/moved, error message when opening attachments: "The attachment appears to be empty", but "repair folder" fixes it

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[yoman]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11

Steps to reproduce:

Repeat procedure:
- use existing gmail account, created back in Thunderbird 3.0 (already has subfolders and existing message filtering rules)
- craete new subfolder in inbox (12 char name, alpha char only)
- create new message filter that moves messages from Inbox to the new folder
- run message filter (about 7 messages were moved, 3 had attachments on them)
- goto new folder
- read a mail message with multiple attachments and double click attachment -> messagebox: attachment appears to be empty


Actual results:

Workaround: clicking 'Repair' in folder properties fixed the problem.
Debug info: i have not noticed this issue in previous versions (there are many subfolders in my Inbox)
Debug info: issue is probably related to the nature if IMAP protocol?


Expected results:

'Open attachment' should have appeared

Comment 1

[Barry Ralphs]

I'm having a similar issue, but it's no when using filters.
We have our main Gmail account where our emails arrive.
We then have an internal mail server with a Public (shared) namespace.
When a project related email arrives, we drag\move that email into the "Public" folder, so everyone in the office can access it. Now sometimes a few weeks later, if you go back to a message in the "Public" folder & click on an attachment you get "attachment appears to be empty...". If we do repair folder it fixes the issue (for awhile, but comes back some time later) or if we drag the message back into our Gmail Inbox, we can open the attachment from there. We'd really like this one fixed, it's becoming a pain to manage. I'm having to "repair" folders almost every day for users now.
Running TB 14.0

Comment 2

[Igor Vitorac]

- I have the same problem. I am using the latest TB ESR 17.0.3
- I did not have this problem with TB ESR 10.x.
- I have also noticed that this issue was present on TB 16, and maybe even earlier. This issue was one of the reasons why I stayed on ESR version 10


Description:
- I have one IMAP account connected to "CommuniGate Pro"
- I have second IMAP account connected to Zimbra.
- When I move message from CommuniGate Pro Inbox to Zimbra Inbox (i.e. folders between accounts), I am not able to open attachment anymore at the destination folder.
- I am NOT keeping messages locally (for both accounts) i.e. Synchronization & storage > Keep messages for this account on this computer

- if I do Properties and Repair folder on the destination account i.e. Inbox, attachment is again available.
Seems that folder index ( .msf or whatever) is not immediately updated when the message is moved to that folder.

- restarting TB without Repairing the folder does not help.

Regards,
Igor

Comment 3

[Travis]

Same problem Thunderbird 17.0.8
Using Filters to file emails.  Destination folder was the one that needed to be repaired to view the attachments.  Now it seems to be happening in other folders.

Comment 4

[WADA:World Anti-bad-Duping Agency]

As written in bug 1000589 and as seen in bugs listed in dependency for that meta bug, there are two know patterns in "broken mail data in offline-store file".
(A)   (a) 2048 bytes from top of mail y previewText.
    + (c) Entire mail data, by quick viewing of new mail, or by auto-sync.
(B)   (b) Mail data after download with mime part on demaand.
          (data of non-displayable parts is not fetched)
    + (c) Entire mail data, by quick viewing of new mail, or by auto-sync.
Same as (A) or (B), or (a) only? or (b) only?
Or different from (A)/(a)/(B)/(b)?

Comment 5

[Mateus]

I have the same problem in Linux Client, version 31.1.
The problem occours when a filter is applied.

Comment 6

[Joshua Nixon]

I encountered this issue only after upgrading my computer to OS X 10.10 Yosemite - Have been using Thunderbird since version 1.0 and have never seen this behavior on my own computer before.

Webmail interface showed attachments intact, Tb told me they were empty and would not allow me to save them.

Wondering if this is related in any way to the ongoing performance issues Tb is having with IMAP (repeated timeout during send and Gmail compatibility especially) - if the program is not communicating with the server correctly, it may erroneously be assuming the attachment is missing or empty due to timeout?

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 ID:20141012121702 CSet: 79a6df4bab3c

Comment 7

[Wayne Mery (:wsmwk)]

so this is filters or imap

Comment 8

[Mark Aitchison]

I had this problem with Thunderbird 31.4.0 under Linux earlier this month; I have had the problem before (going back a few years) - always with IMAP, always had filters in operation (not sure if that was what did it), where empty messages (no subject/date/body/etc) appear at the start of the folder, except this time I didn't see them (probably were there, and fixed up when I restarted Tb) because the problem I got was that listing the message filter log showed nothing after 02/02/2015 (despite plenty of records in the log file after that) and when I looked into this I found where the logging stopped was in the middle of 79 lines like this:

Applied filter "Other Mailinglists" to message from - at 01/01/70 12:00:00 tagged

Applied filter "Other Mailinglists" to message from - at 01/01/70 12:00:00 tagged

The real date would have been around 02/02/2015 (looking at surrounding messages)

I think (only think) I didn't actually loose any emails because of this. 

These are the clues, in case they help:

1. I had been running Tb on my laptop (under windows) and *might* have had both Linux server and Windows laptop Tb's going at the same time when it happened;
2. Tb has been gobbling up bandwidth at a dreadful rate by re-downloading the same messages on my laptop (when downloading was expensive, so I took notice when it happened, and switched to offline mode quite often when I wasn't actively reading email, sometimes when it was in the middle of moving messages). I have followed the procedure to fix up corrupted index files but the massive repeated downloading starts again after a few days. This was only something I noticed on the laptop (might have happened on the Linux server too, but I didn't see it)
3. Always no problem seeing the folders from webmail at my ISP.
4. I have had really strange repeated filter log messages, saying the same message has been moved to another folder (as it should have been moved) yet how could it be moved again when that filter isn't supposed to apply to messages in the folder to which it was moved??)
5. I am guessing something simple I did (like restarting tb) fixed up those empty messages.

Mark

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to Mark Aitchison from comment #8)
> Applied filter "Other Mailinglists" to message from - at 01/01/70 12:00:00 tagged
> Applied filter "Other Mailinglists" to message from - at 01/01/70 12:00:00 tagged

It's Bug 762318, isn't it?
I don't think cause of that bug is "flawi n code of filter logging" only. And I guess filter logging is a victim of other problem.
Sorry but I don't know what kind of problem occurs when phenomenon of 'Bug 762318 happens.

A partiqularity of Laptop PC : For power saving, power save mode(clock speed change at a Core), sleep/wake-up, supend/resume, frequently occurs. When sleep, power supply to network card may be stopped. If so, upon each wakeup, connection loss/connection recovery happens at all imap connections. This may cause problems while fetching/filtering new mail.

If problem disappears by "Repair Folder", it indicates that "data held in Offline-store file" is not correct.
What message source is shown when problem occurs?

Comment 10

[Mark Aitchison]

(In reply to WADA from comment #9)

> It's Bug 762318, isn't it?

I'm not sure - these log messages started and stoppped again by themselves, it seems, with no change in teh before/after junk classification setting or anything else. The messages are consistent with empty from, subject, etc fields which is what I saw last year (and before) on several occassions in folders (there were something like 3 header lines, but cannot recall details). This time I only noticed something odd over a week later, and then only when I looked at the logs. Maybe it is the (solved) bug 762318 or 935934; not sure; but maybe the problem yoman reported is related to thsi (and maybe there are similar symptoms from other reasons, and maybe the problem Barry Ralphs (Comment 1) had are something else? 

I suggest a bit of code be added at various places - a sanity check to spot totally invalid message entries like this when moving, logging, looking at an IMAP folder the first time after starting up, etc and log/report helpful information as soon as possible rather than leave users only crumbs of clues later?

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Mark Aitchison from comment #10)
> I suggest a bit of code be added at various places - a sanity check to spot totally invalid message entries like this when moving, logging,
> looking at an IMAP folder the first time after starting up, etc and log/report helpful information as soon as possible
> rather than leave users only crumbs of clues later?

Improvemenr is in progress in bug 854172, bug 1119529 etc.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to Mark Aitchison from comment #10)
> Maybe it is the (solved) bug 762318 or 935934; not sure; but maybe the problem yoman reported is related to thsi
> (and maybe there are similar symptoms from other reasons, and maybe the problem Barry Ralphs (Comment 1) had are something else? 

Do you use POP3 account?
If so, both bug can occur at same time.
   Bug 935934(Bug 762318 ) on POP3, when multiple messages are moved by filter in single POP3 download.
   This bug or something else on IMAP.
For IMAP, body condition may be a workaround.
   As know by Bug 971401, if "body contains" is used in filter rule, Tb issues "uid fetch M:N BODY.PEEK[]" prior to filtering.
   i.e. Even when IMAP, entire mail data is downloaded upon new mail detection, as done on POP3.
"Slowness in filering of new messages" is far better than "broken message data in Offline-store" which forces "manual Repair Folder" to user.

Comment 13

[WADA:World Anti-bad-Duping Agency]

FYI.
Cause of known "top 2048 bytes of mail + entire mail data" is interfere of Tb's new mail fetch by "fetch of top 2048 bytes by Biff for previewText display in new mail alert". "uid fetch N body.peek[TEXT].2048" is requested by Biff only. "uid fetch M:N BODY.PEEK[] prior to message filtering" is surely executed before "fetch of top 2048 bytes by Biff for previewText display in new mail alert". So, interefere by "uid fetch N body.peek[TEXT].2048 by Biff" can't happen, if body condition is used.

Note:
Killing  "uid fetch N body.peek[TEXT].2048 by Biff" is not done by "stopping new mail alert".(unchecking "Sow an alert"). Following is needed.
    "Cutomize..." button of  "Show an alert" -> Uncheck "Message Preview Text"

Comment 14

[WADA:World Anti-bad-Duping Agency]

FYI.

In bug reports for thiss kind of issue, I frequently ask about "message source of mal on which problem happened".
But "message source" won't be provided by propblem reporters in almost all cases.
In pretty rare valuable  "message source of mal on which problem happened" provided by problem reporters, I could observe following patterns.
(a) top 2048 bytes of mail body + entire mail data 
(b) less than 2048 bytes mail body data + entire mail data
(c) absolutely different pattern from (a)/(b).

(b) was "mail body size is less than 2048 bytes" case.
Ratio of (a) : (b) : (c) was approximately 8:1:1. Number of provided mail data : 1< (a) + (b) + (c) < 10..

If pattern (a)/(b) phenomenon is pretty simple.
  Data in Offline-store file
     message body line #1      A
     message body line #2      |
                                      |      |
     message body line #N    top  2048 bytes of mail body
                                      |      |
     message body line #P     V
     mwessage header lines of the mail
     Null line (delimter of messaage header lines and messge payload lines)
     message body lines start from "message body line #1"

(Case-1) text/html or text/plain mail
(1-1) If Null line is contained in "top 2048 bytes of mail body", "message header lines of this broken mail data" ends at the null line.
     Becuse no correct message header is usually not contained in message body,
     it's treated as "text/plain" mail, and "data after the null line" is shown as message body of the "text/plain mail" by Tb.
(1-2) If Null line is not contained in "top 2048 bytes of mail body", these are merely wrong message header lines, so is simply ignored.
     "Broken data"  is not exposed.

(Case-2) multipart mail.
    Because multiprt mail, "message body text" is as follows(lines after first null line)

    This is mime message
    --boundary
    Content-Type: ...
    null
    data lines
    --boundary
    Content-Type: ...
    null
    data lines
    --boundary
    Content-Type: ...
    null
    data lines
    --boundary--

   So, broken mail is;
            Top 2048 bytes of abobe data
       +   entire mail data of this mail
   Because "null line" exists in each part, data after the firste-detected null line is treated as "mail body",
   and, data before the firste-detected null line is treated as "message header lines".
   Becaause 1-st part of multiprt has Content-Type: header, this is used as "Content-Type: of this broken mail, and is shown by Tb.
   If multipart mail, shown garbage usually depends on "first sub part of original mail".
   If multipart/related, "first subpart of multipart/related" is usually text/hrml part., and this is used aas "message body data" in normal display.
   So, even when "broken mail", HTML message is shown as if "normal HTML mail".
   However, "structure of multipart" is lost, "embed image" won't be shown.
   If multipart/mixed, "attachment is not aaccessible" happens, because correct "multipart/mixed structure" is lost.

So, "repeating broken mail, garbled display, Repair Folder was needed without message source data" is usually useless for problem analysis.

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to Mark Aitchison from comment #10)
> but maybe the problem yoman reported is related to this
> (and maybe there are similar symptoms from other reasons, and maybe the problem Barry Ralphs (Comment 1) had are something else? 

Comment #0 sounds for me following phenomenon(already reported in other bug(s).
   Offline-use=Off, or when Offline-use=On, mail is viewed before auto-sync => fetched in Disk Cache
   Because of multipart, when attachment part is non-displayable, or when displayable(image/jpeg etc.) but Display Attachments Inline=Off,
   Temporry message source is created in Disk Cache for message display with dummy attachment subpart os "this part will be downloaded...".
   This "Temporry message source" should be ignored because it's partial data, but this "Temporry message source" is used as valid data by Tb.
If so, "filter move" itself is irrelevant to comment #0. Filter merely created new mail in move target folder, and the new mail is fetched at the move target folder because new UID is created by server.

Mark Aitchison, is your "The attachment appears to be empty" actually problem only on "messages that were filtered/moved at imap account"?

Comment 16

[WADA:World Anti-bad-Duping Agency]

FYI.
Following is bugs relevant to mime_parts_on_demand which I read in the past.

> Bug 434054 - IMAP PDF attachments corrupted on download, affected by mime_parts_on_demand
> Bug 570914 - When working with IMAP attachments are sometimes corrupt (if base64 encoded part,
>                         data of "This body part will be downloaded on demand." is base64 decoded upon save
>                         and 27 bytes file is created. offline-use=on, auto-sync is paused by new mail click)
> Bug 647562 - Thunderbird claims the attachment is empty although it is visible in the raw message 
>                         ("mail data without attachment subpart data by mime_parts_on_demand=true" 
>                           + "entire mail data by auto-sync" are merged into IMAP offline-store file)
> Bug 746711 - Saving attachment produces mail file saying "This body part will be downloaded on demand"
>                  　（when browser.cache.memory.enable=false/browser.cache.disk.enable=false and IMAP offline-use=Off folder)

IIRC, following actuallt occurred in the past. i.e. Attachment was always fetched, regardless of "already fetched into Disk Cache" or not.
> Bug 629738 - Attachment is immediately re-cached after successful cache operation (no offline sync, download on demand) 
Something was perhaps changed after it.

Change by Bug 92111 was landed on Tb 16. This may be cuse of difference.

If recent Tb, following may be relevant.
> Bug 1096127 - MsgHdrToMimeMessage, when used in conjunction with IMAP parts on demand,
>                          returns wildly incorrect results starting with Thunderbird 31

Comment 17

[Mark Aitchison]

> Mark Aitchison, is your "The attachment appears to be empty" actually
> problem only on "messages that were filtered/moved at imap account"?

They all did get moved, but the filter log was giving no from/subject/etc even before the move (in tag operations, etc).  There are some odd things going on with my filter log reporting the same message processed - including moved to a folder - multiple times, which I don't understand... in case my system has some odd problems/corruption/whatever maybe what is going on here is spurious and confusing the issue so I'll stay out of this until I've thoroughly checked out the hardware and software and see if it happens again. Unless, of course, somebody thinks there could be a problem where two copies of the filter processing could be trying to run at the same time and interferring with each other.

Comment 18

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 19

[andreapfel]

I can confirm this bug in Thunderbird 45.6.0 on Linux while using gmail. "repair folder" also fixed it.

Comment 20

[Pete Kaiser]

I confirm this bug on Thunderbird 54.0 on Linux and also on earlier versions on Windows (using IMAP), and "repair folder" doesn't fix it. (See also 647562 and 680004.) I've seen this problem only with messages originating from Apple mail. Perhaps it's related to the formatting of the message: looking at the source shows that there is no empty line between the first uuencoded attachment and the second. They follow like this:

[everything including the beginning of the first attachment
...
Lfx+rgaQJsC4E2BcAcRQpCoAmbAvrAFY1AmwMzSAQBSXFCqKmYTjjiBNgXArQMwJomKqxFCqKgBg
/wD4uv/Z
--Apple-Mail=_2684EB11-E4BB-446B-A452-DE2F459E724F
Content-Transfer-Encoding: base64
Content-Disposition: inline;
	filename=Second-attachment.jpeg
Content-Type: image/jpeg;
	x-unix-mode=0666;
	name="Second-attachment.jpeg"
Content-Id: <[e-mail]>

/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAAA8AAD/4QN9aHR0cDov
...

I can easily decode the base64-encoded parts when I save the raw message. FastMail's web interface has no problem with them, and can properly display and download them. So my guess is that Thunderbird is being too careful with messages that don't quite meet the standard for delineating attachments.

Comment 21

[Marco De Vitis]

I confirm this bug, exactly as described, in Thunderbird 52.6.0 and 58.0b3 on Mac OSX 10.13.3.

On my IMAP account I have a filter which moves certain messages to a subfolder.
When such messages have attachments, they cannot be opened anymore with the "appears to be empty" error message (and inline images are not displayed).

I tried moving the messages to another folder or even another account and then back to the same folder, but nothing changes.

The only way to fix this is folder repair.

I also compared the source of a message after repair with the source of a message before repair, but I see no differences.

Comment 22

[Marco De Vitis]

I'll add more details here.

# First of all, it's not related to the sending client: I can easily reproduce it by sending an email with Thunderbird itself, from a different account towards the account where the filter is set.

# It also happens if I set the filter to COPY the message, instead of moving it: the original incoming message is left in the inbox and can be displayed with no problems, while the copy is broken.

# It does not happen if I copy the message manually (right click, copy to...) between the same IMAP folders.

# If I look at data files in my profiles after the COPY operation, it seems that the message has not been copied at all! See the following grep results for the message subject ("starless" is the name of the subfolder where the message was copied to):

$>grep -Rl "Test attachment" ./
.//INBOX.sbd/starless.msf
.//INBOX.msf
.//filterlog.html
.//INBOX/cur/1518352638786501

This shows that only the original message is present, even though the index for the target folder contains the string.

The following is instead the result after a manual copy between the same folders:

$>grep -Rl "Test2 attachment" ./
.//INBOX.sbd/starless.msf
.//INBOX.sbd/starless/cur/1518351695206383
.//INBOX.msf
.//INBOX/cur/1518351684223896