763390 - When multiple messages moved between IMAP and local mail folders in Ubuntu, the messages have only date header and X-Mozilla- headers. The body is missing.

Status: RESOLVED DUPLICATE of bug 538375

(MailNews Core :: Networking: IMAP, defect)

Description

[Arun Shrimali]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:12.0) Gecko/20100101 Firefox/12.0
Build ID: 20120423122624

Steps to reproduce:

I am using Thunderbird 12.0.1 over Ubuntu 12.04
I have configured the thunderbird for my office mail account, which has a limited space where, thus I have configured local folders also. Whenever I reach to the space limit of the office mail account, I do move (in bulk) from office account to the local folder.
( While googling, I found this issue for windows version and marked as resolved, but I am facing this issue even in latest version for Ubuntu)


Actual results:

Whenever I do try to move (or even copy) the mails, I do get the mails header with blank message in the local folder. And all the copied message has the size of 0.2 KB


Expected results:

Message should be properly moved or copied to the local folder.

Comment 1

[Hashem Masoud]

(In reply to Arun Shrimali from comment #0)
> ( While googling, I found this issue for windows version and marked as
> resolved, but I am facing this issue even in latest version for Ubuntu)
Was that a bug filed in Bugzilla? was it bug 742072?

Comment 2

[Arun Shrimali]

It may not be bug 742072 because ...
1) I do have the leased line connectivity thus there is very less probability of network error.
2) That bug was on Windows platform, which may have been resolved, where as I am working on Ubuntu 12.04

Comment 3

[Hashem Masoud]

(In reply to Arun Shrimali from comment #2)
Read the comments in that bug because other bugs are mentioned. Do you similarities between your problem and theirs?

Comment 4

[Arun Shrimali]

The difference is that ....
1) I do not have any network / Internet problem .....
2) While copying multipal message from IMAP folder to local folder, messages got copied, but with blank subject, blank "from", blank message, but proper date and time with only 0.2KB size
3) I am working on Ubuntu 12.04

Comment 5

[Ludovic Hirlimann [:Usul]]

Can you try to get a log for MsgCopyService as describe at https://wiki.mozilla.org/MailNews:Logging ?

Comment 6

[Hiroyuki Ikezoe (:hiro)]

(In reply to Arun Shrimali from comment #2)
> It may not be bug 742072 because ...
> 1) I do have the leased line connectivity thus there is very less
> probability of network error.
> 2) That bug was on Windows platform, which may have been resolved, where as
> I am working on Ubuntu 12.04

Though bug 742072 (duplicated as 522675) is no specific on Windows, this issue is definitely different from bug 742072.

Comment 7

[Hiroyuki Ikezoe (:hiro)]

(In reply to Arun Shrimali from comment #4)
> 2) While copying multipal message from IMAP folder to local folder, messages
> got copied, but with blank subject, blank "from", blank message, but proper
> date and time with only 0.2KB size

Can you post a corrupted message here?

Comment 8

[Steve Kelem]

Any progress here? I noticed that some of my message filters were supposed to move messages into certain folders. The messages disappear from my Inbox, and a corresponding number of empty messages appear in the destination folder.

A corrupted message looks like:

From - Mon Apr 28 10:22:40 2014
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:

Comment 9

[Steve Kelem]

I had an idea...
My messages go into my Inbox and then the filters run.
The messages get disemboweled only when one of my filters moves them into one of my local folders.
For a long while I had "Delete from Pop3 server" as the last action (move, then delete from server) in my filters. (Because after "moving a message to a local folder", the message would still exist in my Inbox on Gmail.)
But...I just looked at my msgfilter.dat, and there are no longer any "delete from server" actions. I'm pretty sure I didn't remove those actions.

Maybe there's some vestige of that old procedure that's disemboweling the messages.

Comment 10

[Steve Kelem]

I'm trying another experiment. I recently upgraded from ubuntu 13.10 to 14.04. A fix for a different Thunderbird bug for too many open files (https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/808752) was not in place after the upgrade. So I re-added the lines:

@user hard nofile 4096
@user soft nofile 4096

to the end of /etc/security/limits.conf

I then re-scanned for empty messages under Local Folders, and moved the 550 empty messages (created since March 26) to a separate folder. (I have 1,087 folders under Local Folders.)

I'll scan for disemboweled files in a few days to see if the problem goes away.

Comment 11

[Steve Kelem]

I ran:
% lsof | grep thunderbird | wc -l
11645

Seems like a lot of open files, especially because I had set the 4096 limits, above.

Comment 12

[Wayne Mery (:wsmwk)]

Can you attach a file with the list?

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #12)
> Can you attach a file with the list?

Comment 14

[Wayne Mery (:wsmwk)]

Arun, ...

(In reply to Ludovic Hirlimann [:Usul] from comment #5)
> Can you try to get a log for MsgCopyService as describe at
> https://wiki.mozilla.org/MailNews:Logging ?

Comment 15

[Steve Kelem]

Attachment: lsof +c 0 | grep thunderbird | sort -k 1.100 -o foo2

This is the output of lsof +c 0 | grep thunderbird | sort -k 1.100 -o foo
but with 21,000 lines removed (the list of every mail folder I have, but replicated 50 or more times.)

Comment 16

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 17

[Wayne Mery (:wsmwk)]

What Steve reports is related to Bug 855836 - Crash on loading mails from IMAP-folder with "Creating pipes for GWakeup: Too many open files". Perhaps the same core issue for Arun, the reporter. Although he seems to be gone.

Comment 18

[Steve Kelem]

Why is this marked Unconfirmed if more than one person is experiencing this problem?
I thought that the empty messages were "duplicates" of messages that were successfully moved into a folder, but recently some of the empty messages had no corresponding "complete" message in that folder. I had to use the browser to look at the gmail folder to get the contents!

This is data lossage!

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to Steve Kelem from comment #18)
> Why is this marked Unconfirmed if more than one person is experiencing this problem?

As for me, it's pretty simple. I can't reproduce problem of this bug, and not all similar phenomena is problem of same cause/same flaw in code/same solution.
If any similar phenomena are considered actual bug of code, only one bug of "Thunderbird has bug!" is sufficient. Any bug report at b.m.o. can be dup-closed to it :-)

> This is data lossage!

Actual data loss?

IIRC/IIUC, major cause of this kind of issue in "move from IMAP folder to local folder" is "connection error/interruption by someone including other Thunderbird task etc. while gettting mail data from imap mobox in order to add mail data to local folder.
Known "innturppution by someone" cases :
 New mail alert while Dragging mails in order to move mails to other folder -> Drga vanishes.
 Pull off LAN cable while moving many mails from an imap mbox to other mail folder(my intentional test)

"Move mails from imap mbox to local folder" consists of:
  (i)   Get mail data from server by "uid fetch xx:yy body[]"
  (ii)  Copy the mail data to local mail folder.
  (iii) Delete mail from imap mbox by "uid store xx:yy +Flags(\Deleted)".
Possible phenomena when error occurred during (i) are:
At (ii), because mail data is not available due to error while (i), null data is written as copied mail to local folder.
  => phantom mail in local folder
At (iii), because of error in server connection, "uid store xx:yy +Flags(\Deleted)" is impossible.
So, mail in imap mbox can not be deleted.
However, display of an imap mail is based on messageKey=AAA where AAA is UID of a mail, and when error occurred, msgDBHder for the messageKey=UID can be removed from msgDB(==xxx.msf file) by Thunderbird due to error condition.
In this case, because msgDBHdr of the mail of messageKey=UID doesn't exist in msgDB, mail is not shown at Thread pane.
Because "uid store xx:yy +Flags(\Deleted)" is not executed, mail is shown again by "Repair Folder" == re^fetch message header of all mails.

This was true when similar phenomenon was observed during MailDir test while MailDir had many problems.
Even if server connection error never occurs, due to problem in MailDir, any error during move/copy ccould occur :-)
And, similar phenomenon could be observed by intentional test of "pull off LAN cable while moving 1000 mails from imap folder to local folder". Because LAN cable was pulled off, no one can do "uid store xx:yy +Flags(\Deleted)". So, even after "not moved successfully" mail disappeared from "move source imap folder in Thunderbird", mail was shown by "Repair Folder".

I don't like behavior like shouting "This is data lossage!" in bug report at B.M.O.
Here is B.M.O for developers to resolve bug. Here is never support forum nor Customer support Center.
What is needed is "providing sufficient data for problem analysis to developer", instead of adding comment of complaint only.

> but recently some of the empty messages had no corresponding "complete" message in that folder.
> I had to use the browser to look at the gmail folder to get the contents!

It's evidence of that mail in imap mbox was not deleted by Thunderbird when you saw problem simlar to this bug?
If deleted by Thunderbird, mail shouldn't be shown by Web mail.

> I thought that the empty messages were "duplicates" of messages that were successfully moved into a folder, (snip)

Phantom mail("From - line + some message header such as X-Moxzilla-Status: + null mail data" in msgStore file of local mail folder) is not so rare phenomenon.
You can condistently see it by utilizing problem of Bug 1319299. You can condistently see it by utilizing problem of Bug 1110583 too
It's pretty simple. After msgDBHdr is created, if something bad happens and writing mail data is impossible, or if something bad happens and fails to remove the exess msgDBHdr with no mail data, it's shown as phantom mail.
As seen in Bug 1106225, phenomenon of Bug 1319299 was "duped mails" in former Thunderbird, but it was morphed to "consistent phantom mails" perhaps from Thunderbird 38 because of some changes.

Are you talking about this problem?

If error occurred while doing "uid fetch xx:yy body[]" in order to copy/move mail to local mail folder, following may occur.
  msgDBHdr for copied mail at local mail folder is created.
  but due to fetch error, mail data can not be added to msgStore file for the local maol folder.
It may be similar phenomenon to what I saw when I intentionally pulled off LAN cable.
This is a phantom mail case which is produced by diffedrent cause from bug 1110583, although "no mail data is available when copy of mail data is tried" is common.

However, I can't understand why you can see problem so frequently or consistentluy?
Do you use Drag&Drop to move many mails from imap folder to local mail folder?

Comment 20

[Wayne Mery (:wsmwk)]

> However, I can't understand why you can see problem so frequently or consistentluy?
> Do you use Drag&Drop to move many mails from imap folder to local mail folder?

Arun?

Comment 21

[Steve Kelem]

I still see the problem in 56.0b3 (64-bit) on Ubuntu. I don't use drag and drop, just filters or right-click-move-to-folder.

Comment 22

[Wayne Mery (:wsmwk)]

(Arun seems to be gone)

Comment 23

[Ben Lerner]

Wayne, I see the problem also on TB60.5.2, and am happy to try to get some more information here.

(In my case, my IMAP account is a work account, and Local Folders are used to save mail for archiving, so data loss is not good, but moving mail into local folders is kinda frequent. I typically trigger the move using the Nostalgy extension, but I get the same blank messages if I try using right-click-move-to-folder or filters, so it's not the extension at fault here.)