Open Bug 462156 Opened 17 years ago Updated 7 days ago

Thunderbird "loses"/corrupts email messages when downloading from the mail server to a local folder

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Version:
1.8 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

People

(Reporter: paul.barlow, Unassigned)

References

Details

(Keywords: dataloss, steps-wanted, Whiteboard: [needs reproducible steps][testcases: comment 35, comment 38, comment 84][tools: comment 52])

Attachments

(1 file)

dataloss.log
15.85 KB, text/x-log
Details
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30) Build Identifier: Thunderbird version 2.0.0.17 (20080914) On a several occasions (most recently today), Thunderbird has "lost" my mail messages that are in my inbox when I move them to a local folder. Effectively it corrupts the messages - they appear in the local folder as 1 KB messages with no subject or sender info. They are empty messages. There is one 1KB mesg for every message that was in my inbox. This is a serious problem because I use Thunderbird for my work. I work for Sun Microsystems. Reproducible: Sometimes Steps to Reproduce: 1.Select all messages in my inbox 2.Select "Move to", select "recent" and select "local folder name" 3.Wiat for move process to complete Actual Results: This doesn't happen everytime. When I used Mozilla, I can't remember it ever happening. I used Mozilla for many years. I've been using Thunderbird for about a year and this happened about fout or five times. I have made a copy of the local that contains the "blank" 1 KB messages. However, it's a big file. It my main folder for my incoming messages.
Version: unspecified → 2.0
I have taken a screenshot showing the 1KB blank messages in my local folder
Does "Rebuild index" in the folder properties help?
Sorry for delay in responding. I tried using the "Rebuild Index" and it doesn't resolve the problem. The same problem happened again this morning. I lost about 40 email messages while they were in the process of being downloaded. I have made a copy of the mail file with the "blank" messages. This is a serious problem.
Does it happen with all extensions disabled (thunderbird.exe -safe-mode)? To try to diagnose the problem, find the folder you moved them to (in the file system), and check if the messages actually got moved or not. The mailbox is the file with no extension - you can open it with a text editor (like wordpad). See http://kb.mozillazine.org/Profile_folder
This is an extract from Wordpad. As you can see, no message content. From - Wed Nov 26 09:00:09 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:09 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:09 2008 X-Mozilla-Status: 0013 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:09 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:24 2008 X-Mozilla-Status: 0013 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:24 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:24 2008 X-Mozilla-Status: 0013 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:25 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:25 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:25 2008 X-Mozilla-Status: 0010 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:25 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0003 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0010 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:26 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:27 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0003 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:28 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:35 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:35 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:40 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:41 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:44 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:45 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:45 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:45 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:51 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:58 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:00:59 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:08 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:08 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:16 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:17 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:17 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:17 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:17 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:17 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:18 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:18 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:18 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:18 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:26 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:27 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:28 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 From - Wed Nov 26 09:01:28 2008 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000
Ok, so does it happen with all extensions disabled (thunderbird.exe -safe-mode)? And when it does, do you get anything of interest in the Error Console?
Hey, soory, I didn't try with extensions disabled. Not sure this is a relevant request. Let me explain. The emails are in my inbox and I'm moving them to a local folder. Once I have moved the emails and they have been "lost, I can't recreate the problem until it happens again. It doesn't happen each and every time I move mail messages from inbox to the local folder - I always use the same local folder. It's a big folder - 600MB+ because it my local version of my work email inbox. Once I have moved the mail into this floder, I run filters to move mail to various local folders.
Oh, I always see the mail message in teh process of being download and moved. I see the message "Downloading x message of y messages" in the message window. Looks like the process is working and then the process doesn't finish. When i go to thye local (target) folder I see the "blank" messages and nothing in the inbox folder. They're mno longer on teh mail server either.
Do you want a screenshoot of what the folder looks like with the "blank" messages?
No screenshot needed.
Hey, this just happened again. I lost all my email message from yesterday and today. I had decided to download my message to a "smaller" local folder, and then move then from that "temp" folder to my main local inbox folder. My local inbox folder is very large. I thought the size of the mail file receiving the downloaded messages could the problem. Well, it's not. When I downloaded my messages today, the "temp" folder was empty - therefore, small size. I'm stopping using Thunderbird until you get this resolved.
Could it be anti-virus behaving badly? Do you have the quarantine option enabled under Options | Privacy | Anti-virus?
Yes, it's enabled. I'll disable it. I'm looking for the option to leave a copy of messages on my mail server. this would gves me backup. At present, when this problem occurs, I lose everything, which I can't aford.
What I'm going to do as a workaround is copy the messages from inbox on our mail server to my local folder, verify successful download, and then delete them off the mail server. this will stop me losing messages. I have been using the "move" functionality.
> I'm looking for the option to leave a copy of messages on my mail server Account Settings | Server settings | [x] Leave messages on server ?
I don't see that option., but I can "copy" the messages to my local folder first and then delete them after I have determined they have been downloaded successfully. This is the workaround to using "move" which loses messages periodically. I'm good with this, but I still think there has to be a bug that is causing the "move" functionality to lose messages.
Keywords: dataloss, perf
There were a number of move bugs that were fixed for TB3, including "Bug 497622 - IMAP cross-server move fails silently after a copy to a different folder" which could explain the symptoms of this report. If no further reports of this occur in TB3, perhaps we could close this bug as WFM. But we need some experience with TB3 first.
Bernd, is your problem gone with version 3? Paul (reporter) wrote me a couple months ago, that he wasn't yet on version 3, and that "I'm still using Version 2.0.0.24 (20100228) but have changed my antivirus s/w. I'm now using McAfee distributed by my company, Oracle. I had been using Symantec when I was working for Sun. I haven't had the problem for several months now. I've gone back to using "move" to move my email from my inbox to a local folders upon which I run filters to move my mail to various local folders. When I was experiencing the problem, I had restorted to using "copy" followed by deleting if the copy was successful."
Whiteboard: [needs retest v3]
I don't know if this is the same problem. I routinely see a corruption of messages when moved from an IMAP store to Local Folders. I just moved 5, and one was corrupted; it is a blend of the text body of message just deleted, and the message which was being moved. The body is appended immediately before the first message header of the message Delivered-To:
Sorry, Lanikai nightlies: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10pre) Gecko/20100828 Lightning/1.0b2 Lanikai/3.1.4pre
It still happens with thunderbird 3. Lastest version. We use symantec endpoint protection. This problem does not happen with the old netscape. I install that for a test and it works perfectly so I do not think its an antivirus issue.
disregard my comment for another bug.
If paul's issue still exists in v3.1, hopefully it will be resolved by one of bienvenu's cache magic fixes.
Keywords: perf
Whiteboard: [needs retest v3] → [dupme][needs retest v3]
(In reply to Wayne Mery (:wsmwk) from comment #24) > If paul's issue still exists in v3.1, hopefully it will be resolved by one > of bienvenu's cache magic fixes. Can anyone confirm this to be the case?
Whiteboard: [dupme][needs retest v3] → [dupme][closeme 2011-09-29]
I started scanning messages in my local folders to look for possible corrupted messages. I found one dated Jul 29, 2011. It appears to be the combination of two different messages, one blasted in between: From - Mon Aug 08 23:29:00 2011 X-Mozilla-Status: 0006 X-Mozilla-Status2: 00000000 and Delivered-To: ... so this problem appears to still exist.
All, I still experience this problem. I recently uopgraded to 3.1.14. I've been using Version 3 for many months. A few weeks ago, I was "copying" (not "moving") emails from the mail server to a local folder and the problem recurred. I see the message that Thunderbird is downloading mail messages, and the number increments. Then the process stops for some reason (I suspect a connectivity issue). I have to restart Thunderbird to be able to access the local folder. When I looked at my local folder, which should have contained the copied message, it had "empty" blank messages instead of the copied messages. Thunderbird needs to check that copies and moves from the mail server are completed successfully. If not, it should clean up the blank messages in the local folder and, in the case of a move, restore the messages to the "inbox" on the mail server. I've had this problem probably a dozen times since I started using Thunderbird. I avoid using "move" for this reason. "Copy" is safer because it leaves the original messgaes on the mail server. I only delete them once I have verified that they were copied successfully from the mail server to the local folder. A developer should look at the logic at the end of the copy/move routine and add code to check that the copy/move of the messages was succesful. It shouldn't be rocket science. Paul
I fully agree Paul. This happened to me once and almost saw me fleing Thunderbird. Tha pin, and the awesome shame. One of the most important secure operations a user demands, moving mails, so easily broken. Surely one of the design principles is DO NOT REMOVE THE ORIGINAL UNTIL THE COPY IS VERIFIED! Sorry, but I am agape and at a loss as to why that got overlooked. I too am manually copying and verifying then deleting as much as I can. That said, this is so cumbersome that I've been using move again in the hope it's fixed and nervous as, but it hasn't destroyed an emails lately. I would dearly like to see a fix to this and it has me half tempted to download the source and attack the issue myself. How hard can a move method be? Aaargh, if only I had time to scratch myself let alone hack around code like this anymore. So I stand in unabashed gratitude to those who do contribute! I make no criticism of motives and efforts, but sheesh it's hard to believe this kind of bug survives and it can only be that none of tose who contribuet code regulalry have experienced it, else they'd have turned their attention to it right away!
All, I still experience this problem. I recently uopgraded to 3.1.14. I've been using Version 3 for many months. A few weeks ago, I was "copying" (not "moving") emails from the mail server to a local folder and the problem recurred. I see the message that Thunderbird is downloading mail messages, and the number increments. Then the process stops for some reason (I suspect a connectivity issue). I have to restart Thunderbird to be able to access the local folder. When I looked at my local folder, which should have contained the copied message, it had "empty" blank messages instead of the copied messages. Thunderbird needs to check that copies and moves from the mail server are completed successfully. If not, it should clean up the blank messages in the local folder and, in the case of a move, restore the messages to the "inbox" on the mail server. I've had this problem probably a dozen times since I started using Thunderbird. I avoid using "move" for this reason. "Copy" is safer because it leaves the original messgaes on the mail server. I only delete them once I have verified that they were copied successfully from the mail server to the local folder. A developer should look at the logic at the end of the copy/move routine and add code to check that the copy/move of the messages was succesful. It shouldn't be rocket science. Paul
Thanks,Bernd. I'm surprised that more users aren't reporting this as an issue. Perhaps not that many users download their email to local folders. Anyway, I did want to add that I'm now running Windows 7. When I first reported the problem, I was running Windows XP. So it appears that this not OS related (or Thunderbird version related). Let's see if there's a developer willing to investigate.
Paul, I don't see any feedback that safe mode was tested. Can you test this please with daily build ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central/ started in safe mode?
Whiteboard: [dupme][closeme 2011-09-29] → [closeme 2011-11-25]
Aha! I've been searching for a reference to this bug that I've been experiencing for several months now, and currently with Thunderbird 8.0 on Win32. Seeing exactly the same thing as MrC reports in Comment 26: body of a message duplicated between another message's "X-Mozilla-Status2:" header and its following header, generally "Delivered-To:". Have not been able to reproduce it reliably, though it is now occurring on a daily basis. The symptom, when it occurs, either while downloading a folder via IMAP or manipulating messages by replying, changing status, and moving them around between locally stored folders (just normal operations in the handling of email), is that all headers except Date will suddenly disappear from a message or two at random, and the "From" column will contain a single "-" character, apparently taken from the From_ header due to the lack of a From: header. To clean it up, I quit Thunderbird, wait until the process has disappeared, remove the .msf index file that corresponds with the folder containing corrupt messages. Then edit the folder file with a text editor, removing errant text between "X-Mozilla-Status2:" and what should be the immediately following header. It seems that in most or all cases, the extraneous text is duplicated from another, intact message in the same folder; no messages seem to be lost, only corrupted in this fixable fashion. For most users who lack the ability to repair the damage, this is a critical bug that causes data loss. Its increasingly frequent occurrence is causing me to push everything into the gmail cloud, rather than rely on easily corrupted local storage for business-critical messaging and archiving. Will try to narrow down to a simple case that can be reproduced, but as I said it happens randomly as far as I can tell. I'd suspect the code that constructs and outputs the "X-Mozilla-Status2:" header. Possibly a buffer overrun that causes other data to be leaked into the file at that location.
iirc bienvenu recently worked on something like this, but it was a trunk regression. Paul, do you have an update? (In reply to ppww from comment #32) > > Will try to narrow down to a simple case that can be reproduced, but as I > said it happens randomly as far as I can tell. I'd suspect the code that > constructs and outputs the "X-Mozilla-Status2:" header. Possibly a buffer > overrun that causes other data to be leaked into the file at that location.
Whiteboard: [closeme 2011-11-25]
It happened again yesterday. I was downloading messages to my laptop. There were a couple of large messages to download (9 MB). I guess I lost connection to the Internet and the "copy" from the mail server Inbox to one of my local folders was interrupted. When I looked in my local folder, some of the messages had been copied but others were "blank" message - no subject text (just shells). Now, I delete them and re-copy them. I no longer use "Move" because when you use "move" you lose the messages.
(In reply to Paul Barlow from comment #34) > It happened again yesterday. I was downloading messages to my laptop. There > were a couple of large messages to download (9 MB). I guess I lost > connection to the Internet and the "copy" from the mail server Inbox to one > of my local folders was interrupted. When I looked in my local folder, some > of the messages had been copied but others were "blank" message - no subject > text (just shells). Now, I delete them and re-copy them. It sounds; (assume move mail #1 ... #N ... #X) (i) Mail #1 to #(N-1) is successfully copied. (ii) While fetching mail #N from IMAP server, network error occurs. So, mail #N in local mail folder is partial. (iii) Similar phenomenon to bug 522675 comment #5 happens. (Move of single big mail. Tb went Offline because LAN cable was pulled-off) (iv) Tb still working in Online mode, and goes ahead. (v) Tb silently stops at mid of bulk mail copy step of bulk mail move. (vi) Mail #(N+1) to #X can not be copied to local mail folder, because network error is occuring. > When I looked in my local folder, some of the messages had been copied > but others were "blank" message - no subject text (just shells). It looks that Tb continued "copy mail #(N+1) to #X" even after fetch failure of mail #N occurred due to network error. Test of bug 522675 comment #5 was "move single mail". So, I think Tb wrongly considered that "copy mail step of move mail" is ended when fetch error happened then Tb goes ahead to "delete step of move mail" when going back to Work Online mode by Plug in of LAN cable. If Tb considers that "bulk mail copy step of bulk mail move" completed, I think Tb goes to "mass delete step of bulk mail move" sooner or later. (a) Was mail #1 to #(N-1) deleted at IMAP folder? (b) Was mail #N deleted at IMAP folder? (c) Was mail #(N+1) to #X deleted at IMAP folder? (I guess No for (c) as you say "I delete them and re-copy them".) Because network error is occurring, "mass delete step of bulk mail move" is also fails, and mails are fortunately not deleted from IMAP folder(move source folder)? If so, if error is temporary failure like timeout or "NO" response to fetch due to internal failure at server, Tb may delete mails from IMAP folder(store Flag \Deleted) even though copy is not successfully executed. Does your network produce "connection loss while downloading IMAP mails to local folder" consistently? if yes, get IMAP log, please.
Summary: Thunderbird "loses" email messages when downloading from the mail server to a local folder → Thunderbird "loses"/corrupts email messages when downloading from the mail server to a local folder
wada, I fear Paul may be gone. But perhaps Bernd or ppw could offer reply to comment 35?
I'm not sure I can comment on comment 35 meaningfully. I have done very few moves since reporting this as I am inherently nervous. But the few I have done, I have not seen this proble and breathed a heavy sigh of relief. Mostly I tend to do a copy, then a delete as two separate steps. I'm of the perosnal view that dioiagnosing this is irrelevant as it is very intermittent (low probability and hard to reproduce) but very serious (loss of data). To me this warrants an overhaul of the the entire move code. That would mean a walk through it, and I've ben tempted to download it and try, but I've been far too busy and alas am not set up to build Thunderbird so the first step would be a time consuming gear up to get to that point. If I were to reasess it, it would be to simply asses whether it implements an optional (driven by a configuration setting) or hard coded, check for success of the copy before deleting. Most every step must assume that the previous may have failed and be feigning a success status code. Prior to the delete a check of true success on the copy should be performed. It would suffice probably to simply read the copied mails one by one and chekc some key headers and calculate message size and compare with records noted when read for the copy. If all checks out, proceed with the delete. But I'm not I admit a file system expert in coding (a background more in automation systems, simulution and modeling software and other high level, not low level applications). It simply amazes me that this kind of data loss is even possible and not designed out at a very low level in the Mozilla engine form a very early stage. Imagine if the Windows kernel or the Linux kernel had an unreliable move that resulted in data loss occasionally ... ouch. I find it hard to imagine. My ocnfidence alas is deeply shaken and I approach moves with the greatest of trepidation in Thunderbird and preference to avoid. I do it with low imporance data at times to keep a finger on the pulse in a sesne and see if the many subsequent releases still exhibit the issue at times. WOuld love to hear of an overhaul. I read recently that Mozilla foundation won't develop Thunderbird any further and is throwing it pen to the community to maintain. If that is true I guess it's up to us to revisit this for confidence's sake.
I recently encountered this also: and it seems in part a consequence of a slow / intermittent internet connection (wireless) to my IMAP server, and a transfer involving a large-ish number of files, including some that are particularly large (e.g. > a couple of MB). I do not seem to have this problem (or, at least, nearly as often) if I 'plug into' a faster wired network. To test this properly I think you will need to reproduce those conditions: some kind of really slow / bad latency connection to the server.
This bug has been around forever, and really needs some attention. A drag of two messages today from my gmail IMAP account to a local folder corrupted the local folder.
Ian, MrC, do you symptoms exactly match comment 0? "they appear in the local folder as 1 KB messages with no subject or sender info. They are empty messages. There is one 1KB mesg for every message that was in my inbox."
Yes, I saw the same local folder listing of 1KB messages with no subject or sender info. Saw this a couple of times, in fact. In my case it was from copying files from my inbox into a local folder. Rebuilding the index did not fix this. However, it turned out that, for me, the files were still on the server, so there was no data loss (AFAICT). To my mind, a key factor to reproduce this is having a very slow / intermittent internet connection to the IMAP server. Another factor may be having the computer running TB go into sleep mode (might be shutting down while TB is still trying to load from the server).
I see what I reported back in Comment 20.
Hi, got the same error since upgrading from TB3.6.xx as forced by my company. Upgrading to TB 10 didn't solve the problem, same error in TB 11.x and now same error in TB 17.0.2 yesterday and today. Mail server is IMAP. Mails look good on server. Downloading to local folders results in corruption. After downloading message to local folder OTHER messages in local folder are now corrupt (empty body or body of another message). Sometimes the downloaded messages itself is corrupt. Sometimes messages in OTHER local folders are now corrupt (empty body or body of another message). Using the repair option deletes corrupt messages from folder. Unfortunately i've lost about 200 mails from customers using this way before being aware of the problem and implementing some workaround (backup mails to other folder on mail server for re-download). As i'm using TB for work this is a very serious problem. Unfortunately our local support wasn't able to fix it. Neither TB forum nor using Google presents any solution or user error. This is a very serious problem.
With all due respect Wayne I think you're missing the point. An intermittent problem may never have reliable steps to reproduce and I reiterate this question: How on earth can Mozilla (or any sensible data secure engine) code include any code that removes an email without checking first that a) it has been properly copied elsewhere, to a chosen destination or a trash can, or b) has been explicitly permitted by a user - let alone for mass operations! The problem here is not to reproduce it, it is simply to walk through and certify the Mozilla code as data secure. It is currently not, would not come anywhere near a data critical application like a bank say and scares the bejesus out of a lot of us who have lost email on moves. There work around is a copy, then manual delete. The amazement is that this cannot me automated and isn't the existing paradigm. How many parts of the code can be involve in mail moves? How much work can it be to assess it and certify it as data secure? Why is this not a priority somewhere? It is not asking much of a mail application, in fact it is an expect ask, as it is of an operating system and file management system, that move is secure, and never removes the source until the destination is securely in place.
In fact the only way I can imagine this happening at all is if Mozilla at some place delegates a copy to someone who reports success untruthfully, and then it deletes the source.
Please, I don't think any of us are developers, so let's not get on a tangent of playing developer or presuming what a developer should do. Bernd, intermittent does not necessarily mean that the problem is not reproducible. Reproducible can mean you have steps that result in reproducing 10%, 20%, etc of the time - and,just as important, that someone else can reproduce using those steps. Thus some means of reproducing is available to a developer. Anyway, AllUses' issue sounds like something different - might be one of the bugs in comment 44 query, or might need a new bug. Ian's, with a theory regarding slow connection, sounds like it's worthy of being it's own bug. I don't know about Bernd's and MrC's. ppww, you seem to have something reproducible, or have a working theory for your problem. Does your theory fit with Paul's description of the bug?
I attempted in comment #38 to provide information so that someone could create a 'reproducible' test environment. However I am not in a position to do that, nor do I have the tools / skills to diagnose the problem, should it re-occur. I think there is sufficient information in the preceding comments to construct a test scenario: just don't think there is anyone who can take advantage of that.
getting back to the question of whether to close this bug... (In reply to MrC from comment #39) > This bug has been around forever Yes, *Paul's* bug. But in the past year there are *other*, newer bugs. And even in Paul's casein a 4.5 years span it is possible, even probable, that he got hit by more than one problem. The conundrum/difficult question then for anyone with a generic symptoms of "messed up mail" is, am I being affected by "Paul's bug" (this bug) or something else? Unfortunately without precise steps that someone other than reporter can use, it is not possible to know. Myself for example, only started seeing corrupted messages starting fall 2012. Which means I am seeing a newer, different bug. In the interest of seeing these issues fixed, I suggest 1. in this bug we focus on clearly identifying the steps required to reproduce Paul's and Ian's "interrupted session" whether it be by failed network, or terminated thunderbird. NB Bug 522675 and Bug 792198 2. all other issues that cannot be clearly linked to a interrupted session be taken to other bugs
Whiteboard: [needs reproducible steps][testcases: comment 35, comment 38]
Close it or leave it open. I don't really care much at this point, as I think the real issue is that it has never been taken seriously. So we can all open new bugs of our own particular flavor that will languish for 5 years. I'm sorry to be so cynical, but data corruption bugs should garner attention and be show stoppers. Apparently not though. I'd have been more than happy to do what was necessary to reproduce it, provide logging, etc. There may not be a sequence of steps that are reliable (this could be a race condition somewhere).
There is information in this report that differs from / is additional to that in the potential dups you identified (e.g. Bug 792128). I tried in my own comments (38, 48) to outline when and how things failed for me, so that someone could build a test environment and diagnose this properly. To summarize: 1. Build an environment with a slow / potentially intermittent network connection from TB to the IMAP server 2. Attempt to move lots of file (100+) at the same time from the IMAP server to a local folder. 3. Repeat (2) until something blows up. That seems a pretty straightforward (if hard to set up) test case. I can't, though, guarantee this tests for teh same problem uncovered by the original reporter.
(In reply to MrC from comment #50) > Close it or leave it open. as implied in comment 49 this should bug should stay open and be pursued. perhaps I was too optimistic about the community? > I'm sorry to be so cynical, but data corruption bugs should garner attention > and be show stoppers. Apparently not though. I'd have been more than happy > to do what was necessary to reproduce it, provide logging, etc. There may > not be a sequence of steps that are reliable (this could be a race condition > somewhere). If you should change your mind about not wanting to help reproduce, then the help would be appreciated because there are *in fact* people who can help fix as proven in this list https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring;keywords=dataloss;keywords_type=allwords;list_id=5829646;field0-0-0=short_desc;type0-0-1=substring;field0-0-1=keywords;type1-0-1=allwordssubstr;resolution=FIXED;classification=Client%20Software;classification=Components;chfieldto=Now;chfield=resolution;query_format=advanced;chfieldfrom=12m;chfieldvalue=fixed;type0-0-0=anywordssubstr;field1-0-0=short_desc;product=MailNews%20Core;product=Thunderbird;field1-0-1=short_desc http://juristr.com/blog/2013/02/test-your-app-under-slow-network-speeds/ describes using tools to simulate slow connections. So, anyone can try to help reproduce. And there is no lack of tools ... http://www.fiddler2.com/fiddler2/ http://info.iet.unipi.it/~luigi/dummynet/ http://ws.apache.org/commons/tcpmon/tcpmontutorial.html#slow http://www.netlimiter.com/ http://www.tmurgent.com/tools.aspx http://www.dallaway.com/sloppy/ http://bandwidthcontroller.com/trafficShaperXp.html
Wayne, I think the community is great - but as a member and reporter of bugs for almost 13 years I have to say there's a big difference between reporting a dataloss bug and being able to build out a test environment to reproduce it. First, no one can afford the risk of testing and finding dataloss bugs on their personal email account. Second, very few have the resources to build out a duplicate environment for testing. In my case I know I don't have the hours it would take to do this, and the hours it would take to test over and over to try and reproduce the failure. Third - and to MrC's point - even if I could find the time, I don't feel very motivated when there is nobody on the development side looking to do anything about this bug. I really, really appreciate your work trying to facilitate Thunderbird bug reports - and trying to mentor the conversations about orphaned bugs like this one - but in this case I think the test case creation problem is a huge barrier.
Hello everyone, I would like to confirm that this bug is still active. I just lost over 500 emails with the exact same symtoms as described in this thread. I am running TB 24.1.0 on XUbuntu 13.04. The remote server is gmail with IMAP. The local folder is now full of blank (header, subject, body -- everything) emails with a single dash in the "From" field. Of course, uppon failing to write the emails to the local folder thunderbird hapilly deleted them from the server. Here is a screenshot of the local folder now: http://i.imgur.com/x97zGpB.png How the status of this bug can still be "Unconfirmed" is beyond me. Multiple users are reporting the same problem with the same symptoms by performing the same opperation. I love TB but I cannot keep using it with such a risk of data loss. I understand we're run by volunteers' time, but seeing that such a critical bug has been barely acknoledge in 5 years doens't help either. We're talking about permanently deleting the users email from the server! It doesn't really get worse than this.
I can also confirm that this is a bug. The issue is that selecting all items in the Inbox and then dragging and dropping into a local folder, The messages are removed from the Server Inbox but only blank lines are created in the destination local folder. The icons etc for the messages are visible, but the messages and attachments are lost. In my case they have also been remove from the remote server so I am stuck and not best pleased either. The account is an IMAP account with my ISP. Selectively moving emails or multiple emails seems to be ok. Common factors - IMAP, Selection of all emails on the server, drag and drop. Can this be sorted out as a matter of urgency please. Thanks Darren
Thunderbird Version 24.1.1 before anyone asks!.
I've been haunted by this also (see bug #792198). The only way to be safe is to enable message synchronisation and local copy in the server settings. Of course, that renders IMAP obsolete, as we now have essentially POP again...but better than loosing mails. I really suspect that there's nobody at Mozilla who knows of this bug and really knows where the suspect code is and how it works, otherwise someone would have had a look at it, it shouldn't be that hard to verify an alogrithm and fix it (ok, that'll be a bit harder) if it's wrong. So please guys, if there's anybody out there who can give us at least a pointer to where to look, maybe one of the victims of this bug may further can dig into it and find the suspicious sequence.
Hello, We've been experiencing the exact same bug with one of our clients : drag & drop a bunch of messages from a remote IMAP folder to a local foler, local copies are corrupted (blank) and the original messages are removed from the server. I've been investigating this bug for a few days and I think I have narrowed it down enough to share my results. My setup : Cyrus IMAP server installed on a Debian Linux virtual box. SSL and DEFLATE were disabled and IMAP protocol captured thanks to Wireshark. I generated a lot of big fake mails with attached files. I used various netfilter options in order to simulate network delay, random packet loss, connectivity loss etc. First conclusion : After looking at the protocol dump. TB never deletes before successful copy. TB doens't delete in batch. Second conclusion : TB never triggers deletion if there was any protocol or connection error. Therefore, this problem is not caused by network problems. IMAP uses TCP for transport. TCP provides a reliable logical data link between the client and the server. Either the data is delivered without any corruption no matter after how much delay and retransmissions. Either the logical link is broken and TB aborts copy. All my attempts at degrading the connection gave two results, either the TCP link survived and there was no problem at all, either the link died and copy was aborted. There is no other outcomes. Third conclusion : The only way for the local copy to return success and write corrupted messages at the same time is for the data to arrive already corrupted inside the IMAP protocol. I wrote a small python script that acted like an IMAP proxy. I made it insert garbage inside the IMAP fetch responses after a few successful fetches. With this script, I reproduced the exact same symptoms described in this bug. Surprisingly, TB checks very few things before writing messages to local folders. The messages are written even if essential mail headers are absent. This explains that many people see empty From and Subject lines. All in all the most plausible cause for this bug is a server-side error where the server returns garbage in response to an IMAP fetch request. Maybe a patch could be written for TB to check if messages are well formatted before writing them, but we could still get corruption inside bodies and it wouldn't solve the problem at its root. Another surprising fact is that nobody spoke about the exact IMAP server software they use. Maybe if people who report here can give us some details, we could correlate this bug with a particular IMAP server configuration. In my case we're using Cyrus IMAP in a cluster configuration (Cyrus Murder) and there is a lot of things that could go wrong in there and possibly cause this bug. I'll post details if we find out what's going exactly. Alexis
To all problem reporters: When your problem of "lost mail"(deleted from Imap Inbox, but X-Mozilla-Status; heaer on ly mail in local mail folder) occurs, (Q1) is the mail actually flagged as \Deleted by Tb? Show "order Receivedcolumn" always. if IMAP, value is UID of mail. Server Settings, "When message is deleted", change to "Just mark it as delete" Do "Repair Folder" of IMAP Inbox. Is "mail of strike thru line" shown at thread pane? If yes, is it mail you tried to move to local mail folder? If it is the "lost" mail, is the mail shown normally by "Undelete" of the mail? (Q2) local mail folder8move target folder) Show "order Receivedcolumn" always. if local mail folder, value is "Offset" in file. What value is shown at "Order Received" column for the null mail? Is null mails disappear by "Repair Folder" of the local mail folder? Does lost mail" appear by repair folder? For "broken mail in IMAP mbox". If Offline-Use=Off folder in Tb(auto-sync doesn't download mail data", Tb saves "mail data stream sent from server" as-is in .eml file. Do following. (1) Create a new IMAP folder(call FolderX), set Offline-use=Off (Folder Property, Synchronization) (2) Copy problematic mail to FolderX. ("uid copy nn FolderX" is issued, and copy is done by server) (3) Folder Property of FolderX, Repair Folder, in order to force re-fetch of all headers from scratch. (4) Save the mail in FolderX to .eml file. Is mail data already broken at server Mbox?
I've been encountering a similar problem with a POP3 mail account. When the internet connection is working fine, all is well. If I'm retrieving email when a connection problem occurs. I get some emails (usually all, but not always)listed, but some of the entries point to the content of the previous email. Closing TB & deleting the .MSF file & restarting TB rebuilds the list & points to the content of the correct email. However that content can be incomplete. So it looks like there is something going adrift in TB's error handling, especially as it then deletes the emails from the POP server as though it had correctly downloaded them. (the data & corrupt MSF file are available if needed)
I've also just experienced this bug - very frustatrating!
Definitely frustrating. Please see and reply to the 3 questions in comment 59. This will help narrow the issue for YOUR particular situation.
Flags: needinfo?(victor)
Flags: needinfo?(paul.barlow)
Flags: needinfo?(me)
Flags: needinfo?(lists-general)
Flags: needinfo?(darren.goddard)
Flags: needinfo?(alexis.bezverkhyy)
Flags: needinfo?(AllUses)
(Paul's address bounces - bug reporter is gone) BTW, in case it's not obvious, there are, and have been, multiple causes to losing data. Which means there are multiple bugs, and that this bug 462156 can't address a fix to every one of you seeing dataloss - multiple bug reports are needed. Which is why it is important for us to have information (ref comment 59) from you that can be used to precisely determine what you are seeing.
Flags: needinfo?(paul.barlow)
Hello, What makes you think that this issue is caused solely by Thunderbird ? It could also be a server-side problem. Can we please also try to narrow down the server-side situation of people experiencing this issue ? Alexis
Flags: needinfo?(alexis.bezverkhyy)
> What value is shown at "Order Received" column for the null mail? A number of non-consecutive values between 2819965 and 28216890. > Is null mails disappear by "Repair Folder" of the local mail folder? No. > Does lost mail" appear by repair folder? No.
Flags: needinfo?(me)
I've now found out about & turned on TB's logging so I'll try to get you some better info next time it happens. But in the meantime, I seem to be getting two errors: Emails just "disappearing" - except that when I look in the database with a text editor they are there. Emails being listed, but they point to the content of the previous email in the receive sequence. For the most part neither come back when I repair the folder, but they do when I delete the index file & make TB rebuild it. In some cases the "faulty" email, though present in the database has the end missing. The occurrence of both problems started coincident with hardware problems on the ADSL link. Having fixed at least some of those the incidence is now much reduced. Web browsing also often plays up coincident with the TB "mail loss" & the old (just replaced) router often reported loss of link & occasionally ADSL connection at the same time. Which suggests that "dirty" comms is the trigger for TB's problems.
(In reply to Wayne Mery (:wsmwk) from comment #63) > (Paul's address bounces - bug reporter is gone) > > BTW, in case it's not obvious, there are, and have been, multiple causes to > losing data. Which means there are multiple bugs, and that this bug 462156 > can't address a fix to every one of you seeing dataloss - multiple bug > reports are needed. Which is why it is important for us to have information > (ref comment 59) from you that can be used to precisely determine what you > are seeing. With all due respect, I disagree. Indeed there may be multiple causes for a move to fail, but only one cause that Thunderbird allows it. And that is, it has coded a non-atomic move. No single database application would pass its first quality test with such an incredible bug. The data being moved, people's emails, are important to them, or can be, critically so, just as with file moves on a file system. And as a consequence a move should be coded atomically. What this means, is before a source mail item is ever permanently deleted an integrity check is done on the copy made is conducted. If it fails the operation is rolled back with an error and the original message stays intact. The failure to have such an atomic move coded is one and only one bug and the effort to find individual causes of move fails, which may indeed be a multitude and as Alexis Bezverkhyy points out involve serve-side problems as well. Might I ask what I'm overlooking here? Why is any effort to find causes being considered rather than the implementation of a simple atomic move operation with roll-back on failure. Subsequent to that failures become an inconvenience and not a crisis.
I have to second Bernd, I've already pointed in this direction in comment #57. There's no reason to seek for more and more evidence, the clearly evident fact is that thunderbird does not correctly handle errors on the server side or transmission line. If ever a Mozilla developer would have done an analysis of the code pathes running in such error cases, the bug would be clearly visible (I would have done it on my own if I had time and expertise and knowledge of the code base, but I haven't). So, use the code, Luke !
(In reply to Bernd Wechner from comment #67) > Indeed there may be multiple causes for a move to fail, but only one cause > that Thunderbird allows it. And that is, it has coded a non-atomic move. No > single database application would pass its first quality test with such an > incredible bug. > > The data being moved, people's emails, are important to them, or can be, > critically so, just as with file moves on a file system. And as a > consequence a move should be coded atomically. So, since you're obviously an expert, please tell me how to implement an atomic move across multiple databases. Hint: you can't. The best you can do is a copy-and-delete-if-not-failed, but that is certainly not atomic in several situations (particularly if the delete fails). > Might I ask what I'm overlooking here? Why is any effort to find causes > being considered rather than the implementation of a simple atomic move > operation with roll-back on failure. Subsequent to that failures become an > inconvenience and not a crisis. The best that can be implemented in a multi-database situation is effectively already implemented in Thunderbird: a move from IMAP server to local folders is treated as a copy, and, only if the copy succeeds, does the message get deleted from the server. What that means is that the reason why this is failing is because the copy is appearing to succeed when it is actually failing. And what we're trying to understand is *why* that is happening.
No definite diagnostics yet, but maybe a clue to what's happening.... I've been chasing comms errors (hardware line faults) for the last 3 months & in the interests of keeping things simple, only ever got emails for one account at a time. Over that three months there was not a single corrupted email. A week after the hardware fault was fixed I clicked "Get Mail" (unfortunately without starting the LAN monitor first) & simultaneously triggered the downloading of emails from the regular POP account plus two hotmail accounts plus two Yahoo accounts. One of the emails from the regular POP account had some large attachments & the email after that pointed to that emails content rather than its own. So maybe we're looking at a locking/timing issue ? Back to collecting LAN packets!!
I now have a Wireshark capture of the LAN packets, TCP Stream extraction, the Inbox and the Inbox.msf of an email corruption. Wireshark shows that the contents of the email came in on the LAN, and browsing the two "Inbox" files with a text editor shows that the content of the corrupt email never made it to the Inbox file, but the header info did make it to the Inbox.msf file. So this looks on the face of it like a pretty clear cut case of TB corrupting emails. This happened on Windows 7 64 bit (but apparently identical symptoms have previously occurred on XP) To whom do I send the diagnostic files for further investigation ?
Joshua, can Tom send his captured stream data exposing this bug to you (comment 71)? If not, can you recommend someone to whom such test data should be sent for analysis?
Flags: needinfo?(Pidgeot18)
(In reply to Tom Coxon from comment #71) > I now have a Wireshark capture of the LAN packets, TCP Stream extraction, > the Inbox and the Inbox.msf of an email corruption. > Wireshark shows that the contents of the email came in on the LAN, and > browsing the two "Inbox" files with a text editor shows that the content of > the corrupt email never made it to the Inbox file, but the header info did > make it to the Inbox.msf file. > So this looks on the face of it like a pretty clear cut case of TB > corrupting emails. > > This happened on Windows 7 64 bit (but apparently identical symptoms have > previously occurred on XP) > > To whom do I send the diagnostic files for further investigation ? Can I please have a copy of this Wireshark capture ?
(In reply to Thomas D. from comment #72) > Joshua, can Tom send his captured stream data exposing this bug to you > (comment 71)? > If not, can you recommend someone to whom such test data should be sent for > analysis? I'm too far removed from IMAP code in general to be much help, and my ability to catch up on this is extremely limited. For IMAP debugging, I'd recommend irving.
Flags: needinfo?(Pidgeot18)
irving, can Tom send his captured stream data to you?
Flags: needinfo?(irving)
Tom, you can send your data to me; I can't promise to figure out what's going wrong, but I'll take a look. Just to confirm, in your case - are all your accounts POP, or do you have a mixture of POP and IMAP?
Flags: needinfo?(irving) → needinfo?(fengshui)
Actually, for Tom's case, since it's POP mail, we should probably move that to its own bug; all the previous comments on this bug that specify a server type are about IMAP. For any of the IMAP users still listening, do you have the Folder Properties / Synchronization / "Select this folder for offline use" checkbox set? If not, does setting that checkbox and synchronizing the folder before trying the move make a difference? As WADA explained in comment 59, you may be able to test this more safely on IMAP by creating a new folder on the IMAP server, copying a bunch of messages into that folder on the server, and then trying the moves from that folder to local folders.
Thanks Irving - Log info sent as requested. I've just doubled checked the mail servers & they're all POP
Irving comment #77: See comment #57
I think that there might be a way to recreate this one at will (or at least sufficiently often that it gives a developer a fighting chance to locate what's going on). I just downloaded about 20 emails & got 3 emails where the header pointed to the contents of another email. The computer was VERY busy at the time doing nightly backups, plus scanning an SD card for deleted files, and on top of that one of the emails had a couple of large photos attached. So the key to reproducing the problem seems to be to load the test system with I/O (& maybe processor) to the point where it's beginning to struggle, & then download the emails.
Flags: needinfo?(fengshui)
I've just found this page after experiencing a loss of emails when moving them to a local folder in Thunderbird 24.5.0. I am no expert, just a user. I use IMAP, I moved 5 emails and they then appeared in the local folder with no subject, no recipient, no text, size of 0.1 KB. If I understand correctly what is discussed above, these emails are now permanently lost. Is that right? This is a serious problem for me, as I don't know the senders. The workaround appears to be either to only move single emails, or to copy instead of moving. Is that right? And a third question - how is it that Thunderbird has such a serious, unfixed problem?
See Also: → 923387
(In reply to Joshua Cranmer [:jcranmer] from comment #69) > So, since you're obviously an expert, please tell me how to implement an > atomic move across multiple databases. Hint: you can't. The best you can do > is a copy-and-delete-if-not-failed, but that is certainly not atomic in > several situations (particularly if the delete fails). Apologies for a very tardy response. I make no claim to being an expert and the accusation isn't exactly cordial. But as it stands, if you think the word "atomic" does not cover "copy-and-delete-if-not-failed" then you have another definition in mind to that which I learned studying database design 20 years ago and which the FOLDOC mirrors: http://foldoc.org/atomic or: An atomic ... transaction is one which is guaranteed to complete successfully or not at all. If an error prevents a partially-performed transaction from proceeding to completion, it must be "backed out" to prevent ... an inconsistent state. In short your "copy-and-delete-if-not-failed" paradigm can be improved if lengthened to: copy-and-delete-original-if-not-failed-else-delete-copy-if-it-exists And that satisfies my understanding of atomic. That is the move works completely or it does not work at all, no partial results tolerated. And this is possible within limits with no real excuses and the lack of it smacks of poor design in the first instance which fails to appreciate the critical importance of the data being moved. I repeat if a filesystem move results in corruption users would be up in arms, even if it were seldom and hard to reproduce and of unknown causes. They would demand an atomic operation, one that guarantees the data being moved remains present somewhere. A classic relational database implements this with the transaction and rollback ideas. You can begin a transaction, and if content end it and the results are committed, but if not, roll it back and the state of the system is as it was before the transaction started. A filesystem can do same (and good ones do I am confident) and so can mail systems. This is not without cost. There is a performance cost to verification and in a secure move operation a mail client would: 1) copy source email to destination 2) read the destination email back again 3) compare with source 4) if identical, deleted source, if not, delete destination and report error. Step 2 and 3 come at a cost and in any environment that is "pretty secure" it's easy to imagine a programmer skipping it. But here is the crunch and what puzzles me. It does not and should not, matter what is going wrong, and secure design is not predicated on identifying all the possible sources of failure and handling them. To be sure it is prudent to identify as many as you can and handle them elegantly, but secure design predicates a failsafe mechanism and this clearly isn't implemented in the Thunderbird move. And my point is, chasing the error is both difficult (as it is intermittent) and not the right thing to do. Fixing the move operation to be failsafe is the right thing to do. If the cost of failsafe is too high say (the speed of moving drops by an order of magnitude or more) it is entirely OK to implement a fast and pretty safe move and a slow(er), but guaranteed safe move. Totally fine with that. It would mean at least if someone using a standard move suffers this failure (finds only empty emails at the destination and the source ones gone) and sheds a tear at their loss, they can at least in future over that link use the failsafe move with confidence and wear the performance cost. I mean that is just one failsafing method. Another commonly used is the idea of a trash can. So Thunderbird never ever deleted source emails but moved them to the local trashcan instead (catch 22 hear as that needs to be reliable) then at least if the move failed the originals could be recovered from the trash can. > The best that can be implemented in a multi-database situation is > effectively already implemented in Thunderbird: a move from IMAP server to > local folders is treated as a copy, and, only if the copy succeeds, does the > message get deleted from the server. > > What that means is that the reason why this is failing is because the copy > is appearing to succeed when it is actually failing. And what we're trying > to understand is *why* that is happening. It also means the that test for success is not robust. If Thunderbird reads back the headers and they check out, great, but not if the body didn't copy. A robust test for success reads back the entire message and compares with the source. As observed above, there are reasons not to be robust in this sense and a plethora of compromises that can be entered into that delegate trust to assumptions or other services but in so doing they depart from failsafe. To save time Thunderbird could for example calculate a SHA-1 hash on the copy it has, and ask a remote service to return the SHA-1 has of its copy. Alas in so doing it delegates trust to the remote servers SHA-1 hash reporting and isn't failsafe any longer. Look, I deeply appreciate anyone who's dedicating time to this and I don't in any way mean to be critical - heck I don't have time to get Thunderbird building locally here and work out how to contribute code. I may be misunderstanding something and am open to learning. But much of what I'm sharing here is in a sense, the philosophy of failsafe implementation and not a programming question, and what is clear is that Thunderbird's move is not failsafe, and I think it should probably have one. When moving masses of emails, like whole mailboxes, it is VERY desirable to use a failsafe tool. Regards, Bernd.
i already moved my emails from inbox to local folder the problem is only half of them were moved the rest are dated by time that i moved them to local folder and they are empty how can solve it please.
Flags: needinfo?(paul.barlow)
I have the following notes from 2/7/2014 1. tried to move two messages from vseerror Inbox to vseerror moz-crash 2. move failed because of quota exceed. 3. raised quota 4. moved messages 5. attempted to move message from vseerror to moz-archive3 local folders 6. corrupt moz-archive3 and imap dataloss
Flags: needinfo?(paul.barlow)
Mike is no longer using TB. Victor is no response. Darrin was able to reproduce with "30 plus at messages" at some point in the past. AllUses writes "Haven't seen this issue for some time. The last months i'm mostly connected using broadband compared to UMTS before".
Flags: needinfo?(victor)
Flags: needinfo?(mike)
Flags: needinfo?(darren.goddard)
Flags: needinfo?(AllUses)
We are just not complaining anymore, since the evidence has been clearly shown, no need to further investigate if this is "real". If just anyone would finally try to fix this instead of looking for reasons to close this bug....
I don't think Wayne was trying to dispute that, he was just clearing the needinfo requests since the information has been given. I also had this bug with a client and we haven't found a solution yet, despite debugging.
I was able to determine that the source of this issue for me was actually a tool called DavMail. DavMail is an email gateway which sits in front of an Exchange server and exposes an imap interface. It can communicate with Exchange in one of two ways either via WebDav or using the EWS REST interface. I found that when it was using Dav if the server errored DavMail would return a blank email. This seemed more likely to happen when moving a large number of emails and it would result in what looked like Thunderbird corrupting emails. I was able to resolve the problem by switching DavMail to use EWS. (I've also reported the issue here http://sourceforge.net/p/davmail/bugs/574/)
(In reply to Philipp Kewisch [:Fallen] from comment #87) > I don't think Wayne was trying to dispute that, he was just clearing the > needinfo requests since the information has been given. I also had this bug > with a client and we haven't found a solution yet, despite debugging. Yes, quite right. This bug has been open a long time. As a result any users are cc on the bug who may no longer see the problem, or whose issues don't actually all match. So I'm peeling the onion to get to the core of users who have useful information, on the same exact problem. N.B. comment 84, meaning I also occassionaly experience this issue and care about getting a resolution. (my apologies to others for the following comment, which doesn't help solve this bug) (In reply to beanladen from comment #86) > We are just not complaining anymore, since the evidence has been clearly > shown, > no need to further investigate if this is "real". If just anyone would > finally > try to fix this instead of looking for reasons to close this bug.... Your frustration is understandable, but frankly your comments here have not been helpful. Given that you have already authored bug 792198 I suggest you persue your issue there.
Component: General → Backend
Keywords: steps-wanted
Product: Thunderbird → MailNews Core
See Also: 923387 →
Whiteboard: [needs reproducible steps][testcases: comment 35, comment 38] → [needs reproducible steps][testcases: comment 35, comment 38, comment 84][tools: comment 52]
Version: 2.0 → 1.8 Branch
Hi all, Same problem here, having reached the quota allowed by my IMAP server, i wanted to move my emails from the virtual archive to an archive in local folder. My emails disappeared from the server, disappeared from the thunderbird virtual folders but most of them appeared blank in the local folder : no sender, no title, nothing. Lost a huge part of my work with thoses emails In the profil folder/local folder, they now appear in the file without extension as : From - Wed Feb 11 13:59:35 2015 X-Mozilla-Status: 0003 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Wed Feb 11 13:59:35 2015 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Wed Feb 11 13:59:35 2015 X-Mozilla-Status: 0011 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: I tried restart in safe mode, repair folder but it didn't change anything. Didn't try to erase index file, too afraid to do it. Strangely enough a 443 Mo file is still in the ImapMail folder in which they originally were before I moved them. Is there any hope to restore my datas to a previous state ? Thanks for helping, those mails were professional ones and this date loss puts me in a dire situation Sorry for my grammar mistakes, I am not an english native speaker
Adding to this thread, I have experienced the same issues - SMTP server, moving emails to a local folder, all those emails in a single downloaded batch were blank, in exactly the way described here. It's farcical that after all this time this is still considered unconfirmed. I have switched the filter to copy instead of move, but that is a short-term fix. If this does not see resolution soon, my fix will be to stop using Thunderbird. Given the years this thread spans, I suspect that that will be my course of action, which is a shame, because this is the only problem I have had with Thunderbird.
To clarify, I meant IMAP, not SMTP. I experienced no issues until my company switched from POP3 email to IMAP.
And I should note, that this is with version 31.5.0
This also happened to me. I'm on version 31.5.0. I moved hundreds of email messages from an IMAP account to a local folder and about 80% of them turned into blank messages. I've lost all this email permanently. I'm not sure if this might be related, but a few seconds after the move seemingly completed and my mail got deleted from the IMAP account, I closed Thunderbird, which still had the busy mouse cursor on it. Could it be at the time still recording changes to the local folder from memory and closing it prematurely causes this bug? This is a really grave bug as it causes the permanent loss of information. How is this around since 2008?
It might be a cause, but it's not the only one. In my case I'll be running my filter list against an IMAP account and the filters will just stop running. It'll finish the "moving X of Y messages" bit, and then the status bar will go blank. So it seems that Tbird notices something's wrong but doesn't have the ability to tell me. At least now I know that I have to (1) not purge the folder, (2) quit, (3) figure out which folder it died on, (4) delete the MSF and manually remove the trash from the end of the file, (5) restart Tbird and manually run the filters that move to that folder, (6) run the duplicate-deleting extension, (7) purge, (8) run all filters again. But hey, it only wastes about 10 minutes of my time a week and as long as it doesn't croak on the last folder I'll at least catch what's happening.
This sounds similar to a problem which I've been having with POP3 accounts for about a year now(on 3 machines with 2 different OS's - TB is the only common factor) Basically if the TB computer is busy for some reason when I'm receiving Email, TB lists the subject of each email correctly, but sometimes index's the body of the preceding email. A couple of nights ago the system was exceptionally busy & instead of the odd email being affected there were 5 occurrences in 30 emails & in each occurrence, instead of one email indexing the body of the preceding one, two emails did (i.e. emails 2 & 3 both indexed the body of email 1). Speaking as someone who spent 20 years working with multi thread system software, this is strongly suggestive that some TB processes are using wait loops and assumptions to synchronise between processes rather than using a system of setting, checking and releasing locks. Why is it showing now? Possibly because the comms, servers and receiving machines are all getting faster & have whittled away the safety margins in the wait loops. The problem referred to by nf.pereira sounds similar, the "close" process shutting everything down without checking that all other processes had completed, and the "delete" function doing the delete operation without checking that the copy had first completed successfully. Sorting this probably won't be easy (as it seems to be quite widespread) but it is SERIOUS and needs to be addressed as a matter or urgency or there is a real danger that TB will lose it user base.
PS One of the software platforms is Windows 7, so this issue is very unlikely to die when XP does.
(In reply to Tom Coxon from comment #97) > PS One of the software platforms is Windows 7, so this issue is very > unlikely to die when XP does. My problem occurs in Windows 8.1 too. (In reply to Tom Coxon from comment #96) > The problem referred to by nf.pereira sounds similar, the "close" process > shutting everything down without checking that all other processes had > completed, and the "delete" function doing the delete operation without > checking that the copy had first completed successfully. > Sorting this probably won't be easy (as it seems to be quite widespread) but > it is SERIOUS and needs to be addressed as a matter or urgency or there is a > real danger that TB will lose it user base. Agree. Executing a delete function without a confirmation of a correct operation is a very grave bug. I can't trust my Email with Thunderbird if it destroys part of it with no warning. The explanation by Tom Coxon seems plausible.
"Executing a delete function without a confirmation of a correct operation is a very grave bug." As far as we know, the code is designed to confirm the copying to the local folder before the deletion is done on the IMAP server. Obviously that is failing in some case, but we do not know where or how. What this bug needs is for someone to generate a reproducible test case, that is a sequence of operations in a test profile that a developer could also do that demonstrates the problem. Without that test case, trying to fix this is looking for a needle in a haystack. It would be very helpful if those of you who have experienced this, could generate a test profile and experiment with it, seeing if you can reliably make this failure happen.
As far as I've been able to tell, there is no reproducible test case -- sometimes it happens and sometimes it doesn't, and it's independent of anything the user does. So, as far as steps go: Use IMAP for a while, copying messages to local folders, until it bites you. If someone can point me toward being able to log what Tbird is doing while it's running filters (my particular failure case) I should be able to provide something useful within a week or two. I'm actually a little overdue for this bug to show up right now.
Common test case base conditions are repeated several times in the 100 above comments: the following steps will apparently cause the problems sometimes, but not always: (see comment #51, for example) * Have a lot of files on the IMAP server * select a lot of them and move them from the IMAP server to a local folder * Afterwards, check to see if there has been any data loss / loss of message content It is possible (but not proven) that having a slow / high latency connection to the IMAP server may exacerbate the problem(s) It is also possible (my speculation, but not demonstrated) that the problem may be related to the IMAP server software. But no regular email use can test this, as you really need to: (i) setup a test IMAP server (or multiple servers with different S/W known to manifest this bug), (ii) load it with 100s of MB of data, (iii) set up network connectivity that you can control to reduce speed / add high latency / network interruptions, (iv) and then start playing at transfers over and over again (reloading the IMAP server each time to restart), (v) dumping MBs of diagnostics, (vi) all documented against a defined set of test cases. Even just doing (i) and (ii), with one server, would be hard. If you seriously want to squash this, you need a proper test environment (and someone committed to the work), with test cases built based on the reporting in this bug.
(In reply to Ian Graham from comment #101) > It is possible (but not proven) that having a slow / high latency connection > to the IMAP server may exacerbate the problem(s) > > It is also possible (my speculation, but not demonstrated) that the problem > may be related to the IMAP server software. I can attest that my occurrence described a few posts above happened on the Outlook.com's IMAP server which is extremely slow. Another thing you might want to test is copying a big number of emails from an IMAP folder to a local folder and close Thunderbird (not forcefully) during the process.
It sounds like many have experienced this after manually initiating a copy, so I will assume that this bug does not require filters to be seen. If filters are required that complicates things. Just to be clear, this bug is only about IMAP to local copies. I keep seeing the phrase "downloading to local folders" and I don't understand what that means. IMAP messages are not downloaded directly into local folders. In a cross-server move like this, if I remember correctly each file is temporarily copied to a local file, and then that local file is copied into the mbox folder for the local storage. In comment 77, Irving asked the question "do you have the Folder Properties / Synchronization / "Select this folder for offline use" checkbox set?" That significantly changes the codepath involved, and I was also looking for evidence of that in the comments before I saw his question. Can those of you still commenting please answer this? I don't see the point of leaving this unconfirmed.
Status: UNCONFIRMED → NEW
Ever confirmed: true
At Kent James #103: Thanks for finally accepting this bug as a real bug... "Downloading to local folders" means option "sync to local folders" for IMAP. The problem is leveraged a bit when using this option, but not completely (I was hit again with this option active 4 days ago when moving and loosing to Trash). So a test will be more likely "successful" with this option turned off. The best test will be on a slow IMAP server that interrupts connections for a long while after a few connections (I have a DSlite cable connection which does that on a "regular" basis for unknown reasons). I don't expect that testing will help a lot, tracing the code should be a more direct way to find the wrong code path, as it seems obvious that the sequence of operations is clearly wrong. It's not so obvious how to really fix that, as the check operations for existence or removal on the remote server could also be interrupted or just hang, or the user just gets impatient and closes thunderbird (and shutdown his computer at all after), so this will be a bit trickier than I thought before.
(In reply to Kent James (:rkent) from comment #103) > In comment 77, Irving asked the question "do you have the Folder Properties > / Synchronization / "Select this folder for offline use" checkbox set?" That > significantly changes the codepath involved, and I was also looking for > evidence of that in the comments before I saw his question. Can those of you > still commenting please answer this? > > I don't see the point of leaving this unconfirmed. The bug happened to me in folders with that option checked BUT, in my account settings I have TB set to only keep the most recent 30 days of mails. So to my understanding, all my older than 30 days emails were not stored locally.
Thanks Kent. Per your questions in comment 103 - yes this is all w.r.t. interaction with an IMAP server. In my case the remote server INBOX folder is "selected for offline use." No automated filters are used. By "download to local folders" I mean the following: * manually select a large number of INBOX message items (100++) * drag the selected items to a folder in "Local Folders" * wait for the activity to complete. * check local folder content and see if entries were correctly moved from server.
(In reply to Ian Graham from comment #106) > * wait for the activity to complete. In fact, I recommend you also test closing TB (non forcefully) during the copy operation.
(In reply to beanladen from comment #104) > "Downloading to local folders" means option "sync to local folders" for IMAP. I'm also not familiar with this phrase "sync to local folders". Can you give precise instructions of where you see this phrase in the Thunderbird user interface?
(In reply to Ian Graham from comment #106) > Thanks Kent. Per your questions in comment 103 - yes this is all w.r.t. > interaction with an IMAP server. In my case the remote server INBOX folder > is "selected for offline use." No automated filters are used. Another user mentioned that the option to limit offline storage to 30 days was set. Do you also have a limited storage span, or are you storing all messages offline?
(In reply to Kent James (:rkent) from comment #109) > (In reply to Ian Graham from comment #106) > > Thanks Kent. Per your questions in comment 103 - yes this is all w.r.t. > > interaction with an IMAP server. In my case the remote server INBOX folder > > is "selected for offline use." No automated filters are used. > > Another user mentioned that the option to limit offline storage to 30 days > was set. Do you also have a limited storage span, or are you storing all > messages offline? I have no size or time/age limits on my IMAP (online) or offline (local storage) folders.
(In reply to Kent James comment #108) > I'm also not familiar with this phrase "sync to local folders". This has been renamed a couple of times, currently it's in "Account Settings"/ "Synchronization & Storage"/"Synchronize all messages locally regardless of age", together with "Don't delete any messages". This effectively gives you a POP-like account and minimizes the risk, as there's a higher chance a local copy stays there even if the operation failed remotely, but still not 100% safe. But the problem more likley appears when using the steps in comment #106 or moving between two remote folders without local cache or copy.
(In reply to beanladen from comment #111) > (In reply to Kent James comment #108) > > I'm also not familiar with this phrase "sync to local folders". > This has been renamed a couple of times, currently it's in "Account > Settings"/ > "Synchronization & Storage"/"Synchronize all messages locally regardless of > age", > together with "Don't delete any messages". This is a more limited version of saving messages for offline use (wish we had a single word for that). The bug though does not occur when you "sync to local folders", the bug occurs when you move messages from an IMAP server (where the messages are already downloaded for offline use, according to all current bug reporters) to a local folder. What is confusing to me is that the reports imply that this is correlated with poor IMAP performance, and yet in the described case, there is little to no interaction with the IMAP server in the critical period where the messages are copied from the IMAP folder to the local folder. The first stage of the move (where the failure must occur) involves grabbing the locally stored copy (which presumably is good, or the message would display incorrectly in the IMAP folder), moving that to a temporary local file (or so I believe, I would have to confirm in the code), and then moving that data in that local file to the local folder MBOX. It is that copy that is silently failing somewhere. Is it possible that this failure is instead correlated with an overloaded local file system, rather than a poor IMAP connection? That would make more sense.
> where the messages are already downloaded for offline use This is not the case for me. The checkbox "Keep messages for this account on this computer" is not checked. There are no local copies of mail that is on the IMAP server, as I've verified just now by looking in my profile.
Just to be complete on my settings for comment 106, here are all the potentially relevant server & Local folder settings (lord knows which ones may be relevant): Note also there are no quotas / limits that I know of on the IMAP server. <Remote server settings> ======================= Server settings * Connection security: SSL/TLS * Authentication method: normal password * Check for new messages at startu * check for new messages every 5 minutes * When I delete a message: move it to the Trash folder on the remote server * DO NOT "Expunge" Inbox on exit * Empty trash on Exit Copies & Folders: * When sending messages, automatically: * Place a copy in: "Sent" folder on Local Folders * Place replies in the folder of hte message being replied to * Message archives: * Keep message archives in Other: (Archives on Local Folders) Junk Settings: * Enable adaptive junk mail controls. Do not marke as junk if sender's name is in Personal Address book Synchronization and Storage: * Keep messages for this account on this computer * Synchronize all messages locally regardless of age * Don't delete any messages * Always keep starred messages <Local Folder Settings:> ======================== Junk settings: * Enable adaptive junk mail controls. Do not marke as junk if sender's name is in Personal Address book Disk space: * Don't delete any messages * always keep starred messages
(In reply to Kent James comment #112) > The bug though does not occur when you "sync to local folders" ... It does occur also in this case for me, moreover I'd never "offline use" switched on. It occurs more likely when none of "sync" and "offline" are active, this was my original configuration with which I encountered that problem very often (and switched to "sync" to get less problems). I think that this "offline" or "sync" stuff is not that relevant to the bug, but masks the problem somehow, the real bugger is the remote interaction part.
(In reply to beanladen from comment #115) > (In reply to Kent James comment #112) > > The bug though does not occur when you "sync to local folders" ... > It does occur also in this case for me, moreover I'd never "offline use" > switched on. I see my wording was unclear. What I meant was that whatever the failure is, it is probably not occurring during the interaction with the IMAP server to get the initial message, but rather in the process where a message is put in a local file, and then that file is copied into the mbox folder. I haven't actually looked at the code yet wrt this particular bug, I'm just trying to reduce the size of the haystack that I have to look in. At the moment, my suspicions are leaning toward a missing error check on a local file operation.
(In reply to Kent James (:rkent) from comment #116) > ... > I haven't actually looked at the code yet wrt this particular bug, I'm just > trying to reduce the size of the haystack that I have to look in. At the > moment, my suspicions are leaning toward a missing error check on a local > file operation. in that case, hopefully Ishikawa to the rescue
Flags: needinfo?(ishikawa)
Blocks: 792198
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #117) > (In reply to Kent James (:rkent) from comment #116) > > ... > > I haven't actually looked at the code yet wrt this particular bug, I'm just > > trying to reduce the size of the haystack that I have to look in. At the > > moment, my suspicions are leaning toward a missing error check on a local > > file operation. > > in that case, hopefully Ishikawa to the rescue Dear Wayne, Thank you for making me feel like Batman :-) [and you can call me Chiaki. Chiaki is my given name, and Ishikawa is my family name. Oh well, the order of names and how they are written down is hard deal with in cross-culture setting as in the Internet, isn't it?] Seriously, I noticed quite a few places where the network errors were not properly caught in POP3 code, and I suspect Imap code also has such deficiencies. Some W-I-P patches are in Bug 1122698. Bug 1134527. Bug 1134529. (A few Imap source files were also touched, but imap changes are not exhaustive and the investigation is incomplete.) As for moving a message from one folder to the other, I recall a hand-crafted file copy/file append was buggy and fixed, but it still has an issue of failure to retry in the face of short read/write caused by transient network error. I mean TB can try harder to complete the complete copy, but may fail prematurely without retrying. The bugs and fix history are as follows. Bug 842267 - copy or move messages from IMAP folders to Local Folders on Linux Thunderbird 17.0.2 results in bad index file, fixed by Repair I was fixing a different bug when I noticed the file error not handled correctly and I hope I fixed the above with the next patch. Bug 931720 - Returning low-level error correctly from nsLocalFile::CopyToNative() (fixed) Bug 936987 - Catch and propagate PR_Close() error in CopyToNative() (fixed) [this one is for UNIX.] In the patch for 936987, I left the following comment. http://mxr.mozilla.org/comm-central/source/mozilla/xpcom/io/nsLocalFileUnix.cpp#919 919 // TODO/FIXME: If CIFS (and NFS?) may force read/write to return EINTR, 920 // we are better off to prepare for retrying. But we need confirmation if 921 // EINTR is returned. I did not introduce the EINTR handling (basically retrying) when the low-level error code is EINTR. This is because I was not familiar with the code base. The same functionality is implemented by using a system call under Windows and this Win32 API seems to handle the low-level network errors, recovery by retrying and everything under the hood and so we don't have to worry about it in the case of Windows. [This matches the observation by Eric Valette that the corruption issue does not happen under Windows!] So I can think of at least a few issues: - POP3 download code does not catch errors completely. I have tried to do this in Bug 1122698. Bug 1134527. Bug 1134529. and tested them to see that some places where code was added DID trigger an error when network error was introduced. - By code inspection, I did introduce a few error checking in Imap download code, but that is far from exhaustive. Unlike pop3 case, I have not done testing on imap side very much yet. I believe IMAP code also have many places where file and network I/O errors are not properly checked :-( Moving messages that depends on |CopyToNative()| patched in Bug 936987 may not perform as best as it can possibly do: this is because it lacks the retrying in the face of short read/write (data read/written is shorter than expected) when network error occurs (or even scarce system resources issue) and low-level EINTR code is returned: EINTR is not handled today there. Now after all these years and bug reports, I think it should. The error code from CopyToNative() may not be handled well by Imap code also. More investigation is needed. To be frank, I am appalled to hear that Imap has such serious mail message corruption issue today. My office is thinking of using an external imap-based e-mail service. That is not a good move if I want to stick to TB for now :-( Maybe fixing the points outlined above can erase many simple bug cases, I hope. Any thought? TIA
Flags: needinfo?(ishikawa)
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
I add some info on that bug, hoping will be useful to fix it. Server: IMAP, dovecot 2.1.7-7+deb7u1 (debian wheezy), SSL Client: Thunderbird 38.3.0, Windows 7 x64 I've instructed some of my users to ''archive'' email moving them to local folder; in that particular case, local folder is on a network share (samba), backed up on client by CSC (offline file and folders). I instruct users to move email ''one by one'', but effectively they wait till there's no more room on imap server and move hundred of message in one time. A user complain me that they ''cannot find a mail''. After fiddling a bit, i've found, on destination ''archived'' folder, the infamous empty email: From - Wed Jun 10 10:58:36 2015 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: the date is the moving operation date. The user complain about missing emails, but seems there's no apparently ''holes'' in the mailbox, so i need to investigate more on that. Clearly, i'm a bit scared about that bug... Thanks.
This is quite an amazing bug, given that it's going on 7 years old now. That must be some kind of a record! After reading through this entire thread, I must say the most thorough analysis seems to be that in comment #58 by Alexis Bezverkhyy. According to him, this is a server-side problem that Tbird fails to catch because it does not check its local copy of the downloaded messages before committing them to the local final destination folder. Or, as Bernd Wechner put it in comment #67, it does not perform an "atomic" database operation. In my case I have a buggy internet connection, but I also have my INBOX selected for offline use with no restrictions on how long local copies should be kept and the "never delete" box checked as well. That should mean (especially since the messages I wanted to download are nearly a year old) that dragging messages from the INBOX to the local folder should result in the locally stored copies of those messages being moved to the destination folder, with no server interaction at all save for an instruction to delete the server's copy after a successful move. Nevertheless, the time it took and the little messages at the lower left of Tbird's main window indicate that Tbird downloaded all the messages again from the server, rather than just moving the local copy that was already on my hard drive. Need I point out that this is really dumb? If there's already a copy of the messages on my hard drive, why doesn't Tbird just do a local copy, rather than a really slow and apparently error-prone re-copy of the server's version of the messages? Good grief!
(In reply to Marco Gaiarin from comment #121) > I add some info on that bug, hoping will be useful to fix it. > > Server: IMAP, dovecot 2.1.7-7+deb7u1 (debian wheezy), SSL > Client: Thunderbird 38.3.0, Windows 7 x64 > I am hoping that I can check this imap bug since the office where I work may want to use gmail as an infrastructure and I hear that gmail is better used as imap server rather than pop3 server. I don't want to use a mail client with a bug of this seriousness. Right now I am trying to get the patches in Bug 1122698. Bug 1134527. Bug 1134529. landed on the source tree. Then, I am quite sure, that some errors are caught. But with my patches on the tryserver, I hit upon a set of strange errors under Windows and when Maildir storage format is used. Can it be possible that some of the reporters have used Maildir storage format instead of mbox? (Reading through the comments, I notice X-MOZILLA-* header lines, and so most of the comments are for mbox style storage.) But my seeming trivial patch (from my point of view) triggering a strange error under Windows (xpcshell-tests failed for some imap-related tests, for example) DOES suggest something is wrong in the Windows-support code. Are there LINUX TB users who reported this bug in this thread ? I already notice that the Windows-support code is not uniform in handling long vs short file name Look at the paths below. They got reported by my debug code when a warning or error was noticed. c:\\users\\cltbld\\appdata\\local\\temp\\XPC-PR~1 c:\\users\\cltbld\\appdata\\local\\temp\\xpc-profile-5yfawz\\ImapMail\\localhost\\INBOX\\cur\\1450206446958000 In one part of the code, Windows-support code seems to prefer to use the short path as in XPC-PR~1. This is done despite the warning produced by DEBUG version of TB. e.g. The line starting with "(debug)" is printed by my own modification of the code. 11:07:27 INFO - PROCESS | 6572 | (debug) EnsureShortPath: mShortWorkingPath is empty 11:07:27 INFO - PROCESS | 6572 | [6572] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/toolkit/crashreporter/nsExceptionHandler.cpp, line 2339 11:07:27 INFO - "Running test with maildir" 11:07:27 INFO - (xpcshell/head.js) | test MAIN run_test pending (1) 11:07:27 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2) 11:07:27 INFO - (xpcshell/head.js) | test MAIN run_test finished (2) 11:07:27 INFO - running event loop 11:07:27 INFO - mailnews/imap/test/unit/test_imapFilterActions.js | Starting setupIMAPPump 11:07:27 INFO - (xpcshell/head.js) | test setupIMAPPump pending (2) 11:07:27 INFO - PROCESS | 6572 | [6572] WARNING: This method is lossy. Use GetCanonicalPath !: file c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/xpcom/io/nsLocalFileWin.cpp, line 3459 11:07:27 INFO - PROCESS | 6572 | (debug) nsLocalFile::GetNativeCanonicalPath: aResult became <<c:\\users\\cltbld\\appdata\\local\\temp\\XPC-PR~1>> I have a tough time to figure out why this short name is used. Since Win95 is no longer supported, maybe we can forget about using short path. I don't know if all CIFS/SMB servers support producing short names, even. Also, since the errors my patches triggered on tryserver seem to happen in so called async tests (e.g.: see https://treeherder.mozilla.org/logviewer.html#?job_id=13779&repo=try-comm-central This may disappear a few weeks later.) I have a feeling that there may be issues related to the strange mix of short vs long path names, asynchronous handling not done correctly, AND maybe stack overflow or maybe the time allocated to xpshell-tests is too short. The stack overflow is suspected because I saw in the above log the following error messages in one failure case. 11:07:31 INFO - PROCESS | 9196 | ******************************************* 11:07:31 INFO - PROCESS | 9196 | Generator explosion! 11:07:31 INFO - PROCESS | 9196 | Unhappiness at: undefined:undefined 11:07:31 INFO - PROCESS | 9196 | Because: 2147500036 ... As you can see that, except for the long vs short path name issue, async issue, possible stack overflow, test time running out are all very context-dependent and hard to reproduce in real life and so my problems I face during testing on tryserver may indeed shed some light on the problems seen here. I thought that was great. Right now, though, I think there is an infrastructure problem on tryserver which I am afraid won't be fixed until the holiday season is over, and I have to wait until then to continue investigating. Me, too, noticed a few dataloss issue (a buggy compaction ate half my mail folder in 2008), and ever since, I started to send patches to fix such serious issues. TIA
(In reply to ISHIKAWA, Chiaki from comment #123) I can't help so much on this, only two little note: > Can it be possible that some of the reporters have used Maildir storage > format instead of > mbox? I can confirm i'm using mbox format server side. I've tried to narrow down with users the error: seems to happen ONLY when users put under heavy stress the server (and the client); both the time the bug happen, users are trying to MOVE 6-month of messages (some hundred) from INBOX or Sent to a TB local folder. For now, i've suggested to user to move only some messages at a time. Thanks.
(In reply to Marco Gaiarin from comment #124) > (In reply to ISHIKAWA, Chiaki from comment #123) > > I can't help so much on this, only two little note: > Thank you for your information. > > Can it be possible that some of the reporters have used Maildir storage > > format instead of > > mbox? > > I can confirm i'm using mbox format server side. > > > I've tried to narrow down with users the error: seems to happen ONLY when > users put under heavy stress the server (and the client); both the time the > bug happen, users are trying to MOVE 6-month of messages (some hundred) from > INBOX or Sent to a TB local folder. > For now, i've suggested to user to move only some messages at a time. > I am a little confused. So does this mean the particular user's folder (Inbox or Sent) is stored on a remote file server when the users tries to move it to "local folder" which is either on a local disk or a remote file server? Copying to/from remote server may cause timeout and other network-related errors especially if we overwhelm the server, and the current unpatched TB code is full of error NON-handling issues :-( The patches in the following bugzilla at least try to IMPROVE (not completely solve)this situation. I hope these help at least some error cases. Bug 1122698. Bug 1134527. Bug 1134529. > Thanks. Thank you for the info. I think I better check my patches for a possible subtle issue with Maildir format.
(In reply to ISHIKAWA, Chiaki from comment #125) > I am a little confused. > So does this mean the particular user's folder (Inbox or Sent) is stored on > a remote file server when the users tries to move it to "local folder" which > is either on a local disk or a remote file server? Yes, INBOX and/or Sent folder are stored on my Dovecot server, using MBOX as mailbox format. Local folders are stored mainly on a remote file server (samba), but a user that catch the bug have enabled CSC (Client Side Cache, eg 'Offline files and Folders'), so write operation could be happened on local disk and then propagated on server (i suppose).
I've had a similar problem for years, at least since 2011. Currently on Mac OS X, but similar corruption can occur with Windows and Linux versions, and it even occurred when I tried an alternate email reader (can't remember the name) that was a split off from the thunderbird code. When you have low-quality internet access, or are really punishing the server (downloading hundreds or thousands of messages for multiple accounts all at the same time), it seems to be particularly bad. When you only work with one message at a time with a good internet connection, it seems to be a pretty rare phenomenon. The IMAP synchronisation sometimes gets confused. It happened to me today (first time in a while), 45.2.0, Mac OS X. (But as per above, this is a very very old issue, and is cross-platform.) In the listing of all the email headers, the correct information was shown; however, when I opened the message in its own tab, it was a different message. This had to be a synchronisation issue; the message was displayed correctly on other computers and on my phone. I've seen variations on this, for example, when there is a big JPEG file attached to a message, sometimes only part of it is downloaded, and viewing from the affected computer shows only the top part of the picture, whereas it displays correctly on other computers. The "repair folder" function under the "Properties" menu item for the folder seems to fix it, at least some of the time. (It fixed the problem I observed today.) The much more serious problem occurs if you decide to move the message, without noticing that it hasn't synchronised properly. Then Thunderbird copies the wrongly or partially synchronised message to the destination local folder, and deletes the correct message from the server. Bye bye, your message is gone forever. As long as we have a finite speed of light and less than perfectly reliable internet, I suppose it's not possible to have a client inbox that is always perfectly synched with the server, although it might be possible to have it synched with a very high degree of reliability. I don't know anything about the IMAP protocol or how IMAP servers work, and whether they support atomic operations. I think ideally, when moving a message from an IMAP folder, the email reader would lock the corresponding message on the server, do some sort of check that the synchronisation has been done correctly, and then and only then, move the message to its new destination and delete it from the server. But I don't know enough about IMAP to know whether that is easy, or even possible.
Linux Mint 17 Qiana Thunderbird 45.2.0 Because of this 6 year old bug I lost *many* emails today; some of which were important for me. I seriously doubt intentions of the team when such a major bug exists in code for at least 6 years. Sorry to say, Thunderbird is not ready for production. Either fix this bug OR sleep the project and leave it. Thanks for understanding the frustration.
Ravi - I've got to agree. I like Thunderbird, I really do, but "randomly shreds emails" is pretty much fatal for an email reader ...
I'm not sure this is the same bug or not, but it certainly seems related. I also have the problem of mail data loss and/or blank messages. I've been using POP for years without this problem (to the best of my bad memory). I decided to give IMAP a try recently. I told it to fetch headers and leave the email on the server. When I went to read a specific message, it told me it needed to download the body... but the body never came. Result was a blank message. I checked 2 messages. Both messages by chance, had been moved by a filter from the inbox to another (user-created) sub-folder underneath the inbox. I checked a couple of others that needed to download the body - no problem. Only the ones that got moved by the filter (not to local folders, but to a subdir of the inbox) had a problem. I do not use offline storage. Ubuntu 16.04 LTS, Thunderbird 45.3.0 I had been running ubuntu 14.04 and recently upgraded. I mv'ed the entire thunderbird profile folder from my old 14.04 /home/apb/.thunderbird to my new 16.04 /home/apb/.thunderbird (I mv'ed the original to .thunderbird.org for safekeeping). I noticed the following oddness. The incoming server is pop-server.bak.rr.com Port 110 from the 14.04 profile and there was no change to this when I selected imap. It acts happy, I don't know if this is OK or not. I haven't checked all emails, only a few... but the only ones that got corrupted from the few I checked are the ones that the filter moved. I also created a new account on TB which had mail sitting in it on the server. It uses the server mail.twc.com and port 993 based on probing the server. There is no option in TB to only fetch headers - the page is completely different from the other account page which was originally POP. All of it came down without problem as far as I can tell from the few messages I checked.
I should mention that the new account I created was IMAP from the start.
So it seems to me then, there is a bug lurking with imap code. I think I have been spared the bug because I have not used IMAP before seriously (except for a few test occasions when I needed to test my TB patches locally at that).
Just to be clear, for me the bug manifested when I CHANGED my account from POP to IMAP (but made no other changes). A filter moves the incoming bodyless msg headers to a sub-folder of the inbox, where it is then read by me and the body is requested, but fails to actually download. Starting a new IMAP account has not to date manifested the bug.
(In reply to apb1963 from comment #133) > Just to be clear, for me the bug manifested when I CHANGED my account from > POP to IMAP (but made no other changes). A filter moves the incoming > bodyless msg headers to a sub-folder of the inbox, where it is then read by > me and the body is requested, but fails to actually download. Starting a new > IMAP account has not to date manifested the bug. Andy, if you can describe in detail what you did someone else may be able to use this as a TESTCASE to reproduce and thus diagnose this bug. Can you do the following: - Identify which POP server you used (don't need your account details, but would be nice to know the domain name of the server, if you used encryption on the connection, and potentially the name and version of the POP server software) - Identify which IMAP server you used ((don't need your account details, but would be nice to know the domain name of the server, if you used encryption on the connection, and potentially the name and version of the IMAP server software) - details of the filter you used to move the files (e.g. it was a Thunderbird filter with these parameters) if you could document as precisely as possible what you did (i.e. which POP server [i.e. server to which IMAP server, nature of the filter (i.e. was it a TB filter or something else)
Ian, Please read comment #130 as it includes some of the info. you're requesting. In general, I simply accept the defaults, there was no encryption on the POP account. As I recall the defaults didn't work for the change of the POP account to IMAP, the IMAP server wouldn't let me in based on the values in the TB database. I had to do a probe. As I changed back to POP to stop the data loss, I don't have the settings of the transformed IMAP account; I suppose I could create a new POP account and then transform it and report back... The new IMAP account uses SSL/TLS, but I get virtually no mail at that address; none recently. All filters used are TB filters.. just a simple match and move.
Thanks Andy. This is a hard bug to diagnose as it happens only intermittently and, when it does, the conditions are hard to describe or reproduce. Your approach may lead to a reproducible way of making TB lose data while moving files. So with information about what you did a developer could make a test case, reproduce the bug, and then start monitoring what TB is doing to see what is going wrong, and why.
I notice that this bug report states: Platform: x86 Windows XP Someone with the ability, should change that as obviously is affects multiple platforms. Changing it will likely attract additional developers that aren't looking at XP bugs. Probably none of them are looking at XP bugs, which would explain the lack of effort on it in the last 8 years.
I would also urge the developer that takes the plunge to take a slow and careful look at comment #82. That idea should have been implemented long ago regardless of anything else. I would also like to point out that I'm NOT stating what I did is reproducible. I'm merely reporting on what happened to me. I would have edited the previous comment rather than create a new one, but you can't do that here.
(In reply to apb1963 from comment #138) > I would also urge the developer that takes the plunge to take a slow and > careful look at comment #82. That idea should have been implemented long > ago regardless of anything else. > > I would also like to point out that I'm NOT stating what I did is > reproducible. I'm merely reporting on what happened to me. > > I would have edited the previous comment rather than create a new one, but > you can't do that here. Andy, is your profile (the preference and Mail lives) on a remote file system? Just curious since I have a patch set that try to deal with some remote file system I/O error issues. TIA
abp1963, Ian (comments #133, #134), that maybe just is a red herring, it also happens on new IMAP accounts never moved from POP, see my comments #57, #104, #111, #115, and there appear no empty mails, but mails are completely gone, unrepairable without any trace anywhere. I suspect that we have at least two heavily intertwined bugs in this report here. But in any case, it seems to be the interaction with a **** server causing failures that lead to corrupted or lost emails of all flavours because the interaction protocol implemented in thunderbird is not safe.
We should focus on what the "whiteboard" asks for: a reproducible test case / approach for making this bug occur. Speculating as to cause does not help unless there is a way of reproducing the defect in a repeatable way. You can then use the test case to see if a proposed patch works, or if your ideas as to cause are accurate. This bug has lasted this long because there is no reproducible test case, so thus no way for an engineer to diagnose the problem and test a patch.
I agree with #140, I think there are at least 2 bugs involved. I figured I'd try to reproduce... I created a new matching TB account for an existing alternate email alias I have. i.e. a different identity. I created a POP account. I note that the new POP account looks like the IMAP account as it allows only fetching of headers, which the previous version I was using did not. I also note that it's now using the newer server (twc) by default and auto-sets-up correctly now. Thanks to whomever fixed the TB DB. I downloaded the few messages already in the account and I also sent a test msg to myself, no problems. Then I selected Fetch Headers and leave email on server until I delete it. Sent myself a test msg, downloaded only the headers, clicked to download the body - no problems. Then I added a filter to move messages into the trash. I sent myself a test message - checked the trash, clicked to download the body - BLANK MESSAGE. Perhaps someone else can try to duplicate this to see if it's reproducible on your end or if I have to go into more detail.
(In reply to Ian Graham from comment #141) > We should focus on what the "whiteboard" asks for: a reproducible test case > / approach for making this bug occur. Speculating as to cause does not help > unless there is a way of reproducing the defect in a repeatable way. You can > then use the test case to see if a proposed patch works, or if your ideas as > to cause are accurate. > > This bug has lasted this long because there is no reproducible test case, so > thus no way for an engineer to diagnose the problem and test a patch. Here is a reproducible (mostly reproducible, not always though) test case: OS: Linux Mint 17 Qiana Thunderbird version: Thunderbird 45.2.0 Email server: hotmail.com Connection type: IMAP (imap-mail.outlook.com:993) with SSL/TLS security. Internet connection: Slow and unstable. I use wireless broadband which jumps between 3G and 2G. Usage scenario: Select 20-25 emails from hotmail inbox and drag them to a local folder. Some of them (say, around 10 emails) should have attachments of few 100 kB. Size of all emails put together should be few MB. One of these things happen: 1. Movement of mails begins. Status bar shows mails being downloaded one by one and after few such email downloads it stops updating status bar. Mail client does not respond to clicks very well. Shows busy icon when mouse is in the left hand side mail folder view. I have left Thunderbird overnight hoping it will complete transfer. Nope, it does not work. Only way to start using Thunderbird is closing and starting it again. After this emails are gone from server inbox and local folder has series of mail entries with some fake time stamp (time stamp is around the time when mail move action was started), empty "from" and "subject" data. This is main data loss situation. This happens most of the times. 2. Movement of mails begins. Status bar shows mails being downloaded one by one. All mails are downloaded and moved to local folder. No data loss. This happens very rarely. This happens when the size of the emails is within 1-2 MB and number of mails selected and dragged is few - like 5 emails. 3. Nothing happens - meaning messages remain in server inbox without any mail transferred to local folder. Sometimes no data loss. Sometimes data loss similar to item No. 1 above. Please let me know if any more information is needed. Eager to help :-)
(In reply to Ravi from comment #143) > (In reply to Ian Graham from comment #141) > > We should focus on what the "whiteboard" asks for: a reproducible test case > > / approach .... > > > Here is a reproducible (mostly reproducible, not always though) test case: > > OS: Linux Mint 17 Qiana > Thunderbird version: Thunderbird 45.2.0 > Email server: hotmail.com > Connection type: IMAP (imap-mail.outlook.com:993) with SSL/TLS security. > Internet ...... > > Please let me know if any more information is needed. Eager to help :-) Ravi I think this is a good testcase! :-) But now we'll need a developer to take it up and work on it (I am not one, alas). I think the main difficulty will be in reproducing the poor wireless broadband connection (3G <-> 2G) ... but it's a great start! Thanks for submitting (and also thanks to Andy for his similar submission.
Just wanted to chime in as I've experienced this issue 4 times in the past month or so (most recent being today, 2017-02-06). Basic details are: Mac OS X El Capitan 10.11.6 Thunderbird 45.7.0 The affected account is a Yahoo email account setup as IMAP. I have two accounts in total, the other one is a personal email on my own server, also setup as IMAP. From what I can tell, the issue appears to happen when trying to Move mail from Inbox to a local folder. The MOVE happens, but the messages are blank afterwards and because they were moved from Inbox, the messages are deleted on the remote (Yahoo) server. I don't really want to switch to another email client because Thunderbird's UI is precisely what I like---my accounts are separate and I can easily manage local folders/archived messages. For now, I reckon I'm going to try and train myself to COPY messages to local folders, then delete the ones in the inbox once I've verified they copied completely and without error. Also, the past two times it's happened, I noticed that when I drag&drop the messages to the local folder, Thunderbird appears to be inactive---as though I hadn't just dragged&dropped anything. After perhaps a minute or so, the messages then show up in the local folder with blank contents (Subject remains). When you try to reply to a blank/corrupted message, it does not auto-fill the "To:" field. I sincerely hope this issue can be resolved, but until then, I guess I just have to use the COPY method. I'm not sure I can provide any other useful information, but just wanted to express that the issue still exists.
This started happening to me about one month ago and today wiped out another dozen emails. I'm running 52.2.1 (32bit) version and the bug is exactly as explained 9 years ago by the original bug reporter. Move from IMAP to local folder and get empty emails dated to moving timestamp. The mails in question were not really large and I haven't been experiencing connection problems. My connection is generally very stable 300 MB/s. This shouldn't be possible to happen at all. Verifying a copy should be a standard procedure before deleting the original.
(In reply to co-re from comment #147) > This started happening to me about one month ago and today wiped out another > dozen emails. > > I'm running 52.2.1 (32bit) version and the bug is exactly as explained 9 > years ago by the original bug reporter. Move from IMAP to local folder and > get empty emails dated to moving timestamp. The mails in question were not > really large and I haven't been experiencing connection problems. My > connection is generally very stable 300 MB/s. > > This shouldn't be possible to happen at all. Verifying a copy should be a > standard procedure before deleting the original. Do you store your e-mails on a local file system (file system on a local disk) or on a remote file system. Is your 52.2.1 TB a windows client or linux client (or Mac client)? Do you possibly know the IMAP server ? (I use dovecot for my local testing under linux). It would be insanely great if someone can create the patch to VERIFY the copied content before deleting the original, be it on the server, etc. At least by that way, we should be able to catch error/faulty conditions to figure out WHY the errors occur by sheer number of error messages, I hope.
(In reply to ISHIKAWA, Chiaki from comment #148) > (In reply to co-re from comment #147) > > This started happening to me about one month ago and today wiped out another > > dozen emails. > > > > I'm running 52.2.1 (32bit) version and the bug is exactly as explained 9 > > years ago by the original bug reporter. Move from IMAP to local folder and > > get empty emails dated to moving timestamp. The mails in question were not > > really large and I haven't been experiencing connection problems. My > > connection is generally very stable 300 MB/s. > > > > This shouldn't be possible to happen at all. Verifying a copy should be a > > standard procedure before deleting the original. > > Do you store your e-mails on a local file system (file system on a local > disk) or on a remote file system. > Is your 52.2.1 TB a windows client or linux client (or Mac client)? > > Do you possibly know the IMAP server ? (I use dovecot for my local testing > under linux). > > It would be insanely great if someone can create the patch to VERIFY the > copied content before deleting the original, be it on the server, etc. At > least by that way, we should be able to catch error/faulty conditions to > figure out WHY the errors occur by sheer number of error messages, I hope. It happens when moving from IMAP inbox folder to my local archive folder. The client is running on Windows 10. The IMAP server is owned by the company I buy hosting from. It's a smallish local company.
I think I can safely state that this happens when the message filter interferes with the downloading process. I've witnessed a couple of times now that a message shortly appears in the INBOX, then the filter chimes in, and, within a blink of an eye the message completely disappears. So my current working assumption is that the message filter moves a message before it is completely downloaded. And that will expose prominently slow and **** servers to this bug.
(In reply to beanladen from comment #150) > I think I can safely state that this happens when the message filter > interferes with the downloading process. I've witnessed > a couple of times now that a message shortly appears in the INBOX, then the > filter chimes in, and, within a blink of an eye > the message completely disappears. So my current working assumption is that > the message filter moves a message before it is > completely downloaded. And that will expose prominently slow and **** > servers to this bug. Unfortunately this is not true: I have seen the bug previously, and I do not have message filtering turned on.
(In reply to Ian Graham from comment #151) > (In reply to beanladen from comment #150) > > I think I can safely state that this happens when the message filter > > interferes with the downloading process. I've witnessed > > a couple of times now that a message shortly appears in the INBOX, then the > > filter chimes in, and, within a blink of an eye > > the message completely disappears. So my current working assumption is that > > the message filter moves a message before it is > > completely downloaded. And that will expose prominently slow and **** > > servers to this bug. > > Unfortunately this is not true: I have seen the bug previously, and I do not > have message filtering turned on. I suspect this may be a situation in which the bug manifests itself, but not the only one. If one sends oneself a whole bunch of multi-megabyte digital photographs while using a very poor internet connection, this problem is likely to show up. The download gets stuck at some point, and it stays stuck forever - you can see the top part of the photograph in the display of the message body, and the bottom part never shows up. Move it to a local folder, and the partial downloaded copy gets moved, the good downloaded copy on the server gets deleted. This would be essentially the same mechanism proposed in beanladen's post, that a message gets moved before it is completely downloaded; it just failed to download completely for a different reason.
That is indeed possible - but not proven. But to my mind your description of how to make this break should yield a great, and somewhat reproducible, test case! I've been monitoring this bug for over five years, and that's been the biggest problem in tackling this...
HI, I hope a slightly tighter check of low-level I/O routines in my patch set in Bug 1242030 - Consolidated patch set from bug 1122698, bug 1134527, bug 1134529, bug 1174500 should prove useful once it lands: maybe it can raise the I/O error due to internet time out properly so that TB can kick in its very crude error processing (at least, it should NOT delete the message on the server thinking that the download succeeded). The patch is being worked on: right now, Windows version seems to cause issues due to the fact renaming/deleting a file while an open file descriptor exists to that file. Very tricky since it is a non-issue under linux and OSX. See https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=ab119318130c500bd1bd03134beb4b8aee7e45a2 At least, I am glad that OSX test proves successful (!) because I develop my patch under linux.
I've been seeing this issue for years, and have to reconstruct lost emails from backups. I am working with a cable modem connection, so it's not particularly slow. Because of this issue, I restrict myself to moving about 200 emails at a time from an IMAP server to a local folder. Most times it works, but sometimes only a fraction of the messages are successfully moved. Unfortunately, Thunderbird deletes the source emails (on IMAP), even though it has not successfully copied them to their destination on a local folder. Instead, on the local folder, I get a long list of empty emails (no subject, no body, sometimes with reply and forward markers preserved), all with the same time stamp. The original emails are lost. They are not present in the email file. Rebuilding the index does not recover the data. It is frustrating, as one would think that a basic protocol would be to not delete the IMAP email until messages have been successfully copied to the local folder. This is not happening. I like Thunderbird enough that I put up with it occasionally blowing away hundreds of emails, but it's an ongoing issue (look at how old this thread is!). Everyone who using a storage-limited IMAP server will need to archive emails on a local computer, so copying emails should be a bug-free process.
I've had this problem for years as well. With good connectivity, it is infrequent, but extremely frustrating when it does occur. I just had something very strange happen, I don't know whether it is related, or something else. But, 1) Synched IMAP inbox with server - an email had the correct header, but both the preview pane and the tab (after clicking on the message) showed the content of a *different* message (specifically, the following message in the inbox). 2) Went to a different computer and synched IMAP inbox with the same server - on this computer, everything was as it should be. 3) From the second computer, copied the dodgy email (which was good on this computer) to an IMAP folder on a *different* server, just in case the experiment about to be tried went wrong. 4) Back to the first computer, moved the dodgy email to an IMAP folder on a still different server. The message simply disappeared! Was nowhere to be found in the folder to which it was moved, either on the first or the second computer, even after giving them a moment to synch up. 5) Used the "undo" move message on the first computer. The message, which was nowhere to be found in the last step, was back where it started, but this time, it had the correct content, as viewed on either computer. Both computers are latest version of Mac OS X, first computer is Thunderbird 52.3.0 (64-bit), and second computer is Thunderbird 52.2.1 (64-bit). I don't know what would have happened if I moved the dodgy email to a local folder - I lost my chance by conducting the experiment above instead. Again, maybe it is part-and-parcel of the same problem, or maybe it is something else. But my take is, sometimes things get stuffed up during the synchronisation. That can't be helped - servers go down, there is network congestion, disks get full, etc. But sometimes when things get stuffed up, they stay stuffed up forever - there doesn't seem to be any verification that the operation was successful, and a retry if it failed, at least in some cases.
I have little useful to add, except that I have "Synchronise all messages locally regardless of age" and "Don't delete any messages" selected and I run TB 52.6.0 (645-bit) on PCLinuxOS, so the problem appears to be cross platform and the mitigations suggested don't work. I last experienced it yesterday when I moved 4 messages from my inbox to a local folder. The four messages remained highlighted in the Inbox and multiple attempts to move them failed. Eventually I had to close TB because it was evidently hanging. When I restarted the four messages were still in my inbox for about about a quarter of a second before they disappeared. Presumably they had been deleted from the server, so the synchronising then deleted the local copies. The four empty messages appeared in my local folder, all with the same received time. On my ISP's forum (Plus Net in the UK) I was told that normally deleted messages should remain in the inbox marked as deleted until the folder is compressed and should still be visible in webmail. This did not happen. The messages were fully gone. This is a bad bug, made worse by the fact no one wants to put robust mitigation in effect to reduce its impact and I agree that is poor design. Tracing the fault is difficult. Putting in a workaround so this kind of problem doesn't matter is just common sense. That TB sees no reason to do so is itself a reason to find a more reliable IMAP client. Except, a quick look through the PCLinuxOS repo shows TB is the only major maintained client available (apart from Seamonkey - but that's another fork of the same code base so likely to share the same bug). The others look very here-today-gone-tomorrow.
I think I've stumbled across some significant new information. I installed Claws mail and linked it to my IMAP account and some messages I moved in TB earlier today showed up in its inbox marked deleted, just as the user on my ISP's forum suggested. However, the lost messages did not, and indeed were truly gone yesterday immediately after the data loss occurred. Now that puts a different complexion on things. It means when the bug happens messages are not deleted in the same way they are when they are moved successfully, so this isn't just a failure to check the copy succeeded before instructing the server to delete them. Rather, a different procedure to delete the messages is being followed. Either TB is sending a "delete completely" message to the server (if IMAP has such an instruction) rather than a normal delete, or it is sending an expunge as well. I think we have to consider the possibility that under some unidentified circumstance TB follows some irrecoverable delete procedure when asked to move messages. It might even be that it deletes before trying to move them and that is why the moved messages are empty and it cannot complete the move. That last sentence is pure speculation, but the rest isn't.
Hmm, or maybe that's a red herring. When I opened Claws mail I had not yet closed TB since moving the messages. Maybe the deletion occurs on closing TB. What to do with deleted messages wasn't set, but emptying deleted folder on exit was. Maybe that caused what I saw.
I have faced this issue twice in last six months, the recent being occurring today. I have a filter which moves all the messages that are coming to local folder. filter logs: [3/29/2018 10:59:04 AM] Applied filter "to me" to message from * - * at 3/29/2018 10:32:29 AM tagged [3/29/2018 10:59:04 AM] Applied filter "move to local" to message from * at 3/29/2018 10:32:29 AM moved message id = f*@oracle.com to mailbox://nobody@Local%20Folders/Inbox [3/29/2018 11:20:20 AM] Applied filter "cc me" to message from <* at 3/29/2018 11:20:10 AM tagged [3/29/2018 11:20:20 AM] Applied filter "to me" to message from * at 3/29/2018 11:20:10 AM tagged [3/29/2018 11:20:20 AM] Applied filter "move to local" to message from M * to mailbox://nobody@Local%20Folders/Inbox [3/29/2018 11:26:43 AM] Applied filter "to me" to message from * at 3/29/2018 11:23:48 AM tagged [3/29/2018 11:26:43 AM] Applied filter "move to local" to message from * at 3/29/2018 11:23:48 AM moved message id = *@default to mailbox://nobody@Local%20Folders/Inbox [3/29/2018 11:26:43 AM] Applied filter "move to local" to message from * to mailbox://nobody@Local%20Folders/Inbox I have removed some private data from the logs. Data from inbox file: X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: From - Thu Mar 29 11:01:31 2018 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys:
I was initially seeing the subject so I assume subject was atleast there, but after repair folder steps subject also got removed. Screenshot https://drive.google.com/file/d/1MTKHcmPv11zTTUjOVjx6Waw4NtBd0lTf/view?usp=sharing
I also had this happen again yesterday. Not having Empty Deleted on Exit set seems to have made no difference, unfortunately. I moved two messages, both containing invoices from suppliers, to a local folder and the move hung. I opened the destination folder and saw the messages there, but without the attachments being marked. At this stage subject and sender were visible, but the absence of the attachment paper clip and the timer cursor showed the transfers were not complete. Switching back to Inbox I saw they were now gone, So I opened Claws mail and there they were in my Inbox but marked as deleted. After logging in to webmail I was able to undo the deletion and mark them again as unread, but TB would not download them until I closed and restarted it. I was then able to copy the messages to the local folder using TB. Doing the same thing worked the second time, which is what's so annoying about this bug; it isn't repeatable.
I first encountered this a year ago. This bug report is now 10 years old. Here's what I know and how I've worked around this issue: 1. This issue seems to only occur for me when Thunderbird is left open. If I close Thunderbird when I'm done checking mail and open it fresh when I want to check it again, I don't seem to encounter this issue. However, I leave it open anyways, so I don't have to log into multiple accounts multiple times. 2. Since I've become aware of this issue and have lost email previously because of it, I no longer use the "Move To" (or dragging/dropping emails into local folders) function to save emails to local folders. Instead, I use the "Copy To" function. This is good because, it does not delete the original email, just creates a copy in the new location. I will know if I'm encountering the issue when, after I "Copy" selected messages to a local folder they do not copy immediately. If I wait long enough, they might appear in the folder, except completely blank (no subject; no body). If that happens, I close Thunderbird and reopen it. After that, I repeat the "Copy To" to my local folder. It's worked for me without fail and I haven't lost an email since changing my behavior to use "Copy To" function instead of "Move To" (or clicking and dragging to folders). So, if you're new to this issue (first, I'm sorry you're here, because you probably lost some important email), I would highly recommend ignoring the "Move To" and drag-and-drop functionality and using the "Copy To" function to save emails to local folders. Otherwise, you risk losing emails—and since this issue is so old, it's likely not getting fixed any time soon. I just leave Thunderbird open and when I notice that a "Copy To" function isn't happening immediately after I select it, I restart Thunderbird, do it again, and all is well. Frankly, even if it gets fixed, I'll probably never go back to using "Move To" or drag/drop. "Copy To" is just safer and allows me to confirm that the messages are copied completely intact in the local folder before deleting the original. That's my two cents—hope it's helpful!
I think I might have a clue - The other day I was dragging some files between my inbox and a local folder and saw a message in the status bar about downloading messages as I dropped the files. Could it be the problem occurs when a move operation coincides with a scheduled download of headers from the server, meaning TB is trying to undertake two different transactions with the server at the same time? Sadly, this will be my last comment on this bug as TB also permanently deleted from my server unrelated messages which had been in my inbox for years, pending further reflection and time to deal with them as they deserved. The loss of those mails prompted me to face my irresponsibility in continuing to use an application for critical data with a critical bug unpatched after 10 years and I have spent the last three days migrating from TB to Claws mail and Osmo. I shall be removing TB from my system in the next day or so, so I shall be unable to help further. Sorry about that.
Who sees this issue and does NOT use filters? Private email me or post a comment, and also state whether you use imap or pop. Thanks
I have been seeing this issue for years, I do not use filters. I use IMAP. I see this issue on poor quality network connections when moving large amounts of messages, so I do not move any messages when I am on a poor connection.
(In reply to Wayne Mery (:wsmwk) from comment #165) > Who sees this issue and does NOT use filters? > > Private email me or post a comment, and also state whether you use imap or > pop. Thanks I do not use Filters and am using IMAP. Nothing fancy here. Like I said, I just quit using "Move To" and now only use "Copy To".
Ditto, no filters and using IMAP.
I've just come across this problem as well. Moved a folder from my IMAP to my local directories. Most (but not all) of the emails are blank, as described above. I do not have any filters enabled. Of course, it would have to be on a relatively important folder that this has happened - all irretrievable as far as I can tell. I can't believe this bug has been here for 10 years!!!
(In reply to Tom Kreutz from comment #170) > Using TB 52.9.1 on Windows 7, I just lost thousands (8 months) of email > messages in two holders when Moving from IMAP to Local folders. (Yes, I was > moving LOTS of messages in LOTS of folders and TB was acting very badly, as > it usually does when I do this: losing subscriptions to nested sub-folders, > needing to be re-started repeatedly, etc.) My IMAP server is now empty, > there are no backups of the IMAP mail server, and I've got nothing on the > local server but a huge list of empty messages. > > I'm definitely going to kill myself. > > There just HAS to be a better solution to such an obvious and > mission-critical task. Here's a simple workaround that I've found to be 100% reliable: (1) Set things up so that ALL your messages are synchonized with local copies in your TB profile. (2) Go Offline. (3) Move or copy all your messages to whatever local files you want to store them in. (4) Go back Online and let TB synchronize all your changes with the server(s). Takes a certain amount of diskspace in between moves/copies but better than the alternative!
Today it happened again. Lost 2 important emails from client while downloading to local folder. Same configuration and software as posted in the past. This is going too serious for me and company. While i loved Thunderbird i´n the past too many bugs remained unchanged over the last years. Now working with IT department to move to Outlook.

It happen to me with version 60.4.0 a month ago but I have only noticed it now.
I do remember now that also with earlier versions I saw such "subject-less" e-mails but just thought it were phantom blank messages and deleted them.

Now I see that I miss two months worth of work e-mails and found this bug report.

I do not use Move or drag and drop, but what I use is "archive" which is "move" essentially I suppose.

I have IMAP inbox that can fit around 6 months of e-mail and every few months I pick few months of e-mails from Inbox and "archive" them to a local folder. I never delete things from this local folder or run any filters on it.

I do have filters on my Inbox though.

I keep my local folder file in Dropbox, but I always close Dropbox during the move process so it does not mess with the file. Inspecting the Dropbox file version history did not help since a 30 days old copy is the same as the current one.

I have the blank mails now in trash. Not sure if local file has the data that could be retrieved.

Date is same on this blank messages but statuses are different (some are starred, some replied, some forwarded) and seem as they correspond to the original messages that were supposed to be moved.

Guess I'll have to change my "archive" behavior.

Just a quick follow-up for anyone discovering this...

I'm still using "Copy To" to get mail from the server to my local folder and haven't had any issues.

Basically, I created a local "INBOX" folder---so when I check my email, I highlight all of it and "Copy To" my local "INBOX" folder. This seems to work because rather than "Moving" the file, since it's "Copying", the original email is not deleted from the server after the process.

Once I'm done with a particular email (that's now in my Local "INBOX" folder), I then "Archive" it---which, since it's just moving from one local folder to another local folder, appears to workaround whatever causes the bug.

Hope this helps anyone---it may seem like an unnecessary extra step, but I'd rather that than lose email. Plus, I've been in the habit of copy/paste (rather than cut/paste) for many years, with regards to moving files around in general, for similar reasons. I've seen firsthand files cut/pasted that disappeared instead of pasting (which a quick CMD+Z or CTRL+Z saved the day), so now I only copy/paste (with the exception of moving text around in an editor, which I'll sometimes cut/paste if it's not something I'm incredibly concerned about losing if I mess up somehow).

If you're reading this, I'm sorry for your loss. :(

I have experienced something similar with empty messages before, but just now I had an incident which might give some directions into what is happening:

  • My setup: IMAP folder that downloads messages, and I have some filters configured to automatically delete specific emails. (no copy'ing to local folders though)
  • I just did some housekeeping: deleting & archiving (thus moving) various messages.
  • I opened a 6MB text attachment of another email I had. When examining this attachment (a log file), it turned out that at the bottom, some of the messages I had moved/deleted had become part of the attachment! Headers, html, text, even the attachments of those mails had become part of that single attachment.
    • it seems like a line ending issue: the last log line now ends with an email header, so it looks like: [final line of log file] From - Fri Aug 9 10:32:56 2019
  • Strangely enough, if I download the same attachment a second time it is alright!
    • it turns out the corrupted file didn't contain the full logs.
    • the corrupt file is 6.8MB, while the uncorrupted attachment is 6.4MB - which is also what Thunderbird reports as the attachment size.

I am happy to share the log file if needed, but as you might imagine there is some personal data in there so I will not just attach it.

Forgot to add: I'm using Thunderbird 60.7.2 on Debian (testing).

Not sure but this might be a duplicate of and fixed by Bug 505456. This fix should be in tb 78 I think.

See Also: → 505456

Only if it deletes before it downloads, which would be another bug anyway.

The fix in Bug 505456 avoids downloads from the server at all when doing copies or moves to Local Folders, provided you have the source folder set to the default of all messages stored locally. In any case I would recommend doing only copies and not moves especially if it involves lots of important messages. See detailed discussions in bug 1566717 regarding the issue.

The problem fixed was that copies to local folders always went to the server to obtain the data. With lots of message being fetched, occasionally a network error occurs that stops the transfer. This resulted in a 0.2KB sized essentially blank message being placed in the destination Local Folder and tb thought the transfer was successful so, for moves, the source message got marked deleted. This bug still exists but now won't occur if just copies and not moves are attempted.

A more general bug that covers all move/copies also still exists: bug 538375

See Also: → 1566717, 538375

Uh, oh, ran into yet another issue so old, it's unlikely to get treatment for it any soon.
I can confirm this bug is still alive and well, and while apparently most messages focus on bulk operations, that's definitely not a requirement.

All I wanted is a notification filter to cut down on the spam, letting me read low priority mostly notification messages on demand instead of getting interrupted on arrival. Given the history of Thunderbird not including such often demanded functionalities but regularly breaking addons providing them, figured I'd cook up something with message filters.
Copying to a local folder and deleting (move is magically not the same as copy+delete on Gmail) was apparently not the right answer, as aside from message filters not providing adequate conditions to filter everything, ended up with this mail after just a few days of usage with less than 20 mails after the initial filter run:

X-Mozilla-Status: 0011
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:

Pedro,
According to comment 177, if you have emails in the source folder using offline store, you should be able to reliably move messages from source folder to a Local Folder. Are you using offline storage?

(move is magically not the same as copy+delete on Gmail)

Not sure what this means. When you move or copy and email from a gmail folder/label to Local Folders, this doesn't use and imap copy or move at all. If you have offline store, the message is read from offline store and appended to the destination local folder without involving gmail (other than to mark the message deleted on the server if doing move).

According to comment 177, if you have emails in the source folder using offline store, you should be able to reliably move messages from source folder to a Local Folder. Are you using offline storage?

Wondered a bit about that, and I remembered seeing offline use mentions in the past, but File -> Offline -> Offline Settings just lead to Local Folder Disk Space settings, so felt like it was only a leftover.

However the "Select this folder for offline use" option described in comment 77 is enabled (by default I assume) not just on the affected source folder, but generally everywhere.

(move is magically not the same as copy+delete on Gmail)

Not sure what this means. When you move or copy and email from a gmail folder/label to Local Folders, this doesn't use and imap copy or move at all. If you have offline store, the message is read from offline store and appended to the destination local folder without involving gmail (other than to mark the message deleted on the server if doing move).

If for some odd reason move is expected to work here but copy isn't, then I guess it's an important quirk, but generally shouldn't matter, describing the problem anyway:

At one point I discovered that on Gmail the Thunderbird move operation doesn't actually mean copy+delete, but it does archiving which was a really unfortunate problem as "deleted" emails didn't just still hang in All Mail, they got mixed up with intentionally archived emails with no distinction, so it was quite time consuming to go over all of them manually to deal with the mess.
Couldn't even use just Thunderbird to untangle the mess because the way archiving works on Gmail, the mail just disappears from Inbox but stays around in All Mail, and couldn't filter based on that logic with Thunderbird.

Suspecting that Google is at fault with the silly behavior, but end result is that I just can't use the move operation when I actually want to move, I need to do copy+delete in such cases at least on Gmail accounts.

However the "Select this folder for offline use" option described in comment 77 (actually comment 177) is enabled (by default I assume) not just on the affected source folder, but generally everywhere.

Yes, if you use default settings, full message offline storage (not just header storage) is used for all folders (not sure, maybe except for Trash and Junk).

At one point I discovered that on Gmail the Thunderbird move operation doesn't actually mean copy+delete, but it does archiving which was a really unfortunate problem as "deleted" emails didn't just still hang in All Mail, they got mixed up with intentionally archived emails with no distinction, so it was quite time consuming to go over all of them manually to deal with the mess.
Couldn't even use just Thunderbird to untangle the mess because the way archiving works on Gmail, the mail just disappears from Inbox but stays around in All Mail, and couldn't filter based on that logic with Thunderbird.

Suspecting that Google is at fault with the silly behavior, but end result is that I just can't use the move operation when I actually want to move, I need to do copy+delete in such cases at least on Gmail accounts.

I'm working a fix for these kind of gmail issues. Not sure it covers your problems described, but I think it mostly does. Re: bug 399475.

(In reply to gene smith from comment #183)

Yes, if you use default settings, full message offline storage (not just header storage) is used for all folders (not sure, maybe except for Trash and Junk).

Seems to be the case, you even remembered the exceptions well.

Problem is that this means my configuration is not the problem then, however apparently I'll be able to reproduce the issue rather easily, just depending a bit on the traffic. Just remembered to check the new folder, and it already has multiple completely empty mails.

Now assuming I'm not just hitting bug 538375 as I'm not doing bulk mail moving and it would be incredibly odd to hit TCP timeout this often with moving just small mails on arrival, would it be helpful to gather information? Would MOZ_LOG=IMAP:4,timestamp cover needs for that?

(In reply to gene smith from comment #183)

I'm working a fix for these kind of gmail issues. Not sure it covers your problems described, but I think it mostly does. Re: bug 399475.

IMAP terminology already tripped me up a bit when I looked into the matter a while ago, and I'm still not well-versed with it, but the more recent messages seem to describe familiar issues. Thanks for the heads up, stashing it for later.

Mostly just wanted to make sure that my copy+delete approach isn't what's making the difference here compared to possibly others not experiencing it anymore with move.

Attached file dataloss.log —

Mangled the log quite a bit, but I believe there's plenty of info to use, at least I believe I can see the high level overview of the problem.

Apparently when more than a single email arrives when checking for mails, the order of operations gets silly:

  • Email A header is fetched
  • Email B header is fetched
  • Email A body gets fetched
  • Email A and B gets effectively deleted
  • Email B body gets fetched

Aside from the obvious logical problem, I'm mildly amused by the "OK Success" response to the last fetch after not sending any content because the email doesn't exist (outside of Trash at least, so it's not completely gone).

Thanks for the log. However, I don't know what you were doing that triggered this.
It looks like tb checked for new message (sent DONE to server to exit IDLE (*)) and two messages, A and B, were detected. As you point out, header A and B are fetched. Then header A body is fetched and then both are imap MOVEd to Trash. Do you know why they are moved to Trash? Maybe they are spam or a normal filter is doing this?
Then body B gets fetched but it is already expunged and gmail says OK Success when nothing is fetched. Technically that's not an error as you can see by the 2nd (middle) paragraph here: https://datatracker.ietf.org/doc/html/rfc3501#section-6.4.8. However, it does give the impression that the front end doesn't know what the back end is doing.
Anyhow, please explain how this illustrates how moving gmail messages to Local Folders has a problem.

(*) An unrelated issue, but it also looks like you are not getting instant notification of new mail from gmail as pointed out by this issue: https://issuetracker.google.com/issues/369861263

It seems to be rather easy to trigger, at least I keep on getting it consistently for emails arriving in the same "batch".

Trashing is expected as my filter does the earlier mentioned Copy+Delete instead of a Move, because that's the closest I can get to logically moving emails without them lingering forever on the server. This way they get moved to Trash on the server, lingering for 30 days (at least on Gmail) which I'm okay with, better than forever.

Suspected that the empty message may not be a specification violation, I just found it amusing in a "successfully failed" kind of way.

Moving is a bit subjective here as I'll consider Copy+Delete doing that, but given that there's no spam filtering involved, and I can reliable keep on reproducing the second or sometimes even the third (I guess all but the first) mail disappearing with my filter, I'm not sure what else could be explained. Could show the exact filter but only the Copy (to local folder)+Delete part is what matters, could attempt to reproduce completely manually by sending 2 emails quick from another address to have clear reproducer instructions but maybe it's even easier to use another Thunderbird instance and just add multiple new emails into the Inbox.

There may be actually a bit more work to do, because I already have one completely blank (no subject, no sender) email just showing 1970-01-01 as a date, suggesting that even the header was lost, not just the body.

That linked Gmail bug is actually quite relevant, as the bug I'm encountering highly likely gets exposed due to batched email processing. If I'd get mails instantly, then the chance of processing multiple emails at the same time would go down quite a bit.

Guess I can replace the Copy+Delete filter with a Move one for some more information. The affected account shouldn't be hard to clean up, but even if that one behaves, it's not what I'd normally use (the info in bug 399475 may change that), and the Copy+Delete corruption would be still in need of fixing.

Ok, I see the problem too. I make a filter on gmail account to

  1. Copy matching message to Local Folders/test
  2. Star the message (just to adds some timing delay)
  3. Delete the message

Then I copy 2 messages from another account to gmail Inbox and let biff happen every 3 minutes. The IMAP:4 log shows 2 messages detected and headers are fetched and one message body fetched. I also see the empty fetch like you see. When I look at LF/test, I see my two messages but the 2nd one is blank but the 1st one is OK (the 1st one is starred and 2nd one isn't)
I look in gmail Trash and both messages are there and perfectly OK and starred.

So the fetch for the 1st message obtains data that is copied to LF. Then the move (of both messages) to Trash occurs which expunges the messages from Inbox at the server. Then the 2nd message is fetched which is now expunged and the "empty" message is copied to LF.

There is definitely a timing or "order of operation" problem here that is blanking the 2nd message before it is "streamed" to LF/test.

What happens with three messages or four? That might establish whether it's an order problem or a counting problem.

(In reply to K.J. Petrie from comment #189)

What happens with three messages or four? That might establish whether it's an order problem or a counting problem.

I will try it with more than 2. But there is at least one other bug where "order" of multiple filter actions matter.

Wayne, Do you recall that/those bugs?

The code is at https://searchfox.org/comm-central/source/mailnews/search/src/nsMsgFilter.cpp#221

One of these?

  • Bug 931303 - message filter failed writing to folder (If "outdated msf condition" exists in "filter move target", and if the filter is "before Classification", filter move shows error message after fix of bug 782738, and skips "Move")
  • Bug 419368 - Filter actions after copy to folder applied to copies as well as originals
  • Bug 383792 - Message Filters that forward and then delete a message fail to forward.

Using a bug query like https://mzl.la/3N9oDIr

Wayne, didn't expect you to do a detail search, but thanks! Anyhow, here's the bug I was thinking of: bug 1781792.

Pedro, Please see if this filter sequence works for you:

  1. Copy matching message to Local Folders/<folder name>
  2. Copy the matching message to Trash

This works for me with more than 1 message in the "batch". Step 2 is the main difference from what I did in comment 188 -- Copy to Trash instead of "Delete" the message. (Ignore the setting "Star".)
For gmail, imap MOVE and COPY seem to have the same effect, at least when the destination is [Gmail]/Trash. Gmail auto-expunges the message from the source folder after a COPY.
I don't yet know why, but when Copy to Trash is done in the last step, the TB doesn't try to do a "batch" move to Trash of all the messages but does them one at a time, so I see:

Fetch the headers for all new messages (this is batched).
Fetch and then copy 1st message to LF
Copy 1st message to Trash (gmail expunges it from Inbox)
Fetch and then copy 2nd message to LF
Copy 2nd message to Trash (gmail expunges it from Inbox)
:
Fetch and then copy the last message to LF
Copy last message to Trash (gmail expunges it from Inbox)

Finally I see an attempt to set each of the new messages in Inbox (all now expunged) with the flag "Not Junk" which gets just an OK response.

All the message are now in LF and not corrupted. The message are also in [Gmail]/Trash. None of the new messages are now in [Gmail]/All Mail.

Copy seems to be the key here as one possible workaround.

Copy to trash on Gmail effectively being a move operation is an interesting quirk, and apparently it works the other way too. Also noticed that Thunderbird seems to have no expectations regarding the outcome of such operations, updating mails from the server as it just forgot about copying mails.
Exploiting that oddity, I found a faster way to reproduce issues by copying from Trash to Inbox which is treated as if the recently filtered messages just freshly arrived again.

The reason why Copy seems to work is the lack of coalescing which is done for onlinemove, but not for onlinecopy, so Move to Trash gets a coalesced onlinemove operation which makes mails disappear before they are all properly processed, while Copy to Trash queues onlinecopy operations one by one.
Not sure about IMAP ordering guarantees, but generally it feels like this just happens to work because commands are processed in order without any kind of dependency management actually ensuring that mails don't actually disappear.
https://datatracker.ietf.org/doc/html/rfc3501#section-5.5 considers parallel "COPY + COPY" invalid (isn't "UID COPY" different though?), so maybe the reordering concern is only theoretical.

But the reason why coalescing seem to be causing trouble is because apparently there's nothing taking care of fetching the messages in time.
Based on filter log messages, nsImapMailFolder::ApplyFilterHit was the function of interest, but couldn't really figure out where copyService->CopyMessages leads, so felt things up in the dark, and assumed that nsImapMailFolder::CopyMessages would be my target. Within that, nsImapMailFolder::CopyMessagesWithStream seems to only take care of the first mail, likely leading to ImapMessageService's copyMessage issuing a single fetch for the currently processed (the first) message, and I guess there's an asynchronous suspend at this point allowing the destructive coalesced move to be processed before the other fetches even get into the queue.

Quite ironically something issues a coalesced fetch operation too at some point, just way too late. Would still feel a bit fragile, but if CopyMessages would start with this coalesced fetch (or it would queue up all fetches during initialization), then assuming no parallel command execution issues, all mails should be fetched before they get deleted/moved.

I'll admit that I have tried to avoid the filter code over the years and don't really understand it. Also, don't think it's been touched (other than re-formatting and array and loop modernization) since I've been working on TB since about 2017. But I have found another workaround. Instead of doing a 2 step filter, use two single-step filters that run sequentially:

Filter 1:
Apply when manually run and getting new mail with "Filter before junk classification"
(Make sure junk classification is enabled for account in Junk settings.)
Rule: Copy message to Local Folder

Filter 2:
Apply when manually run and getting new mail with "Filter after junk classification"
Rule: Delete message OR Move message to Trash, either will work {*}.

This should also work for non-gmail servers that don't do the same thing for copy and move to Trash. However, it does require that junk classification be enabled which I typically turn off. Also, not sure why classification has to be "before" in Filter 1; but, if not, the problem still occurs, i.e., all but 1st message is expunged before they are copied to LF.

Unless someone can find a fix for this, I think the filter dialog should show a warning that if "Move" or "Delete" is used in a filter, there may be problems. So user needs to test the filter with "throw away test messages" before relying on it.

Finally, this issue should be written up as a new bug since it is not really the same thing as was described in this bug report at comment 0 which didn't really involve filters.

{*} Edit: If "Delete message" is used, the messages are moved to Trash but when Trash is imap SELECTed, the filter actually marks each message in Trash as \deleted. Gmail doesn't auto-expunged messaged marked as deleted in Trash so the messages are still there but hidden, so you still need to "compact" or empty trash to get rid of them. If "Move message to Trash" is used, the messages are just moved to Trash and not marked deleted and remain visible in Trash (and, in both cases, are also gone from All Mail).
Note: Wouldn't expect the 2nd filter to run on Trash when it is selected, but it does. I thought the filter would only run when new messages arrive at Inbox. But if set to "Move message to Trash" the filter does nothing, unlike if set to "Delete message".

Decided to ride on the back on this issue because the fundamental problem is the same, even if mine can get a filter-specific fix.
Haven't tried the bug starter issue, but I wouldn't be surprised if that would have the same coalesced move problem.

Will try the 2 filter workaround. The currently set COPY+COPY is working, but due to the delayed copy to Trash, it doesn't inhibit notifications, and I expect the 2 filter setup to behave essentially the same.

I still believe that issuing a coalesced fetch query early would deal with it without the complacency of just putting out a warning message, but of course that's just guessing. Contemplating looking into later, but last time I setup a build environment for Firefox it took quite some time and I expect Thunderbird to be similar, and I'm not particularly fond of the messaging interface being involved as that uses a whole lot of auto-generated black magic I'm not familiar with.

Was about to write up a new bug report, but as I was going through the process, I realized it wasn't the right option. While it doesn't contain enough information to make sure it's an exact match, I seem to have a case of bug 1781792 .
Seeing the bug severity discussion there though, I'm wondering:

  • What's the appropriate severity for this? If severity description is supposed to be understood as "may impact more than 25% of users OR causes data loss", then it's S1.
  • Is the "critical" severity here a leftover from the past? Wondering if that affects proper prioritization.

My filter is set to run on Inbox only, so it shouldn't mess with Trash.
Aside from that, I haven some archiving filters which seem to behave okay (assuming mails are downloaded in time), and mails from the Trash disappear with those.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: