589973 - Mail vanishes when moving an email from an IMAP folder to an IMAP folder in a different account

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Peter Walter]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Thunderbird/3.1.2

I have TB configured with seven IMAP accounts. One of the accounts is a gmail account, and another is a yahoo account. The remaining five accounts are on servers using qmail as their MTA. Occasionally, when I move (Message / MoveTo) an email from a folder in one IMAP account to a folder in a different IMAP account, the move appears to succeed, but the email never shows up in the destination folder. It is no longer in the source folder - it just disappears.

Reproducible: Sometimes

Steps to Reproduce:
1. Select an email
2. Using the TB menu, Message-MoveTo, and select an account on a different server.

Actual Results:  
The email disappears from the original folder, but does not show up in the destination folder. A search of all folders on the source and destination accounts turns up nothing - the email has simply disappeared, and TB had no error messages.

Expected Results:  
The email should be moved to the destination folder in an IMAP account.

Comment 1

[Ludovic Hirlimann [:Usul]]

Can you try to reproduce and have imap log turned on see https://wiki.mozilla.org/MailNews:Logging ?

Comment 2

[Peter Walter]

(In reply to comment #1)
> Can you try to reproduce and have imap log turned on see
> https://wiki.mozilla.org/MailNews:Logging ?

Logging enabled with environment variable NSPR_LOG_MODULES=imap:5

Comment 3

[Peter Walter]

I enabled imap logging, and ran TB for about 1.5 days, and I got a very interesting result - the problem refused to show up! I then restarted TB *without* the logging enabled, and, within three hours, TB ate an email while moving it to a folder in a different account. It seems to me that whatever debug logic is in TB is causing a different execution path when enabled. I currently have it running with logging enabled again to see if doing so avoids data loss over several days of use. Any suggestions on additional diagnostics I can do would be welcome.

Comment 4

[Ludovic Hirlimann [:Usul]]

David could logging change race conditions ?

Comment 5

[David :Bienvenu]

I highly doubt this is a race condition.

Comment 6

[Peter Walter]

I have run the imap logging session of TB for an additional 1.5 days, and I am *still* unable to get TB to misbehave. I would like to point out that there are other (possibly related) problems which have also disappeared. One of them is that some emails are not saved in the designated sent folder (a dialog box pops up suggesting a retry, but the retry never works).

Comment 7

[Peter Walter]

I have now run the imap logging session of TB continuously for three days. Not only have I not lost any mail while moving it around from folder to folder, other problems, the major one being that intermittently TB can't save mail in the designated "sent" folder, have also disappeared. In the absence of instructions on how to further help to debug this problem, I am going to add the debug environment variables to Windows XP while the problem is investigated, and get rid of the batch file I am currently using.

Comment 8

[Wayne Mery (:wsmwk)]

Peter, is the problem still completely gone?

Comment 9

[jallen]

Just thought I'd mention that I'm having a similar problem where I create an email and save it in the Drafts folder and then try to copy it to a folder on Gmail.  It disappears from the Drafts folder but doesn't show in the folder I selected but instead ends up in the Trash folder.  I had someone else try this with TB and it does the same thing for them.  I only have the one Gmail account in TB.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> Just thought I'd mention that I'm having a similar problem where I create an
> email and save it in the Drafts folder and then try to copy it to a folder on
> Gmail.  It disappears from the Drafts folder but doesn't show in the folder I
> selected but instead ends up in the Trash folder.

Gmail's implementation around mail flagged \Draft is special.
  If \Deleted flag is stored to a mail which has \Draft flag,
  Gmail IMAP moves it to Gmail's Trash([Gmail]/Trash).
  "move/copy mail to [Gmail]/Trash" means "remove any Gmail Label from mail",
  so mail disappears at any Gmail's IMAP folder including [Gmail]/All Mail.
[e-mail], read all bugs listed in dependency tree for meta bug 402793 first, please. Please don't add comment about special phenomenon with Gmail IMAP which is irrelevant to original problem of comment #0.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> A search of all folders on the source and destination accounts turns up nothing
> - the email has simply disappeared, (snip)

As IMAP, delete operation is merely "store \Deleted flag", as far as you don't do "Compact"(issue expunge), and you don't invoke automatic-expunging by option such as "issue expunge after any delete", and you don't invoke automatic-expunging by other client.
Tb has capability to show mail of \Deleted flag.
  Server Settings : IMAP delete model of "Just mark it as deleted".
You can check it when you will meet problem again by next.
(1) Create a Tb's new profile(call PROFX).
(2) Define relevant IMAP accounts.
(3) Server Settings : Just mark it as deleted
    Synchronization&Storage : Disable "keep messages..."
             (or set offline-use=off of all IMAP folders via Advanced button)
    Tools/Options/Avanced/General : disable Global Search and Indexer
    Disable any automatic deletion/automatic expunge relevant option.
    (retension policy, empty trash upon exit, ...)
(4) Start Tb with -no-remote, using PROFX.
      thunderbird.exe -no-remote -p "PROFX"
    Second instance of Tb starts by -no-remote parameter.
    You can see mail of \Deleted flag : shown with strike line at thread pane
(5) If lost mail is merely flaged as \Deleted, you can recover it by
    "Undelete" of the lost mail by this Tb instance of -no-remote.
At original Tb instance, will lost mail be recovered?

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> other problems, the major one being that intermittently TB can't save mail in
> the designated "sent" folder, have also disappeared.

If error occurred during saving sent mail copy to Sent folder, Tb asks user for Retry or Cancel. No such dialog and Tb silently doesn't save sent mail copy?
Anyway, "one problem per a bug" is rule of Bugzill.Mozilla.Org. Open separate bug for your other problems, after search B.M.O for already reported bug for your problem well, please.

Note:
As Bug 257735 exists(will be fixed by Tb 3.1.8), and next problems exist,
  - auto-save is invoked even after send is requested
  - send is executed even during auto-save is in progress
confusing dialog of "send fail/save to Sent fail" can be shown by auto-save during/after mail send, if you enable auto-save.

Comment 13

[jallen]

I'm sorry but I thought it was related.  Everyone keeps harping on not submitting a bug report if one already exists and yet you complain when I post something to one that I thought was related, so I guess I can't win.  In any case you don't seem to have understood my comment - I'm not trying to delete a draft email but move it to a folder.  I would think this should be possible with TB and a Gmail account, so that's why I posted the comment.

Comment 14

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to previous question. If you feel this change was made in error, please respond to this bug with your reasons why.

Comment 15

[Peter Walter]

Sigh. Moving back to UNCONFIRMED instead of creating a new bug. I posted the original bug, which still exists, at least in TB 6.0 Beta. One of the annoying characteristics of the bug is that turning on the debug mode of TB causes the bug to disappear. Also, in TB 6.0 Beta, the IMAP logging is broken - I start TB as follows in a CMD file:

@echo off
set mydate=%date:~-4,4%%date:~-7,2%%date:~-10,2%
set mytime=%time:~0,2%%time:~+3,2%
set NSPR_LOG_MODULES=IMAP:5
set NSPR_LOG_FILE=%USERPROFILE%\Desktop\RunBirdLogs\thunderbird_%mydate%_%mytime%.log

: sometimes TB won't die completely when terminated normally
taskkill /IM thunderbird.exe /F
start /d "c:\program files\mozilla thunderbird" thunderbird.exe
exit

I started the CMD file so I could track what happens with IMAP on the desktop side, and moved a file from one folder to another in the same IMAP account (it worked, as it usually does in DEBUG mode). Instead of the expected log file of IMAP traces, I get a log file with completely blank lines, as in filled with spaces where text is expected. So, I cannot submit the log file. Any help in getting TB to record the log file properly would be appreciated as a start.

Comment 16

[Wayne Mery (:wsmwk)]

is this a gmail account?
do you get the same logging results if you type the logging settings at the command prompt, instead of using a batch file?

Comment 17

[Peter Walter]

I have sixteen mail accounts, of which two are gmail accounts. The other fourteen accounts are all on servers running the Qmail MTA, and all of the accounts are accessed via IMAP or IMAPS.

Regarding the logging, I restarted the computer, and logging is working fine now when started from the batch file.

Please read comment 3 to comment 7 for background.

I started Thunderbird normally (without any debugging environment flags), then created and sent a test mail from one account to another; that went fine, and a copy was stored in the "Sent" folder. I then moved the mail from "Sent" to the "Inbox", and the mail went to the great bit bucket in the sky.

I then started the debug batch file for Thunderbird, then created and sent another test mail from the same account to another; that went fine, and a copy was stored in the "Sent" folder. I then moved the mail from "Sent" to the "Inbox", and the mail was moved properly. I then quit TB.

I have attached the debug log to see if anybody can figure out what may be going on.

The frustrating thing is that TB works as expected only with debugging turned on, so I can't log what it is doing when it eats the mails - unless someone can suggest some program that might capture what TB is doing when the environment variables are not set. Suggestions, anyone?

Comment 18

[Peter Walter]

Attachment: Thunderbird debug log

Comment 19

[David :Bienvenu]

logging can be messed up if you have two apps, or two instances of the same app writing to the log, or if you launch a second copy of the app while the first copy is running, which then shuts down. Or if you run a different app (like Firefox) with logging pointing at the same log file (Firefox and Thunderbird use the same logging mechanism and environment variables).

Comment 20

[Peter Walter]

(In reply to David :Bienvenu from comment #19)
> logging can be messed up if you have two apps, or two instances of the same
> app writing to the log, or if you launch a second copy of the app while the
> first copy is running, which then shuts down. Or if you run a different app
> (like Firefox) with logging pointing at the same log file (Firefox and
> Thunderbird use the same logging mechanism and environment variables).

Nope, none of the above are / were in effect. In any case, logging seems to be working know.

Comment 21

[Peter Walter]

(In reply to Peter Walter from comment #17)
> I have sixteen mail accounts, of which two are gmail accounts. The other
> fourteen accounts are all on servers running the Qmail MTA, and all of the
> accounts are accessed via IMAP or IMAPS.

A minor correction to the above. I have sixteen mail accounts. Two are gmail accounts, one is a yahoo account, and thirteen are on servers running the qmail MTA. All accounts are IMAP; all are accessed via IMAPS, (SSL) except for one of the qmail accounts which is accessed via IMAP (unsecured).

The same problem exists on all accounts, whether the source and destination folders are on the same server, or are on different servers, so I do not think the problem is related to the access protocol (IMAP/IMAPS) or the MTA on the other end. Also, the problem does not show up if I use a different mail client (Outlook 2007), although Outlook otherwise chokes on the volume of mail I receive and store.

I was once an applications programmer a (very) long time ago, and I have had a bit of familiarity with debugging complex software. Perhaps one way of finding out what is going on is for me to run a IMAP proxy which logs the traffic but otherwise passes it through for one of the servers; perhaps one similar to http://www.aboutmyip.com/AboutMyXApp/ImapProxy.jsp would work for this purpose. However, I don't have the expertise to interpret the log files either from the TB debug logs or from the proxy server output.

I consider this bug to be a serious block to acceptance of TB as a business-ready email client, particularly when the data loss is considered. Similarly important bugs have languished in bugzilla for over six years without being fixed. Is there a TB developer, familiar with IMAP, who is willing to work with me to interpret the debug output, inspect the code, and get it fixed? I am more than happy to do any grunt work necessary within my time constraints.

Comment 22

[Peter Walter]

The following is an abstract from email comments by Mark Crispin, Senior Software Engineer, Messaging Architects.  Mark is the inventor of the IMAP protocol, and has kindly consented to comment on this bug.

> I reviewed your messages and the Mozilla bug report.  I'm afraid that I
> probably can not give you much assistance.  The problem is almost
> certainly in Thunderbird.  The fact that the problem does not occur you
> enable logging makes it more or less impossible for me to tell you what is
> happening.  I can, indeed, interpret an IMAP protocol transcript (for what
> it's worth, I'm the inventor of IMAP); but without a transcript there's
> nothing for me to interpret.
>
> As a consolation, here are some quick notes that represent more or less
> everything that I can tell you:
>
> There is no operation in IMAP to move a message from one server to
> another.  The process of doing so is a sequence of operations on the IMAP
> client (in your case, Thunderbird):
>
> [1] Download the entire message from the source server into Thunderbird
> using a FETCH command to the source server.
>
> [2] Upload the entire message from Thunderbird to the destination server
> using an APPEND command to the destination server.
>
> [3] Delete the message on the source server using the STORE command to the
> source server.
>
> [4] Expunge (permanently purge all deleted messages in) the mailbox on the
> source server using an EXPUNGE command to the source server.
>
> At each of these steps, the server in question returns an indication of
> success or failure.  Each step depends upon the successful completion of
> the previous step.
>
> Thus, in order for your scenario "mail vanishes when moving an email from
> an IMAP folder to an IMAP folder in a different account" to occur, it is
> necessary for steps [3] and [4] to succeed, but somehow step [2] did not.
>
> Yet, you can not even start step [3] until step [2] succeeds.
>
> This means that EITHER:
>   	A bug in Thunderbird caused step [2] to be skipped (or
>   	abandoned), and continue with step [3] without success
>   	from step [2].
> OR:
>   	A bug in the destination IMAP server caused a false
>   	report of success from step [2].
>
> The first explanation seems quite a bit more plausible than the second
> explanation, especially if this happens with multiple destination IMAP
> servers with different software implementations.
>
> The fact that the problem vanishes when you turn on debug logging in
> Thunderbird bolsters my belief that it is the first explanation.  A
> "Heisenberg" bug that changes based on client based logging may plausibly
> affect the client, but not the server.

Because, in my experience, the bug also appears if the move operation is done between folders on the SAME acccount on the SAME server, I will capture an IMAP log (via a proxy server) to determine if the sequence of logged operations is consistent with what should happen, according to Mark.

The status of this bug is currently UNCONFIRMED. It would probably help if another subscriber to this bug confirms that they get the same results as I do, to wit;

(a) running TB normally causes some mail to vanish when moving to a different folder on the same / different server.

(b) running TB in DEBUG mode causes the problem to disappear.

Comment 23

[Wayne Mery (:wsmwk)]

Michael, Nikolay, have  you seen this?

Nikolay, anthing in the imap log?

Comment 24

[Peter Walter]

An update:

I am experiencing the problem running TB on an older laptop - a Sony VAIO PCG-GRV550 (Pentium 4 CPU 2.4 GHz, 512 MB RAM, 30 GB HDD, Windows XP SP3).

For reasons unrelated to this bug, I am now primarily using TB on a HP G42-415DX notebook (MD Athlon II X2 P340 CPU 2.2 GHz, 3GB RAM, 320GB HDD, Windows 7 Ultimate).

While I can reliably replicate the problem (with debugging off) on the Sony Vaio, I have been UNABLE to do so on the newer laptop, despite pounding on TB for a month or so with debugging off.

The only differences between the two environments are the hardware / OS combinations, and, having been unable to replicate the problem on the newer laptop, I suspect that the cause is some kind of interaction with Windows XP, or a race condition on the slower laptop, despite Comment 5.

The older laptop will be fully retired in a couple of days. If the devs would like me to continue to pursue this bug, I can fire up a VirtualBox VM with Windows XP SP3, restrict the CPU performance in the VM, and see if I can replicate the bug there.

Otherwise, as the reporter, I request that this bug be closed.

Comment 25

[Peter Walter]

Sigh. Please don't close this bug. I lost two messages a few minutes ago moving from one folder to another within the same IMAP mail account. Fortunately, I was able to recover the AWOL messages from other sources.

The data loss occurred at a time of high network activity (large files being uploaded to a server) and concurrent local disk I/O (secondary drive being defragmented). My prior laptop had limited memory (512MB) and was always busy with background applications, causing a lot of pagefile activity while I was running TB.

I will revert to logging TB IMAP activity in an attempt to capture the data loss when it occurs next.

Comment 26

[Chuck]

I have a very similar problem, only it's on a Mac.  

When dragging a selected or multiple messages to a local folder, Thunderbird fails to move the entire message (no matter how small) but still deletes it from the server.

Based on my review of the above comments, and assuming there is basic ThunderBird code that is closely mirrored in both the windows .exe and OSX .app executables, no matter what the operating system is, I think there must be some type of issue in the way the base Thunderbird code executes the IMAP protocol. (Nicely explained by Mark Crispin above.)

Specs: Thunderbird 12.0.1
Mac OSX: 10.6.8
MacBook Pro 2.5Ghz

Thuunderbird Key Mail account settings for the yahoo email account that breaks:
- Keep messages for this account on this computer
- Synchronize all messages locally regardless of age
- Don't delete any messages
- Always keep starred messages
- Local settings / disk space: don't delete any messages, Always keep starred messages

This is somewhat random, but seems to happen more often when Thunderbird has been running for a while or when email arrives that is automatically moved to a local folder via rules and marked as read. (Again, this seems to cause the issue more often) 

A freshly opened Thunderbird seems to NOT experience this problem unless email arrived that had rules run on it.

Most of the time when this occurs, the header of the email will move to the new folder in a normal fashion, but the body is missing. (Header Data: From, to, date, subject line, time, etc.)  But, there has been times when even the header info didn't make it to the local folder. 

If it's the case where the email header made the move to the local folder without the body, the tiny black and white beach ball just spins when highlighting or clicking on the message.  

And, there is NO getting the email body back from the server or locally from what I can decipher.

I'm curious if Peter is still having this issue.

Thanks,

Chuck

Comment 27

[Wayne Mery (:wsmwk)]

are there reports of this on mozillazine?

Comment 28

[Chuck]

Thanks Wayne,

I did a quick search of mozillazine, but was not certain how to phrase the search best to get a more limited list.  Got way too many extraneous issues to weed through with the searches I tried.

Still on:
Mac OSX: 10.6.8
MacBook Pro 2.5Ghz
Thunderbird, latest version, 14.0, released July 17, 2012, which still has this issue.

The problem does seem worse if there are any open windows--either trying to send mail or a double click which puts the body of an email in a separate tab.  The more windows which are open, the more likely it is to lock and then not move the email messages from the server to the local repository.

I do run Little Snitch.  That said, I've previously allowed all connections, but the problem remained, so I've gone back to confirming little snitch approval requests on an "as-needed" basis.  

I do know that I have a consistent issue with the "send" side of Thunderbird 14.0 / IMAP on yahoo mail which errors out on the mailbox upload for "sen".  I would say that issue is as high as 80% of emails I send.  I just hit resend and it eventually works. 

The bigger concern is there would appear to be some program issue with the IMAP protocol and Thunderbird.  There logically should be no way to delete any email from the source server until the local server "for sure, for sure" has the email moved local.

Thanks!

Chuck

Comment 29

[Wayne Mery (:wsmwk)]

Peter, do you still see this problem when using a current version?

Comment 30

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 31

[i4320325]

[Tracking Requested - why for this release]:

Comment 32

[Peter Walter]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #29)
> Peter, do you still see this problem when using a current version?

I haven't encountered the issue with the current version of TB (38.2.0). It may have been fixed as a side effect of other changes. Nobody else has referenced the problem since 2012, so I think it is safe to close the bug now.

Comment 33

[Wayne Mery (:wsmwk)]

Thanks Peter