494669 - Some IMAP messages missing/lost/not shown after moving them to another IMAP folder

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[Paolo MARINI ♂]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2

I've been a TB user for quite a few years. I did look in reported bugs for "missing messages" but no relevant entry was found.
This problem only happens with the current Beta version that I'm using.
I know that I shouldn't have been using it for my "live" mailbox but I have and I would simply try to find a good workaround for it, on top of reporting the issue for the purpose of having it fixed in the final version / next release.

Basically when moving a group of messages from a folder to another, some of them go missing from the target folder, although they "appear" to be there in a search folder for the Inbox+subfolders (but not when using "quick" findbox).

The first time that I noticed this was when I moved about 40 messages to archive my Sent emails from last year, and half of the messages went missing (but I noticed this only after a couple of weeks).
Later I started keeping on eye on this and opening the target folder as soon as I moved the messages and it seemed to be fine - but only when moving a single message at a time. When moving more than one, some of them almost invariably went missing.

Please bear in mind that when I say "missing" I always mean:
- They are not displayed in the target folder (where moved to)
- They appear as being in the target folder if I do a "full" search (and am able to open them), but
- They don't show in the "quick" find (upper-right box) for the target folder

For the purpose of avoiding the total loss of these messages (which are there but don't show), is there a way to identify which ones are in this state and how to properly recover/copy/move them? (I can tweak stuuf around on my system if needed)

I can reproduce this quite consistently, by selecting a group (>1) of messages and moving them somewhere else. Most of the time it's just one of them missing from the target folder, but in my first occurrence it was half of them.

Reproducible: Always

Steps to Reproduce:
1. Select a group of messages, the more the better
2. Move them to a different folder
3. Count/check the resulting entries, they will most likely be one less of the original number
Actual Results:  
The original messages, except that one or more are missing

Expected Results:  
All of the messages should be in the target folder

Just a hunch, could be a 0-index counter used in a 1-index loop...

Comment 1

[Bob Clary [:bc] (inactive)]

I can confirm a similar situation. I'm using dreamhost imap and nightly shredder builds on mac.

I had a mail which had been marked as Junk but which should not have been. I marked it as not Junk then drug it from the Junk folder to the Inbox. It did not appear. Reindexing etc did not help. Deleting the msf did not help. I looked in the INBOX file using vim and found the message. Creating a new empty folder in my Local Folders and copying the INBOX over it allowed me to recover the mail.

Comment 2

[WADA:World Anti-bad-Duping Agency]

To Paolo Marini(bug opener):

(Q1) Move of mails between what kind of mail folder?
  (a) local mail folder(POP3, "Local Folders") -> local mail folder,
  (b) local mail folder -> IMAP  mail folder
  (c) IMAP  mail folder -> local mail folder
  (d) IMAP  mail folder -> IMAP  mail folder of same account
  (e) IMAP  mail folder -> IMAP  mail folder of different account
(Q2) "Move" is done by Drag&Drop(or Shfit+Drag&Drop)? Or by Move menu?
(Q3) "Missing messages" is which phenomenon?
  Say mail 1 to N was moved from folder XX to folder YY.
  1 to M == not missing, (M+1) to N == missing.
  (f) (M+1) to N doesn't exist in XX and YY.
  (g) (M+1) to N doesn't exist in YY. Still remain in XX after end of move.
  (h) Other.
  Because you say next, I guess move from subfolder of Inbox to Inbox and (g),
  and "quick search" was executed for Inbox(move target folder).
> although they "appear" to be there in a search folder for the Inbox+subfolders
> (but not when using "quick" findbox)

Comment 3

[Paolo MARINI ♂]

To WADA-san:

Q1) = typically (d) but the first case of 20 out of 40 was on (e)

Q2) = always drag & drop (I have a multitude of folders and using the menu is a PITA)

Q3) = (g) in the sense that they are not displayed in either folder when simply clicking on them. They do appear in the search results for the target folder but only when it's a "full" search and not a "quick" search for the small T-R box. The visual feedback is like when the "move" operation completes fine.

Please note that Comment #1 was entered by someone else, not me. I usually move them out from the Inbox, his example was into the Inbox.

Comment 4

[WADA:World Anti-bad-Duping Agency]

To Paolo Marini & Bob Clary:

If timeout occurs on an IMAP command, "partial move" may occur. Get IMAP log with timestamp. See Bug 402793 Comment #1 & Bug 402793 Comment #17 for NSPR log with timestamp.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(Q4) Do you enable "offline-use" of "move source folder" and/or "move target folder"?
AFAIR, auto-sync is enabled by default. Check Syncronization&Storage/Message Syncronizing setting.

Comment 6

[Paolo MARINI ♂]

Hello WADA-san,

I now remember a similar issue that I had a while ago when moving messages with big attachments but they were totally lost following an some kind of warning on the failure to complete. In this case they simply "hide" themselves after the transfer... (I can open them from the "full" search results and they're fine).

Q4) = I keep offline-use disabled all the time for security reasons.

Anyway, I added the following lines to my startup script:

export NSPR_LOG_MODULES=imap:5,timestamp
export NSPR_LOG_FILE=/tmp/imap.log

and I will let you know when it happens again.

Feel free to decrease the Importance from "Critical" since it would seem that the messages are found somewhere, but I'll leave that to you to decide.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> Anyway, I added the following lines to my startup script: (snip)

Because IMAP:5 writes mail data to log file, log file becomes huge easily. Please watch log file size, if you enable logging during your daily use.

Comment 8

[Wayne Mery (:wsmwk)]

bc, unless I misread these two reports, your issue is not the same as reporter's

Comment 9

[WADA:World Anti-bad-Duping Agency]

To Paolo Marini(bug opener):

As I wrote in Bug 495007, Tb seems to have problem of mail.server.serverX.timeout=29. If it occurrs, timeout of IMAP command like "uid copy x:y or fetch x:y where y>x" can easily occur. If your problem is found to be timeout related issue, and if you see mail.server.serverX.timeout=29 for the IMAP account, try explicit/larger timeout value.
> Put next line in user.js(600 sec., 10 min.) for IMAP account, and restart Tb.
>   user_pref("mail.server.serverN.timeout", 600);
Will mail.server.serverN.timeout be kept 600 always? (Check via Config Editor)
Will frequency of problem be reduced by it?

Comment 10

[WADA:World Anti-bad-Duping Agency]

mail.server.serverX.timeout=29 was for interval of IDLE.
TCP time out was set in mailnews.tcptimeout(default = 60 sec).
If improved by larger mailnews.tcptimeout, set it for specific account only.
   mail.server.serverN.tcptimeout
Will it work well?

> Q1) = typically (d) but the first case of 20 out of 40 was on (e)

For (e), cross server move, Bug 497622 existed and was resolved.
Check with latest trunk nightly build, please.

Comment 11

[Paolo MARINI ♂]

(In reply to comment #9)

I've changed the timeout setting to 120 today, on another TB installation which was 3.0b1 now upgraded to 3.0b3 (while the initial one for this bug was 3.0b2).

I was wondering if the fact that the messages appear in a "full" search but not otherwise means that they are somewhere in TB but not on the IMAP server anymore? (remember that I'm not using "offline mode" on any installation...) - I mean, should I manually move/copy them again from the TB results of a "full" search or are they flagged in some unusual way on the IMAP server? (because I don't see them from the webmail interface...)

-> Thanks for comment #10, I'll try the other timeout too...

Comment 12

[WADA:World Anti-bad-Duping Agency]

> Q1) = typically (d) but the first case of 20 out of 40 was on (e)

For (d), move within an IMAP account. Possibly next problem, because one duplicate message == one lost message until rebuild-index.
> (These tow bug are already fixed)
> Bug 414723 Duplicate messages appearing in list, and some messages not appearing (IMAP)
> Bug 501392 Duplicates messages (or missing messages) in IMAP

Comment 13

[tommyads]

I am using the actual version (TB 2.0.0.23, Nov. 2009), with a similar problem.
With an IMAP-account, there are two mails in the inbox, which appear in my search-folder (showing inbox & sent folders), but which do not appear in the inbox itself.
I also see those messages in the inbox when using the web-interface of the mail-provider, so the messages are really in the inbox.
A "compress" for all folders of the IMAP - account did not help.

Comment 14

[Paolo MARINI ♂]

to partially confirm Comment #13, I did try "compressing" the folders when on 3.0b2 and 3.0b3 but that didn't help...

I have to admit that since installing Beta3, this new version does a lot of "automated" indexing of the folders, as previous comments by WADA indicated that the issue was a timeout of the IMAP server which in turn would have caused TB to leave those messages in a limbo. so well done team: indexing often is definitely better than risking having a message being "lost" (or kind-of-lost)...

Comment 15

[Wayne Mery (:wsmwk)]

 Paolo do you still see this problem?

Comment 16

[Phoenix]

Resolved per whiteboard