Closed Bug 415457 Opened 17 years ago Closed 13 years ago

Losing mail little by little [pop]

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Version:
1.8 Branch
Platform:
x86
Windows Vista
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: pepalogik, Unassigned)

Details

(Keywords: dataloss) 

User-Agent:       Opera/9.24 (Windows NT 6.0; U; cs)
Build Identifier: 2.0.0.9 (20071031)

When I began to use TB, I imported mail from Outlook Express. Later I noticed that some few messages were missing. I thought it was due to a bug in the import.
On 14 Dec 2007, I received a very nice e-mail message. After about 14 days, I viewed it again. After next about 14 days (i.e. a month after receiving it) I wanted to see the message again but it wasn't here. (Of course, I tried to find it many ways.) I was very unhappy about this so I found and opened the "Inbox" file where the messages were stored. Here I have found the message and also the missing imported old messages.
So TB doesn't display some messages. The message that once disappears, won't appear again after any restart of TB.
Please note that this bug affects only a few messages (randomly). I haven't counted them but I think it is approx. 1 % of messages.
I haven't found any missing messages in the sent mail, so it probably affects inbox only.
I am afraid of the folder cleanup because I think it might actually delete the messages (and that's why I consider this bug critical).

Reproducible: Sometimes

Steps to Reproduce:
1. Find someone who has imported a lot of messages to TB from another e-mail client (and still has the imported messages in that client). Alternatively, it can be someone who has a lot of messages simultaneously in TB inbox and on the mail server (but he/she has to know which messages he/she has deleted in TB).
2. Compare the two lists of messages meticulously.
---
If you can compare the list of messages in TB directly with messages actually stored in the "Inbox" file, then this is the best way.
Actual Results:  
Some messages are missing in TB (but they are in the "Inbox" file).

Expected Results:  
All the messages should be listed.

Some unusual settings:
* Installed to another directory than Program Files to avoid Vista's virtualization.
* Localization: Czech.
* Connection via WinProxy on another computer.
* Two accounts.
* Ask if work online or offline on startup.
* Don't download messages over 2000 kB.
* Leave messages on server for 30 days.
If they're in the inbox, but not visible, then they might have been moved to another folder. That also includes marking as junk (= moved to the junk folder) or deleted (moved to the trash). Their original space in the inbox might not yet be reclaimed by new mail, or by the compacting process (called 'expunching' if you're using IMAP). 

Are you really sure that they were not deleted or moved ?
Yes, I am.
Now I remember that I had some troubles with TB in the beginning. First, I downloaded Thunderbird Portable instead of "normal" Thunderbird (I didn't know the difference). Then I imported the folders from Outlook Express (one by one) and moved them to the correct folders. If there was not yet the folder (such as "Sent"), I created it and gave it a Czech name instead of the English one (that TB Portable was in English). After moving the imported messages (of one folder), I imported a next folder. There was a bug in showing folders/messages after the move (but they were actually moved).
Later I found out what TB Portable was and downloaded and installed the "normal" TB, in addition a newer version (2.0.0.6) and localized (Czech). I was disenchanted when I had found out that TB could import mail from four different e-mail clients but not from TB(P)!!! But I developed a quite complicated mail transfer from TBP to TB (based on copying the mail files) and it seemed to work correctly.
We (my mother and me) noticed the missing messages probably while using TB 2.0.0.6. I thought of a bug in import. But when I couldn't find a recent message (mentioned in the initial comment), it was obvious that the bug is not in the import and I decided to report it. I found out that there was v. 2.0.0.9, so I downloaded and installed it but it still didn't show the message. Then I definitely reported this bug.
The multiple mail transfers can be important for the bug but even a recent message which was received by TB 2.0.0.6 and which was visible (at least for 14 days) in TB 2.0.0.6, disappeared in TB 2.0.0.6 somewhen (although it is still stored). And TB 2.0.0.9 still doesn't show the message (which I surely didn't delete nor move nor mark as junk).
Look at the headers of the message in question. What does 
X-Mozilla-Status: 0009 and X-Mozilla-Status2: say? X-Mozilla-Status: 0009 means marked as deleted.

Just so we know if the problem is with the displaying somehow, or that it got deleted in the first place. 

No filters active, that could have done it? Have you looked in Junk too?

Here are the headers:

X-Account-Key: account2
X-UIDL: M100S3887911T1197640933
X-Mozilla-Status: 0401
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 

I have realized that the message had a large attachment and its size exceeded the limit 2000 kB so the attachment was not downloaded. (I didn't need to download it at home because I downloaded it separately at school where the net is much faster.)
But this doesn't apply to the old (imported) messages.
Here are the headers of one of the old invisible messages:

X-Mozilla-Status: 000d
X-Mozilla-Status2: 00000000

I have no active filters (actually, I know only one way to filter messages in TB: using the search box). I have no Junk folder because I simply delete (i.e. move) junk messages to the trash. I have searched all the existing mail folders in TB and the missing messages are simply nowhere.
Another message disappeared in TB. It is also a message with a large undownloaded attachment. Here are the headers:

X-Account-Key: account2
X-UIDL: M25S5124948T1200490085
X-Mozilla-Status: 0401
X-Mozilla-Status2: 10000000
X-Mozilla-Keys:                                                                                 

So there are probably two types of messages which disappear in TB:
* imported, old messages and
* messages which have not been fully downloaded due to exceeding the size limit.

Have you identified the bug? Or can my mailbox be corrupted since the transfer from TB Portable 2.0.0.4?
Should I send some files to you?
Version: unspecified → 2.0
The problem is NOT RESOLVED in TB 2.0.0.12 (20080213).

I have found an interesting fact: the invisible imported messages have X-Mozilla-Status: 000d, while the other imported messages have X-Mozilla-Status: 0011 or 0001 (or some other maybe). What is X-Mozilla-Status: 000d?
Sounds odd, are you using some add-on (or even some external program) that could be causing it?
No, I don't use any such thing.
The add-on manager in TB shows that only Talkback 2.0.0.9 is installed.
Have you tried deleting the .msf summary files?  It is disconcerting how the GUI can sometimes become out of sync.
Is it safe?
I suggest you make a backup first just in case, but, yes, it should be safe.  The .msf files are Mail Summary Files.  It is basically an index of the e-mails in the corresponding mbox file.

Close Tb, Make a backup, delete the .msf files, Start Tb, and when you click on a folder, it should recreate the .msf file for that folder.
(In reply to comment #7)
> X-Mozilla-Status: 0011 or 0001 or 0009 or 000d
4 characters of X-Mozilla-Status is hex-decimal representation of 2 bytes data.
And right most 4 bits(right most hexa-decimal digit) has deleted/not-deleted bit & unread/read bit.
           +-------- 0==Not-deleted, 1==Deleted
           |  +----- 0==Unread,      1==Read
           |  |
           V  V
   0x1 == "0001"B
   0x9 == "1001"B
   0xD == "1101"B

Delete of a mail in local mail folder is action of "move to Trash". Mail data is copied to Trash folder, and "Deleted bit==1" of original mail in original mail folder is set internally by Tb. When Shift+Delete, copy step to Trash folder is bypassed.

You can see above by next test.
 1. Create a folder(say Test), and copy 4 mails in it
 2. mail-1, mail-2 : set as Unread, mail-3, mail4 : set as Read
 3. Delete(or Shift+Delete) mail-2 & mail-4
 4. Click other folder(force close of file of Test)
 5. See file content of Test(not Test.msf) by text editor
    (Right most 4 bits of X-Mozilla-Status:)
    mail-1 : 0x0 == "0000"B == Not-deleted, Unread
    mail-2 : 0x8 == "1000"B == Deleted,     Unread
    mail-3 : 0x1 == "0001"B == Not-deleted, Read
    mail-4 : 0x9 == "1001"B == Deleted,     Read

To Jan Lachnitt(bug opener):

What is your "automatic deletion/purge of mail" related setting?
 1. Property/"Retention Policy" of all mail folders
 2. Account Settings
 2-1. Disk Spaces
 2-2. Junk Settings/Automatically delete junk mail older than NN days
 2-3. Server Settings
 2-3-A. Leave Messages on Server/For atmost NN days
 2-3-B. Empty Trash on Exit

When delete by "Retention Policy" or "Junk Purge", Tb 2.0 seems to execute similar action to "Shift+Delete", although "Junk Purge" of Tb 1.5 executed same action as "Delete"(move to Trash).


I have investigated it more properly and I am quite surprised (or shocked).
While writing all the past comments, I had investigated properly my mailbox only. It has 50 MB. More messages are missing in my mother's mailbox but it has 224 MB so opening it in a text editor is quite "difficult". I had supposed there was only one bug causing these problems (or maybe two) but now I have analyzed also my mother's mailbox and the results are surprising.
The description of the messages missing in TB in both my and my mother's inboxes is following.

*** MY INBOX ***

MISSING MESSAGE #1
It was imported to TB Portable 2.0.0.4 from an Outlook Express's .dbx file. This message absolutely can't be found in TB but it is present in the file. According to the status 000d, it is deleted, but I didn't delete the message and it is in no Trash folder. Here are the first few headers:

From - Mon Jan 1 00:00:00 1965
X-Mozilla-Status: 000d
X-Mozilla-Status2: 00000000

MISSING MESSAGES #2 and #3
These messages have been downloaded by TB 2.0.0.6, they are in the Inbox file, they don't have the deleted status but they absolutely can't be found in TB. Both have large attachments which haven't been downloaded because the size limit was exceeded. Here are the headers of the first of these messages:

From - Mon Dec 17 23:54:52 2007
X-Account-Key: account2
X-UIDL: M100S3887911T1197640933
X-Mozilla-Status: 0401
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 

*** MY MOTHER'S INBOX ***

MISSING MESSAGE #4
This message was imported to TBP 2.0.0.4 from OE (i.e. Outlook Express). It can be found using the search box in TB but otherwise it is not displayed (I am really not joking). It is the absolutely first message in the 224MB Inbox file. Here are the headers:

From - Mon Jan 1 00:00:00 1965
X-Mozilla-Status: 0011
X-Mozilla-Status2: 00400000

MISSING MESSAGES #5, #6, #7 and maybe some other (I stopped searching when it was three times the same)
These messages were also imported. Strictly speaking, they should have been imported but they probably haven't. Today, I have imported the source .dbx files again into OE 6 (on another computer). All these three messages are missing in TB and they are even not in the Inbox file. All these 3 messages have really many recipients (the "To:" header could cover many screens).

*** END OF MESSAGES ***

There are 4 different types of missing messages so it seems that TB is full of bugs.

Now I have deleted (moved) both my and my mother's Inbox.msf files and TB has recreated them. Great, messages #2, #3 and #4 are now visible in TB! Of course, messages #1 and #5 to #7 are not.
So there are probably two bugs in the import from OE in TBP 2.0.0.4 (messages #1 and #5 to #7) and then a bug (or maybe more bugs) connected with the .msf files. The messages #2 and #3 were lost while using TB 2.0.0.6 so I don't know if the bug is still in 2.0.0.12. I'll tell you if a message disappears again.
I have stored all the four Inbox.msf files so I can send them to you.

My settings:
1. Property/"Retention Policy" of all mail folders: I didn't know about these settings so I have never changed them. There is set "Don't delete any messages" in all my folders and "Use server defaults" in all other folders.
2-1. Disk Spaces: do not download messages larger than 2000 KB; Don't delete any messages
2-2. Junk Settings/Automatically delete junk mail older than NN days: No (the "Move new junk messages to:" checkbox is unchecked)
2-3-A. Leave Messages on Server/For atmost NN days: Yes, 30 days
2-3-B. Empty Trash on Exit: No
For mails of deleted_bit==1 :
Did you access mail folders(and execute move) by Tb 2.0.0.3 or before after import or download by TBP 2.0.0.4 or Thunderbird 2.0.0.6?
If yes, possibly due to Bug 368112 which was fixed by Tb 2.0.0.4.
Note: TBP 2.0.0.4 maybe doesn't include fix for Bug 368112, or fix for Bug 368112 maybe doesn't work on TBP 2.0.0.4, because TBP is independent/different build form official Thunderbird.

For MISSING MESSAGE #4 (mail of deleted_bit==0)
> It can be found using the search box in TB but otherwise it is not displayed (I am really not joking).   

Do you use View/Messages or View/Thread setting? Do you use Search folder(Virtual folder, Saved search folder)?
If yes, some bugs relate to View setting exist(view is not correctly set internally in some situations). If this problem, set View/Messages/All, click(open) other usual mail folder(not search folder) and click again(re-open the folder) usually corrects internal error and undisplayed mail wil be displayed.

> More messages are missing in my mother's mailbox but it has 224 MB

It looks that you fortunately never executed "Compact Folder", because mail data  of deleted_bit==1 still remains in your mail folder file. 
> MISSING MESSAGE #1
> X-Mozilla-Status: 000d

Have you kept back up of Profile directory of Tb(or Mail directory)?
If yes, you can recover mail data from back up even if incorrectly deleted mail(set as deleted_bit==1) is removed from mail data file by "Compact Folder", by (1) copy of mail data from backup, (2) reset of the deleted_bit, (3) Properties/Rebuild Index (or delete .msf & restart Tb, as you did).

Reduce file size by (1) Compact Folder(s) and (2) Split Inbox to multiple mail folders/subfolders.
For example, subfolders based on mail date.
  - Inbox
  - MyMbox
    - 2005
    - 2006
    - 2007




> Did you access mail folders(and execute move) by Tb 2.0.0.3 ...
The first TB I had was TBP 2.0.0.4. I imported mail from OE to it. I can't remember what exactly I did but I'm sure that I had to move some mail. It was probably because TB always imported the mail to the same account so I moved either all my mail or my mother's mail. There really was a bug but in displaying only. The messages were actually moved (at least most of them).
The message #1 has never been moved after that.
After TBP 2.0.0.4, I had TB 2.0.0.6, then 2.0.0.9 and now I have 2.0.0.12. I move no messages except deleting some unwanted messages to the trash. My mother maybe moved some messages from her inbox to her "Old mail" folder (which is even larger than the Inbox).

> For MISSING MESSAGE #4 (mail of deleted_bit==0)
> > It can be found using the search box in TB but otherwise it is not displayed (I am really not joking).
> 
> Do you use View/Messages or View/Thread setting? Do you use Search
> folder(Virtual folder, Saved search folder)?
I don't use such things.
Please note that rebuilding Inbox.msf solved the problem with message #4.

> It looks that you fortunately never executed "Compact Folder"
You're right.

> Have you kept back up of Profile directory of Tb(or Mail directory)?
I don't know. I would have to find some back-up DVDs but I'm to lazy to do this because I think I don't need the backup when I have never executed the "Compact Folder" command. Or am I wrong?

> Reduce file size by (1) Compact Folder(s) and (2) Split Inbox to multiple mail
> folders/subfolders.
I don't want to do these because:
(1) It would delete message #1 in my mailbox and there can be similar messages in my mother's mailbox.
(2) My mother's mailbox is already split (old messages have been moved to the "Old mail" folder already in OE). But, of course, if I need to work with the folders using a text editor, I'll reorganise them.

IMPORTANT:
Messages #2 and #3 disappeared again (in TB 2.0.0.12).
Using latest TB3, WinXP
I only started using TB a few weeks ago, then upgraded to TB3 this week. Have not got multiple folders,accounts. Have not imported mail from anywhere. Have not set up any filters.

I check POP3 mailbox with Mailwasher so I can see what is waiting to be retrieved (anddelete spam) before calling TB3 to GET MAIL.  For example:  I can see 3 emails in POP3 from the same person (work colleague). 

Use TB3 to Get Mail. I DELETE two spam emails. Then look at colleague's emails -- there are only 2 of those in the Inbox. I try GET MAIL again. Still only 2 emails.

I look in TB3's \ApplicationData\....\ Inbox file.  Near the end I can see the 3rd email. It has 
     X-Mozilla-Status: 0000
     X-Mozilla-Status2: 00000000

I check the Deleted folder. It is not there.
I check the POP3 mailbox. The 3 emails are still there.

I have noticed similar behaviour since I started using TB. But only occasionally.  

CONCLUSION:  TB3 sometimes retrieves email from POP3 and does not display it in the inbox.  However the email is stored on disk, with Status set 0000.

(In reply to comment #17)
Additional Information.
After writing the above, I notice colleague has sent me another email (I see it on POP3). Using TB3 I use Get Mail. But this email does not appear in inbox.  However, if I look in the datafile inbox, I can see this email. Again the Status is set: 
   X-Mozilla-Status: 0000
   X-Mozilla-Status2: 00000000

These emails are all replies to the same original question, so all emails have the same Subject:  eg  Re:MyProblem 1234
The latest one does have an attachment. Others do not.

TB3 has chosen to display two of the 4 emails, but hide the other two.

If I use a different email reader, it correctly retrieves and displays these emails.

I regard this as a critical bug in TB3.
It's often better to file a new bug for these cases since there is no telling if this would be the same problem or not.

Anyway, are you using threading? Does "rebuild index" from the folder properties fix it? 
I was not using threading. When I turn on threading I can see the "missing" messages, and problem is solved.  The sender of the emails has for some reason been playing with his computer date/time settings and those "missing" emails have a date in last year. They were way out of sight when viewed in date order.

So TB3 was working correctly.
I apologise for suggesting it was a bug in the system.  And thanks for your comment which has solved the mystery.
I have installed TB 2.0.0.16 (20080708). The bug is still there. Some messages disappear and to view them again, it's necessary to rebuild the folder index. It affects especially incomplete messages (which haven't been downloaded due to their size). Now my mother has an incomplete message and this disappears even while TB is running. It is visible only for a while after index rebuilding.

The other bug(s) (i.e. in the import form Outlook Express) is/are not so obvious, simply because I no longer import anything. However, if they haven't been fixed yet, they are still there, too.
My mother often receives large e-mails and these are (automatically) downloaded only partially (we use the limit of 2000 KB due to slow Internet connection). She wanted to fully download such a message (the newest one). One of download attempts was successful (our connection is also a bit unstable). But then we remembered a few days old incomplete message which we'd like to download, too. But it wasn't there. So I rebuilt the index and then it appeared, of course. But more things happened:

TB normally shows size of the whole message even if there are only first few lines downloaded. But since the rebuilding of the index, TB shows only sizes of the downloaded parts of such messages (e.g. 2KB instead of 3658KB). Moreover, the new, fully downloaded message now appears twice: incomplete and complete (each with the corresponding size).

So the problem may be related to the difference between declared size of the message and the size of actually downloaded data.

Btw. that message we wanted to download has disappeared again. (Note: I think storing all the messages in a single big file is not a good idea.)
(In reply to comment #21)
> 
> The other bug(s) (i.e. in the import form Outlook Express) is/are not so
> obvious, simply because I no longer import anything. However, if they haven't
> been fixed yet, they are still there, too.

Jan, if you still have that old outlook message store, could you test using version 3 beta and file a new bug on issues found on import?
 
http://www.mozillamessaging.com/en-US/thunderbird/early_releases/
I've just tried it. I have found a single bug: bug 484605.
Component: General → Import
Keywords: dataloss
Product: Thunderbird → MailNews Core
QA Contact: general → import
Version: 2.0 → 1.8 Branch
The bug of disappearing messages (i.e. this bug) is still there (Mozilla/5.0 (Windows; U; Windows NT 6.0; cs; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2). Here is a REPRODUCIBLE case when a message disappears:

The size of the message exceeds the preset limit (2000 KB in my case) so there are only the first few lines (usually the text without attachments) downloaded. TB shows an info-box at the end of the message with a link to download the whole message. I click the link to download it and TB starts downloading. If the download fails (due to lost connection), the message ALWAYS disappears. It may be restored by index rebuilding or by a successful download (the message disappears from the list but not from the preview so it's possible to retry the download).

There may be other ways for a message to erroneously disappear but the described way has been proved many times.
Component: Import → Database
QA Contact: import → database
from where are you getting the portable version?
Summary: Losing mail little by little → Losing mail little by little [pop]
We should try the new STR in comment 25.

Is Mailnews: Database the right component? I'd think not.
Whiteboard: qawanted for STR in comment 25
(In reply to comment #27)
 
> Is Mailnews: Database the right component? I'd think not.

Why not ?
Whiteboard: qawanted for STR in comment 25
networking - pop3 or mail backend would be better components. This doesn't have anything to do with .msf files.
Component: Database → Backend
QA Contact: database → backend
Well, it seems something is happening here and moreover I still can't find better e-mail client than TB is, so I'll probably have to write a new comment here :-)

(In reply to comment #26)

> from where are you getting the portable version?

I got it from this web: http://portableapps.com/

Now I use TB 3.0, exactly this:
Mozilla/5.0 (Windows; U; Windows NT 6.0; cs; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

Incomplete messages still sometimes disappear. The exact behavior seems to differ between particular beta/RC/stable versions but I haven't noticed any new regular behavior.
Now we have a better Internet connection (both faster and more stable), so my mother downloads most of new big messages. We haven't noticed any message to disappear since we have the better connection (i.e. not since using a new version of TB). So I guess the bug is still there but it just doesn't manifest itself in the new conditions. See comment 25.

TB3 seems to rebuild the index files automatically sometimes. Many messages in my mother's inbox have been duplicated, surprisingly not only the big ones.

Now we use TB 3.1.2 and this hopefully doesn't duplicate messages.
Thanks Jen for the update.
=> WFM unless someone thinks there is something that still reproduces in 3.1
Status: UNCONFIRMED → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
I've just tried to reproduce the bug in TB 3.1.9 using the way described in comment 25. Because our current Internet connection is stable, I aborted the download with firewall.

Result: The progress bar in status bar "kept on moving", just like if the download was continuing. However, the message didn't disappear and it's still here. So you have probably fixed this bug, though nobody knows when :-)
You need to log in before you can comment on or make changes to this bug.