543399 - disk cache getting corrupted with incorrect mapping to imap messages

Status: RESOLVED INCOMPLETE

(Thunderbird :: Folder and Message Lists, defect)

Description

[nicks_bugzilla]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

this has happened at least twice, I completely wiped my profile in between occurrences.  It seems to happen when you have a message selected in the inbox message list pane but the message in question has not yet completed loading from the server in the reading pane and -- before it loads - while it is in process of loading - you click on a different message in the message list pane.  the message that had been loading is now displayed when you click on either the correct message (the new one) or the old one.  it seems like it overwrites the reference to the message content for the currently selected message in the message list pane when the message is finished loading from the server.  this is a huge bug.  it may be related to using the message filter search box while the message is loading.  

Reproducible: Always

Steps to Reproduce:
1. launch TB with new messages on the server. 
2. while a new message is still loading, select a different message in the inbox list

Actual Results:  
3. the message that finishes loading replaces the old previously loaded message contents but not header information  in the list view

Expected Results:  
uh the message that was supposed to be there should still be there.  

let me know if you have any questions or would like to see an example of this behaviour.  i am happy to provide any additional information that might be helpful.

Comment 1

[Nathan Tuggy (:tuggyne)]

Looks like this is a dupe of bug 536918 or bug 265268. Please resolve duplicate of one of those if possible, or note how this bug differs.

Comment 2

[nicks_bugzilla]

bug 265268 refers to version 0.8 from 2004 and the problem referenced is that the header information displayed in the message list pane differs from the actual information in the message pane.  

in the bug I am describing, the headers shown in the message list are correct but two or more headers point to the same message body, making at least one of the messages unaccessible.  I also describe how and when this (apparently) occurs.   

bug 536918 seems more similar, however the message content that is displayed is not a previously deleted message, nor a multipart message, and offline use is not selected.  in addition, rebuilding the index does not resolve this issue as it appears to at least temporarily resolve the issue in bug 536918.  

however if you feel that this is the same bug I would be happy to continue my comments in bug 536918.  


Additional information: 
* I am using IMAP folders (using gmail). 
* Offline use is not checked.
* The message header in the message list pane references the incorrect message body in both the smart folders global inbox, and theh all folders view, as well as in the individual account inbox. 
* The Order Received column in the message list shows different numbers, but both header lines reference the same message body. 
* Neither message appears to be a multipart message (no attachments)
* Thunderbird is online and receiving new messages properly and this still occurs
* View message source displays the duplicated message, not the correct reference

Comment 3

[nicks_bugzilla]

also - the messages all remain correctly referenced on gmail.

Comment 4

[Nathan Tuggy (:tuggyne)]

No, you're right, this isn't the same bug as either of those (or at any rate has different apparent characteristics). I'm unable to reproduce (perhaps because I don't use GMail?) on Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.7) Gecko/20100111 Lightning/1.0b2pre Thunderbird/3.0.1 ThunderBrowse/3.2.8.1 ID:20100111101938.

Comment 5

[Phil Evans]

I think I am seeing the same bug, in Linux (OpenSuse 11.2/kernel: 2.6.31.8-0.1-desktop, TB 3.0). It only happens occasionally, but for example right now I am viewing an email (from an IMAP server) and the header is for a short mail to me from my boss, the body contains a different e-mail, followed immediately by another one (complete with headers). Both of these 2 I've actually deleted.

On the other hand, checking my e-mail with pine shows me that the original e-mail is still there and thunderbird is clearly showing me the wrong thing.
I have tried closing and restarting TB with no effect, although rebuilding the index fixed the problem. Clearly this shouldn't be happening though!

Comment 6

[nicks_bugzilla]

I see someone added "Gmail tester needed" to the whiteboard comments in this bug's header information.  

I would be happy to volunteer to help with this problem - this is an incredibly serious bug and I have already installed Outlook and Zimbra to try to escape it but both of them have their problems too.  If they didn't have such problems I'd be through with Thunderbird already.  As it stands, their problems trump this problem so far, but if this keeps happening all of my messages will eventually start to be unreadable.  

If there was even a workaround I'd be happy to use it - like deleting my mail indexes or something once in a while.  

Should I post screenshots?  Log files?  Anything?  

I've been using Thunderbird for many years and I'd really like to continue, but this bug is a true show-stopper.  

Unfortunately I am not a programmer or a developer, just a run of the mill computer expert.  

Please help and let me know what I can do to help!

Comment 7

[David :Bienvenu]

Nick, if rebuilding the index doesn't help, then you could try clearing the disk cache - tools | options | advanced, network and disk space, click the clear now button for the disk cache.

If that doesn't help, the issue has to be either on the server (highly unlikely), or an issue with the .msf + offline store, even though you say you don't have offline storage selected. You might look in your thunderbird profile under the ImapMail sub-directory for your server and see if there really is an offline store for your Inbox (there would be an Inbox.msf and an Inbox file, if so).

Comment 8

[David :Bienvenu]

nick, if it does turn out to be the disk cache, you can simply turn it off by setting a 0 disk cache size.

Comment 9

[nicks_bugzilla]

David - a thousand thanks!  The disk cache appears to have been the problem.  

My offer to do whatever I can to help resolve this permanently stands, and I am extremely relieved and happy to be back in a 100% usable state again.  

Cheers!  

 - Nick

Comment 10

[David :Bienvenu]

Nick, glad that helped, though I'm not thrilled that the disk cache is getting corrupted :-( I'm pretty sure this is much more likely to happen if the offline store is turned off, because we mostly do not use the disk cache if the offline store is turned on.

I'll try to think how you can help. I may need to add some sort of logging to track this down, in which case I could get you a try server build with the logging. And if I find a fix, I could generate a try server build for you to try to verify that the issue is fixed.

Comment 11

[zug_treno]

Related to MailNews Core bug 531033?

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)

> Steps to Reproduce:
> 1. launch TB with new messages on the server. 
> 2. while a new message is still loading, select a different message in the
inbox list

And, you refer to quick search.
  3. while loading by 2, type chars slowly at quick search box(body search).
     (interere step 2 by incremetal search repeatedly)
  4. Do 2 & 3 repeatedly.

Following occur. (mail-1=UID1, mail-2=UID2) 
 At Inbox;
 (a) uid UID1 fetch body.peek ...
 (b) Cancel of data receiving by "fetch for UID1"
     uid UID2 fetch body.peek ...
 (c) Type "abc" slowly. (see bug 541400)

     uid SEARCH UNDELETED BODY "a"
     uid SEARCH UNDELETED BODY "ab"
     uid SEARCH UNDELETED BODY "abc"
(a), (b), (c) can interfere processing of data returned by prevous command.

> Actual Results:  
> 3. the message that finishes loading replaces the old previously loaded message contents
> but not header information  in the list view

Show "Order Received" column, please. value is UID of mail if IMAP.
Say first clicked mail=mail-1=UID1, second clicked mail=mail-2=UID2.
After UID2 is clicked while UID1 is loading, what is displayed at where?
(what: content of which mail)
(where: thread pane, header box of message pane, message area of message pane, attachment pane, ...)

What is View setting?
(Message Body As/Original HTML, Simple HTML, Plain Text)
(Display Attachments Inline=On/Off)
Plain text mail? HTML mail? If HTML, mail of complex structure? 
Small mail? Larfe mail?

[e-mail](bug opener), can you check with small folder(3 mails only in IMAP folder) and get IMAP log and check Disk Cache file content? (See attachments to bug 552806, please)
> https://wiki.mozilla.org/MailNews:Logging
> http://www.mozilla.org/projects/nspr/reference/html/prlog.html#25328
> SET NSPR_LOG_MODULES=timestamp,imap:5

(1) Create an IMAP folder, copy three mails(UID1, UID2, UID3)
(2) Click the folder(fetch headers)
(3) Clear Disk Cache, terminate Tb, restart Tb with NSPR logging enabled.
(4) Click mail of UID1
(5) While loading UID1, click mail of UID2
(6) After messge pane display is changed, keep back up of files in Disk Cache.
      _CACHE_MAP_ , _CACHE_001_ , _CACHE_002_ , _CACHE_003_ ,
      and other files with no file extention.
(7) Terminate Tb, keep backup of log file,
    keep backup of ".msf" file for the IMAP folder.
(8) Check IMAP log.
    Check file content of files in backup of Disk Cache directory.
Does funny/corrupted data or funny/wrong flow is seen in file or log?
(If no confidential data or sensitive data in files, log, .msf, attach files to this bug, please. Before attaching, check content of files, log, .msf by text editor, and, if there are very many lines of irrelevant data, get file/log/.msg again with minimum test scenario, please.) 
       
Note:
Log for other mail folders makes log analysis hard. Disable auto mail check etc. please when you get IMAP log, for minimum log data.
And, if fetch of other mails happen, Disk Cache file content check becomes hard. Please inhibit any access to mails other than three mails for test.

Comment 14

[WADA:World Anti-bad-Duping Agency]

How to know "Disk Cache" directory location:
  At Tools/Options/General, Thunderbird Start Page, enable stat page.
  Set Location: about:cache?device=disk

Comment 15

[Wayne Mery (:wsmwk)]

please reply to questions in comment 13.
and, is there connection to bug 531033 as suggested by zug?

Comment 16

[Joshua Cranmer [:jcranmer]]

RESOLVED INCOMPLETE due to lack of response to last question. If you feel this change was made in error, please respond to this bug with your reasons why.

Comment 17

[Wayne Mery (:wsmwk)]

bienvenu do we need a new bug for comment 10?

Comment 18

[David :Bienvenu]

(In reply to comment #17)
> bienvenu do we need a new bug for comment 10?
> 
> *** This bug has been marked as a duplicate of bug 531033 ***

Wayne, I think we probably need to distinguish between the offline store getting corrupted and the disk cache getting corrupted, but what we need more than anything is some way of reproducing issues.

Comment 19

[WADA:World Anti-bad-Duping Agency]

Re-opening, as this bug shouldn't be dup'ed to bug 531033.
Not all problems which can be bypassed by Disk Cache=Off is same problem.
Mixing of offline-use=on case and offline-use=off case merely makes problem analysis difficult.
And, closing as INCOMPLETE again, as Joshua Cranmer did on 2010-06-05.

- That bug is for offline-use=on  and bad data in offline-store file.
- This bug is for offline-use=off and bad data in Disk Cache(or Memory Cache).

- "Bad mail data" of that bug seems same as Bug 587528 and Bug 604620.
  (mail body of correct mail + whole mail data) 
- What is "bad mail data" of this bug is still unclear.
　What can be said is only "shown mail data is different from correct data".