209561 - duplicate INBOX headers, mismatched headers/content

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Jim Longino]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030529
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030529

I believe there's an IMAP header caching issue that affects both Mozilla (1.31
and 1.4) and Netscape 7.0.  The most common problem is the duplication of
entries in the INBOX.  We've (University of South Alabama) had 5-10 separate
reports of this problem.  However, if we view the same accounts using IMAP under
Outlook Express, there is no duplication.  Likewise, the iPlanet webmail
interface shows no duplication either. 

Fewer instances (2-3) have been reported where the header entries don't
correspond to the correct messages (not every single message, just a few).

Reproducible: Sometimes

Steps to Reproduce:
I don't know how the problem begins, only that once it appears, deleting the
account profile and recreating it doesn't help.  I also upgraded from 1.31 to
1.4 and the problem still exists (indicating a possible caching problem?).  The
symptoms do not appear on the same accounts using Outlook Express or iPlanet
webmail interfaces (also IMAP configurations).



I think that Mozilla should have an option to rebuild the header cache even if
there were no problem.  This definitely would prevent me from recommending
campus users from installing or using Mozilla as a mail client.  We had only
recently began supporting Mozilla users, but in light of this problem will
probably recommend either Outlook Express (ugh) or the iPlanet webmail interface.

Comment 1

[Paul Casteels]

The same here, I installed Mozilla together with Netscape 7.02 but with a new
profile and all the headers in INBOX appear twice.
Other folders look ok.

Comment 2

[David :Bienvenu]

Has anyone tried this with a recent 1.5 build?  What IMAP server are you using?

If deleting the profile and starting again don't fix the problem, then deleting
the cache files won't help, since the cache files are stored in the profile. Or
maybe I'm not understanding what you mean by account profile.

There are three bugs that sound related, all fixed, but none of them really
seems to match what you're describing. The first one is running against the UW
IMAP server that comes with RedHat - it often rolls uid validity on the imap
server, so that the UID for a message changes out from under us. This resulted
in the client getting the wrong message from the memory cache sometimes, though
shutting down and restarting fixed that. Running against a newer version of the
UW server also fixed that problem.
The second problem had to do with the offline cache getting corrupted when
auto-compact fired when we were downloading messages for offline use. If you
check the box that says "make messages in my inbox available when I'm offline",
you would have been vulnerable to this problem. This has been fixed.
The third, least likely related, problem, had to do with line endings on the
server, where the server would return incorrect line endings, e.g., CRCRLF,
which would produce strange looking messages. That also has been fixed.

If you have a reproducible case of this in 1.5, could you attach an imap
protocol log and say which messages are appearing twice?

http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap

Comment 3

[Simon Paquet [:sipaq]]

not blocking development.
severity -> major

Comment 4

[Matt McClure]

I have about 5 users (out of 15) that have reported this in Mozilla 1.6/1.7
using our Fedora Core 2 dovecot-0.99.10.5-0.FC2 IMAP server.  I have also seen
it in Thunderbird, but I'm not certain what version I was using at the time -- I
recently upgraded to 0.7.3 -- and I haven't been able to reproduce it in a
controlled way.

I don't think any of the three bugs mentioned in comment 2 is the same thing.

Jim, did you ever figure out anything more regarding what was causing this when
you saw it?

Comment 5

[David :Bienvenu]

Are any of these folders configured for offline use?

There was a bug in 1.6, fixed in 1.7, where if you compacted an imap folder on
certain versions of the red hat UW server, you could get the wrong message when
clicking on a header, until you shutdown mozilla and restarted, because of some
problems with the memory cache, and the fact that expunging an imap folder could
cause uid validity to roll on the server. But if clicking on a message
persistently gives the wrong message, I'd suspect the offline store got
corrupted. You have to explicitly turn on using the offline store, however.

Comment 6

[Matt McClure]

Thanks for the quick reply.

I still don't have a reproducible case for this, but it just happened to me, and
I have an IMAP protocol log.

The folder in question is *not* selected for offline use.

I am using Thunderbird version 0.7.3 (20040803) on Windows XP SP2.  Is
Thunderbird 0.7.3 built from the trunk, or does it have its own branch, or is it
built from the 1.6 or the 1.7 branch?  Is there a doc somewhere that has a map
of the branches various releases came from?

Quitting Thunderbird and restarting did get things back in sync.  I have also
observed on previous occasions that moving the out of sync messages to another
folder and back again gets things in sync again.

Comment 7

[David :Bienvenu]

tbird .7.3 is built from a branch off of the aviary branch. 1.6 and 1.7 are
built off of different branches from the trunk, and the aviary branch is also a
branch off the trunk. The aviary branch was made off the trunk after the 1.7
branch was made, I believe, but roughly around the same time...

anyway, I'd be interested in seeing the log. It sounds like the memory cache
entry is getting messed up in your case. I did fix a similar problem in bug
249195 but that fix should be in .7.3, I believe.

Comment 8

[Matt McClure]

Attachment: IMAP log exhibiting mismatched headers bug (see comment 6)

I removed message bodies from the log, but left the X-UID headers downloaded in
those cases.  Otherwise, the attached log is exactly what Thunderbird spit out.


There were three consecutive messages that showed the wrong bodies.  Each
showed a message that I had previously deleted.  The portion of the log that
corresponds to me selecting those messages in the message list starts at line
14228.

Comment 9

[Matt McClure]

Attachment: IMAP log exhibiting duplicate headers bug

David,

Here is a log that exhibits the duplicate headers behavior.  It was fairly easy
to reproduce by having two IMAP clients open on separate machines.  Here are
the steps I used:

1. Open Mozilla 1.7.2 on machine A. (set check for new messages every 1 minute;
check for new messages in Aincoming/test)
2. Open Thunderbird 0.7.3 on machine B. (set check for new messages every 1
minute; check for new messages in Aincoming/test)
3. Send messages from a third mail client to myself so that they'll match the
procmail filter that dumps into Aincoming/test.

In the case of the log attached, after I sent the third message "test 42", that
message started showing up repeatedly.	You'll notice that in the log,
Thunderbird started FETCHing that message's headers over and over with a
different UID each time.

I watched the actual mbox file on the server, and it never contained more than
one copy of the message.  But its UID did change repeatedly.

I'm a novice at interpreting the IMAP log, so it's hard for me to tell if the
server is telling Thunderbird that the UID keeps changing, or if Thunderbird is
changing the UID and telling the server.  Can you figure it out from the log?

Please let me know if there's any more tests you want me to do, or any other
information I can give you.

PS.  The summary of this bug indicates both behaviors -- duplicate headers, and
mismatched headers -- but are we dealing with one bug or two separate issues
here?

Comment 10

[Matt McClure]

I posted a message to the Dovecot mailing list also:

http://dovecot.org/pipermail/dovecot/2004-September/004674.html

Comment 11

[Wayne Mery (:wsmwk)]

David wrote:
> It sounds like the memory cache entry is getting messed up in your case. I did fix a similar problem in bug 249195 [still not closed] but that fix should be in .7.3, I believe.

David, is anything useful in logs?  Matt no longer uses thunderbird and can't tell if reporter is still here.   If not, => incomplete  since data are 3+ major releases old and all commenters gone

Comment 12

[David :Bienvenu]

those both look like issues with the dovecot server, which was very new at that time, iirc. It was giving Thunderbird the same message with different uids, from what I could tell.