402088 - IMAP database is cleared on unsolicited UIDVALIDITY

Status: RESOLVED WONTFIX

(MailNews Core :: Networking: IMAP, defect)

Description

[Thomas Boerkel]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.8) Gecko/20071008 Firefox/2.0.0.8
Build Identifier: version 2.0.0.6 (20070728)

IMAP account, server is uw-imap 2006g on Fedora Core 5.

Thunderbird is running and the newest mail of the inbox is being marked/positioned on. 

Now, when a new mail arrives, TB informs about it with the popup and at the same time the whole inbox mail list (upper right window) goes blank.

When I switch to another folder and back to the inbox, everything is OK again.

I have tested this with TB 2.0.0.5 and 2.0.0.6 on 2 Windows machines and 1 Linux machine.

It does not happen, if the inbox only has a few mails. So it's either the number of mails or the inbox size that triggers the bug.

I already deleted the INBOX.msf file several times and I already recreated the Inbox (/var/log/mail/<name>) on the server.

I am attaching an IMAP log. The log captured the steps to reproduce below.

Reproducible: Always

Steps to Reproduce:
1. Start TB
2. Position on the newest mail in the IMAP inbox
3. Wait for new mail to arrive
4. Inbox mail list goes blank
Actual Results:  
Inbox mail list goes blank.

Expected Results:  
All old mails and the newly arrived mail should be listed.

Comment 1

[Thomas Boerkel]

Attachment: Thunderbird level 5 IMAP log of the repro steps

Comment 2

[Nikolay Shopik]

Could you test it using latest TB?

Comment 3

[Thomas Boerkel]

Still happens with 2.0.0.16, but not as often, I think.

Comment 4

[Joshua Cranmer [:jcranmer]]

Looking through the log, these lines jump out in part:

-1288336480[9716eb0]: 96f4420:linux:A:SendData: 21 select "INBOX"
-1288336480[9716eb0]: 96f4420:linux:A:CreateNewLineFromSocket: * OK [UIDVALIDITY 1193939993] UID validity status
[ ... ]
-1288336480[9716eb0]: 96f4420:linux:S-INBOX:SendData: 25 IDLE
-1288336480[9716eb0]: 96f4420:linux:S-INBOX:CreateNewLineFromSocket: + Waiting for DONE
-1288336480[9716eb0]: 96f4420:linux:S-INBOX:SendData: DONE
-1288336480[9716eb0]: 96f4420:linux:S-INBOX:CreateNewLineFromSocket: * OK [UIDVALIDITY 1193940285] UID validity status

The log then shows us rebuilding the index because the uidvalidity reset. Thoughts, bienvenu?

Comment 5

[David :Bienvenu]

That looks like the UW IMAP server changing UID validity in the middle of a session. I did not think that was allowed.

Comment 6

[Nikolay Shopik]

David, according RFC 2683(IMAP4 Implementation Recommendations) 3.4.3.
The server should maintain UIDs permanently for all messages if it can.  If that's not possible, the server must change the UIDVALIDITY value for the mailbox whenever any of the UIDs may have become invalid.  Clients must recognize that the UIDVALIDITY has changed and must respond to that condition by throwing away any information that they have saved about UIDs in that mailbox.

Comment 7

[Joshua Cranmer [:jcranmer]]

RFC 3501 (which supersedes RFC 2683):
The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions.  Any change of unique identifiers between sessions MUST be detectable using the UIDVALIDITY mechanism discussed below.

This doesn't prohibit UIDVALIDITY from being sent mid-session, but it does prohibit the UIDs from being changed (which is mostly what UIDVALIDITY is designed for).

The issue here is that we're getting a UIDVALIDITY at the end of an IDLE command; the UIDs don't seem to be affected though, so I'm not sure why UW IMAP is bumping the validity.

Comment 8

[Nikolay Shopik]

I don't see anywhere in RFC 2683 or RFC 3501 mention if last one superseed RFC 2683 its only obsolete RFC 2060.
And kind strange to see UW-IMAP doesn't honor RFC 3501 both comes from University of Washington ;).

Comment 9

[Joshua Cranmer [:jcranmer]]

The problem here is obviously a UIDVALIDITY issue. Since the UIDs aren't affected, UW IMAP is probably obeying the spec which only guarantees that UIDs can't change. The important question, then, is whether or not to assume that unsolicited UIDVALIDITY responses are following the spec and not munging the UIDs.

Comment 10

[Nikolay Shopik]

This can be issue of UW-IMAP and it seems UW-IMAP not actively developed. Panda IMAP is now fork of UW-IMAP with some past developers of original server.

Comment 11

[Nikolay Shopik]

So nothing we can do about that, suggesting as INVALID or WONTFIX

Comment 12

[David :Bienvenu]

we have to blow away the .msf file on uid validity change. Doing it sooner rather than later doesn't seem like a bad thing to me, in those rare cases where the server does it in the middle of a session.

Comment 13

[Thomas Boerkel]

Why WONTFIX, when the solution is in comment #12?

Comment 14

[David :Bienvenu]

#c12 describes what we're doing now, afaict - one solution would be to postpone the blowing away of the .msf file until we re-open the connection, but that seems a bit more dangerous. But this is more of a wontfix because the UW server isn't supported anymore, and I don't really have a good way of reproducing this bug, or testing a fix. If we're refetching the headers, I'm not sure why the view is left blank...

Comment 15

[Thomas Boerkel]

That is the only problem. It shows the number of mails and new mails correctly in the folder list on the left. But the mails list is blank.