236708 - IDLE Command: Frequent compacting of folders, re downloading of all messages

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Scott MacGregor]

Ever since I started using the IDLE command, thunderbird is frequently throwing
away the local message store and redownloading all of the messages in my inbox.
Sometimes I see "Compacting Folders" flash by in the status bar before this
action occurrs.

Other times, it says loading folders and does nothing, the throbber just spins.
I try clicking on messages and nothing loads. The weird thing is once I'm in
this state, I seem to be locked out of my inbox. I keep quitting and restarting
and it just hangs for a long time.

Comment 1

[Scott MacGregor]

Attachment: log when starting up in this 'locked state' just causes us to spin...unable to load folders

Comment 2

[Scott MacGregor]

Attachment: another log

This contains the same log as the first attachment. But while typing up the bug
report, eventually the folder got out of this busy state with the arrival of
two new messages and suddenly the connection was loading message urls again. So
after a long while it somehow got 'unstuck'. 

Same session as the first log.

Comment 3

[Scott MacGregor]

Weird, right after saving these two logs, I deleted a message and we ended
updownloading the inbox over again.

Comment 4

[David :Bienvenu]

the first log was from fresh start of thunderbird? First of all, we're issuing
the select to the server, but the server is never responding. We're also queuing
a url to fetch a message, queuing
url:imap://[e-mail]:143/fetch>UID>.INBOX>288 - I have no
idea why we think we know we need that message, unless you clicked on it - that
would explain that, I guess...but as to why the server never responds to the
select, I wonder if the connection was left in the IDLE state from a previous
tbird session, and that confuses the server. Perhaps the code that logs out on
shutdown needs to end the idle first - but we might need to do that
synchronously, which will re-introduce a problem you were having earlier...I can
try it, anyway.

Comment 5

[David :Bienvenu]

I wonder if the failure to go out of IDLE state is also confusing your server
into rolling uid validity...

Comment 6

[David :Bienvenu]

Attachment: proposed fix

if we're idle, and the connection is open, end the idle. Also, need to make
sure we set the connection open flag, so unback out part of the patch that
fixed that. Hopefully this won't cause the hang on exit that Scott was seeing
earlier that caused us to back out the original patch.

Comment 7

[Scott MacGregor]

Comment on attachment 143209 [details] [diff] [review]
proposed fix

so far so good with this patch. I'll keep you posted.

Comment 8

[David :Bienvenu]

fix checked in.

Comment 9

[Scott MacGregor]

Things definetly seem better but I still  have some problems.

If I am away from my machine for a while, I eventually end up in a state where
the inbox will not load messages anymore. Note, in this session I did not click
on any other folders besides the inbox and the sent folder.

213c1c0:mail.scott-macgregor.org:S-INBOX:ProcessCurrentURL:imap://[e-mail]:143/customKeywords%3EUID%3E.INBOX%3E348%3EJunk%3ENonJunk:
 = currentUrl
1088[23ff130]: 213c1c0:mail.scott-macgregor.org:S-INBOX:SendData: 85 IDLE

1088[23ff130]: ReadNextLine [stream=2413248 nb=22 needmore=0]
1088[23ff130]: 213c1c0:mail.scott-macgregor.org:S-INBOX:CreateNewLineFromSocket:
+ entering idle mode

1088[23ff130]: 213c1c0:mail.scott-macgregor.org:S-INBOX:SendData: DONE

0[2942d8]: queuing url:imap://[e-mail]:143/select>.INBOX
0[2942d8]: considering playing queued
url:imap://[e-mail]:143/select>.INBOX
0[2942d8]: creating protocol instance to play queued
url:imap://[e-mail]:143/select>.INBOX
0[2942d8]: failed creating protocol instance to play queued
url:imap://[e-mail]:143/select>.INBOX
0[2942d8]: queuing
url:imap://[e-mail]:143/fetch>UID>.INBOX>267
0[2942d8]: considering playing queued
url:imap://[e-mail]:143/select>.INBOX
0[2942d8]: creating protocol instance to play queued
url:imap://[e-mail]:143/select>.INBOX
0[2942d8]: failed creating protocol instance to play queued
url:imap://[e-mail]:143/select>.INBOX

Comment 10

[David :Bienvenu]

We're trying to store the junk/non-junk flags, but your server probably doesn't
support user-defined keywords, so the first url doesn't do anything.  Then, we
went IDLE, and then we came out of IDLE with DONE. Oh, I see the problem - your
server is never telling us anything after the DONE command, so we're blocked
waiting for a response. It's supposed to send an OK response. I'm not sure if
this is a server problem, or a problem in our code trying to read the DONE
response from the stream - I had this problem during development, but clearing
the async wait fixed it for me...