120106 - IMAP: Empty mail when disconnected

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Nahor]

I have a little IMAP mail notifier and I read my mail with IMAP in Mozilla.

- Load the mail with Mozilla (but don't read them).
- Check the mail with the notifier
- Click on one of the unread mail. The body is empty when it should have something.

I don't know if it will happen with all IMAP servers. I have the IMAP from the
Washington university (I think) that comes with Mandrake Linux. This server
seems to accept only on connection per user. If you are already logged in, it
disconnects the old connection to replace it with the new one.
So in this bug, Mozilla gets disconnected so that my notifier can check if there
are new mails. Then mozilla tries to get the body of the message but being
disconnected, it doesn't get anything.
Then I can't read this email anymore because Mozilla seems to keep a cache of
the body. I have to move the mail to another folder to be able to read it.
People who don't have several folders maybe in trouble.

Comment 1

[Ere Maijala (slow)]

Confirming, build 20020114 Win2K. I'm also using UW IMAP. I think the reporter's
analysis is correct. Anyway, here are also my experiences. 

Steps:

1. send yourself a mail
2. get new messages (the header is displayed)
3. open another connection to the mail server (I used IMP, using Moz from
another machine is okay too)
4. switch back to Mozilla and click on the message header

Result: for a short while the throbber runs, then the message loading stops, but
nothing is displayed. First the message status is not changed to read, but
moving to another message and then coming back will mark it read. 

Only ways to get Moz to show the message are:
1. clear the memory cache
2. restart Moz (effectively clearing the memory cache also)

Comment 2

[Ere Maijala (slow)]

*** Bug 126150 has been marked as a duplicate of this bug. ***

Comment 3

[Nahor]

I have another symptom of this problem:

1. Have a message in your inbox that you want to delete (spam, test mail, ...)
2. Select it 
3. Check for new emails with an other software or on another machine (get
mozilla disconnected from the IMAP server)
4. Delete your message (I have them to be moved to my Trash folder)
=> the message is removed from the list of messages in your inbox and the number
of messages in the Trash is incremented by one.
5. now reconnect mozilla ("get messages" button, may need to be pressed it a
couple times)
=> the deleted message reappears in your inbox. Checking the Trash folder
doesn't show it, the number of message in the Trash has reverted to the previous
value. 
==> In other words, the message was never deleted.

Also, the fact that one has to press the "Get messages" button twice at step 5
or that we have to select the message (step 2) before checking emails (step 3)
suggests that Mozilla detects a problem with IMAP and does reconnect but then it
doesn't redo the action (select/delete/get messages). If it redoes the action
after reconnecting, this would fix the (delete/get messages) problem. The blank
mail would still be an issue because of the cache.

Comment 4

[David :Bienvenu]

Nahor, part of this problem is the way the UW server handles getting multiple
connections to a folder. Basically, in some configurations, it will simply drop
the first connection, and any state changes for that connection like message
deletion. As far as the client is concerned, it made the changes, and through no
fault of its own, the server dropped the changes. The client can't simply keep
trying to delete the message every time it connects to the server - what if the
user undeleted the message from anther client? I know that a patch was developed
for the UW server that changes this behaviour to flush the state changes for the
first connection before dropping it, and I believe this patch was submitted to
the folks responsible for the UW server, but I don't know what happened after that.

Comment 5

[Nahor]

This is not about Mozilla being disconnected while/after doing an action, it's 
about Mozilla being disconnected *before* an action.
Mozilla doesn't check if the connection still exists before performing an 
action, it doest it and assumes that everything went well.
The problem is not just with UW server but with any IMAP server. I think you 
will get the same problem if you shutdown your IMAP server and restart it 
instead of doing the "check your mail with another IMAP client".

Comment 6

[David :Bienvenu]

Nahor, yes, I understand that, and yes, you're right, in your scenario the
delete never gets to the server. 

Comment 7

[Ere Maijala (slow)]

I've analyzed this a bit, and here's my conclusion (which may be correct or not):

When the connection is killed, nsImapProtocol::OnStopRequest() is called. The
call comes from the imap main loop which is in a different thread, so the call
actually goes to the event queue. Unfortunately nobody processes pending events
before an imap url processing is started, and the event just hangs in the queue
until it's too late (nsImapProtocol::CanHandleUrl() has been called and it has
thought everything's still ok). A very simple fix worked for me, although I'm
not sure if it causes problems somewhere else. In
mailnews/imap/src/nsImapProtocol.cpp, ImapThreadMainLoop(), uncomment line
m_eventQueue->ProcessPendingEvents(). 

Comment 8

[Ere Maijala (slow)]

Attachment: Patch to enable event handling in main loop

This patch causes the main loop to process events in the event queue, enabling
imap code to get notified in time when a connection is killed. This will need
testing on different platforms, but at least it helped for me on win2k.

Comment 9

[David :Bienvenu]

Comment on attachment 72369 [details] [diff] [review]
Patch to enable event handling in main loop

r=bienvenu - this patch makes sense. We should try it out on mac and linux as
well. I was a little worried that we don't know what events might be sitting
around waiting to get processed, and if there's some state they rely on that's
no longer around, there could be problems, but if we wake up every second and
process pending events, then the state can't have rotted too much (e.g., the
account getting deleted or something).

Comment 10

[David :Bienvenu]

an alternative or additional fix is to have some way of checking the validity of
the underlying socket when we call HandleUrl

Comment 11

[Scott MacGregor]

Let's ask Darin if we can poll a socket transport to see if the socket is still
live. I bet we can't but it can't hurt to ask. 

Comment 12

[Ere Maijala (slow)]

Just a quick note to tell that I'm working on a more complete overhaul of the
error handling. I should soon have something to show.

Comment 13

[Ere Maijala (slow)]

Attachment: Better error handling to the url processing

I'm open for comments about this patch. It is to improve the error handling so
that if a connection is killed during url processing, the cache entry would be
doomed and user would be notified.

Comment 14

[Ere Maijala (slow)]

Attachment: Better error handling to the url processing

No changes, just diffed against current cvs.

Comment 15

[David :Bienvenu]

Ere, in general I think this looks good. A few comments:

+        if (NS_FAILED(rv) && GetConnectionStatus() < 0)
+        {
+            // tell user that the connection has died
+            AlertUserEventUsingId(IMAP_SERVER_DISCONNECTED);
+            // doom the cache entry
+            if (m_mockChannel)
+                m_mockChannel->Cancel(rv);
+        }
I'm not sure there's much guarantee that connection status < 0 definitely means
the server disconnected - that just happens to be the case you're testing.

Also, if you do a -uw diff, the whitespace changes won't show up and what you're
really changing will be clearer.

Comment 16

[Nahor]

This big problem is now fixed for me.
I still have a little problem (mostly "cosmetic"): when one run the delete test
(see comment #3), the mail now gets deleted (good) but I also hear the sound as
if a new mail arrived. From the look of it, I assume that what happens is:
  1. the mail gets deleted from Mozilla (disappear from the inbox)
  2. Mozilla tries to delete it on the server but can't because it's disconnected
  3. Mozilla reconnects
  4. as part of the reconnection process, Mozilla checks for new mail. It finds
the mail we want to delete on the server and since it's not in Mozilla (not
anymore), it starts the "new mail" sound and the mail reappears in the inbox.
  5. Mozilla restarts the delete process starting at step 1.

Could the mail be deleted in Mozilla *after* being deleted on the server? That
way Mozilla won't detect a "new mail" since the mail is still there.

Comment 17

[Alexandru Savulov]

Ere:

in bug 139222, you mention that the 1st patch attached here is checked in
already. If yes, could you add a comment on this bug, reslove it and so on please?

Comment 18

[Ere Maijala (slow)]

Well, the first patch is in, but the second one is still waiting.

Comment 19

[steve bowden]

Can anyone confirm that this is the bug that I'm seeing? 
It sounds like it. The symptoms are opening a message (previously unopened) in
IMAP inbox when there is a network problem. Attempts to retieve the message from
the server eventually time out and this result i.e. nothing is cached. Therefore
subsequent attempts to retriveve message when network is restored look at the
cache and display... nothing!  Workaround is to delete the INBOX.msf file (why
is this filename in uc?) and restart.
I'm running build 2002052404 on win98. Thanks

Comment 20

[Ere Maijala (slow)]

If you need to delete the msf file, it's not the same thing (I'd suspect index
corruption or something like that). If a message fetch fails, emptying memory
cache should be enough. If that helps, then you're seeing the same thing.

Comment 21

[steve bowden]

Emptying the memory cache does fix it. - Thanks. So it seems to be this problem.
I have had problems in the past where restarting moz didn't help - so that must
have been msf corruption but don't remember the exact circumstances.

Comment 22

[Ere Maijala (slow)]

Once upon a time I had a patch for this. It was never actually reviewed and has
rotten by now. I think I'll try to renew it.

Comment 23

[steve bowden]

That's a good idea. I think this needs to be fixed - imho it's not a minor bug.
Hopefully it will get reviewed this time!

Comment 24

[Ere Maijala (slow)]

Attachment: Improved error handling for situations where server closes connection

This is the latest version of the improved error handling for situations where
the server closes the connection or it is otherwise gracefully ended when
Mozilla doesn't expect it. Also handles a net reset. Comments, suggestions,
reviews please :)

Comment 25

[David :Bienvenu]

Comment on attachment 85263 [details] [diff] [review]
Improved error handling for situations where server closes connection

r=bienvenu - I need to try running with this first, however.

Comment 26

[Ere Maijala (slow)]

I've run with it for a few days and had no problems. 

Bienvenu, have you tested it?

Comment 27

[David :Bienvenu]

I have not had a chance to try this - I've been doing other stuff. I'll try it
out soon, though I think it's OK to check it into post 1.1 alpha.

Comment 28

[Brendan Eich [:brendan]]

Comment on attachment 85263 [details] [diff] [review]
Improved error handling for situations where server closes connection

sr=[e-mail], looks fine code-wise, but it'd be cool to have bienvenu
come back with those test results.

/be

Comment 29

[Ere Maijala (slow)]

Fix checked in trunk.