72928 - UW IMAP: Msg lost during move

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Ben Bucksch (:BenB)]

I just lost a mail during moving it per drag&drop from my INBOX to another folder.

It isn't in my msg store anymore - neitehr the target folder nor the source
folder nor any other folder. (I confirmed that by searching directly on the server.)

Before, the target folder was in that weird state, where deleting a (duplicate)
msg ended up in an empty entry in the folder pane, e.g. "Re: ".

I use UW-IMAP. All folders lie there. Mozilla Linux build from 2001-03-18.

Comment 1

[scottputterman]

possibly related to 72926?

Comment 2

[Ben Bucksch (:BenB)]

My target folder was small (~15 msgs). The source folder has ~500 msgs. All msgs
did have a subject. Just that Mailnews failed to recognize it in one
circumstance (but not in the case of the to-be-moved msg).
Quite likely that the root cause is the same, though.

Comment 3

[marina]

*** This bug has been marked as a duplicate of 72926 ***

Comment 4

[Ben Bucksch (:BenB)]

marina, do you have any evidence, that this is actually a dup??

Comment 5

[Ben Bucksch (:BenB)]

marina, see above.

Comment 6

[Ben Bucksch (:BenB)]

marina said on the other bug:
> and yes, #72928 is a dup of the one i filed ( same problem: moving to a big
> folder)

No, I did *not* move the msg to a big folder. See above.

Comment 7

[marina]

Ben, i am using 3-21 build and my source folder and target folder are both big
(~ 3000 messages), i have to add though that i am losing messages that has no
subject (no subject) but i am able to move them to a small folder 10-15
messages. Maybe it is better to keep both open just in case that the root cause
is the same but it looks like there is one condition that causes the loss
manifesting in different flavors. Feel free to reopen. 

Comment 8

[Ben Bucksch (:BenB)]

Yes, I prefer to keep bugs open until we are very sure. REOPENing. We can still
remark as dup later.

Comment 9

[Ben Bucksch (:BenB)]

As for the relation to bug 72926:

> Before, the target folder was in that weird state, where deleting a
> (duplicate) msg ended up in an empty entry in the folder pane, e.g. "Re: ".

Note: The msg
- did have a subject, it just was not shown after the delete operation on
another msg with the same subject.
- The subject *shown* was "" (empty) or "Re: ".
Other msgs had their subject still shown correctly.

Also, that was just a few msgs in the target folder. It was *not* the msg I
tried to move and then lost.

Comment 10

[Ben Bucksch (:BenB)]

It seems that bug 72926 is *not* the same as this one. In addition to the
reasons above, this bug lost the msg *completely* - as said in the initial
description, it's not in the file that stores the folder. In the other bug, the
msg is just not *shown*. So, this bug is much worse. Thus, it would be nice, if
you gave it a bit more attention, though I cannot assist with further testing
myself.

Comment 11

[Ben Bucksch (:BenB)]

This is IMO a MUSTFIX for any release.

Comment 12

[Scott MacGregor]

I'm unable to reproduce this.

If you delete your .msf files does the file suddenly show up? I guess if it's
not in the server store anymore then it isn't going to come back by deleting the
.msf files

Can you attach an imap log showing the drag and drop happening?

It would get more attention if we saw it in house but you seem to be the only
one seeing this right now...=(

Comment 13

[Ben Bucksch (:BenB)]

> I guess if it's not in the server store anymore then it isn't going to come
> back by deleting the .msf files

Very true.

> Can you attach an imap log showing the drag and drop happening?

No.
- I didn't log the session where it happened (because I didn't know it would
  happen)
- I cannot afford to lose real-world mail
- I have no time to perform dedicated sand-box testing
I see your problem, but I cannot help you. I'm sorry. That's something your QA
(or any other volunteer) will have to do.

> It would get more attention if we saw it in house

Did you test UW-IMAP? Not sure, if that is a factor, though.

> you seem to be the only one seeing this right now...=(

This will probably change once you release the software to a wider audience :-(.
Since losing mails is IMO a "must never ever do", I think this is a show-stopper
for a release (like Mozilla 0.9, which is even supposed to be a base for beta
products).

Comment 14

[Ben Bucksch (:BenB)]

QA:
> Before, the target folder was in that weird state, where deleting a (duplicate)
> msg ended up in an empty entry in the folder pane, e.g. "Re: ".

The dup msgs came from fccs, bcc and mailingslists, IIRC, i.e. I posted to a
list that I am subscribed to myself and activated both fcc and bcc to an
internal account of mine. Then, I moved all copies the target folder. Same for
initial posts and follow-ups (Re:s).

Later, I went to the folder. All subjects showed up fine. It were normal
subjects like "Progress bar" and "Re: Progress bar" (both msgs with 3 copies, as
described above). I deleted the dups.

Now, the first odd bahaviour appeared: The msgs got deleted, IIRC, but the
subjects of the remaining msgs were empty (in the thread pane at least), i.e. ""
and "Re: " respectively showed up. No data loss so far, IIRC.

I ignored the misbehaviour and went on to the INBOX, moving msgs from the INBOx
to the target folder (same as above). I went to the target folder to delete dups
and noticed that some of the moved msgs did not arrive at their target. I
checked and saw that they were gone complete - not in INBOX, not in the target
folder, in fact no where in the msg store at all.

OK, I think, that's all I know about the problem. If you need details about my
systems or profiles, ask. I won't attach the full prefs.js, but I can ask
specific question, and you can check
<http://bugzilla.mozilla.org/showattachment.cgi?attach_id=27883> ("server1"
resembles the account where the problem accoured).

Note that this bug makes my use of newer versions of Mailnews practically
impossible.

Comment 15

[scottputterman]

Esther or sheela, could you see if you can reproduce this?

Comment 16

[Sheela Ravindran]

Scott,
Karen is more familiar with the UW server and she will be back on Tuesday. We
can have her take a look at it. But if it is urgent, then either Esther or
myself will have try to figure this out. Let us know.

Comment 17

[scottputterman]

It can wait until next week. I'd just like to see if we can reproduce this. 
This may not only be happening on UW.  Should we make Karen the QA assigned then
so that she can query on this?

Comment 18

[Sheela Ravindran]

spoke to karen ->changing qa contact to get some attention on this bug.

Comment 19

[Karen Huang]

Sorry for replying this bug late since I just back to office recently....
I tried on 05-04-06-trunk build on UW IMAP and couldn't reproduce this problem.
bug 72926 fixed already, I am wondering to know whether it matter or not?
Ben, can you try again for the latest build? Also, can you attach that message 
here if you still have that?

Comment 20

[Karen Huang]

Above build is today's window build, but even I tried on Linux 05-02-08-trunk 
build...I still cannot reproduce this problem. Ben, can you try on Linux for the 
same build that I have tested? This probably been fixed on bug 72926....

Comment 21

[Ben Bucksch (:BenB)]

> This probably been fixed on bug 72926....

No. See above.

> Ben, can you try again for the latest build?

I'm sorry, no, I can't. See above for reasons.

> Also, can you attach that message here if you still have that?

The lost msg I don't have anymore, of course. I don't remember which msg or
folder produced that weird display in the thread pane.

Comment 22

[Michael Semich]

I have been using the IMAP in Mozilla with a WUIMAP2000 server and have been 
getting this same problem.  It seems that I can move or copy a group of 
messages and only about 90% of them actually move from my local folders to the 
other side.  This is reproducable across two different nightly builds of the 
Mozilla (one is .9.4 (Win32) stable and the other is the 12/31/2001 (win32)
build).  I can compare the two folders and find the ones that were not copied, 
but if I move the messages then the messages that do not get copied end up in 
the bit bucket.  I am using the standard options in IMAP2000 and have not 
applied any patches to the server.  This behaviour is not apparent in local-
>POP transfers.  Folder size does not matter and there appears to be no pattern 
by which messages are dropped.

Comment 23

[Michael Semich]

This is a forwared message by a general user who contacted me independent of 
the Mozilla project.  It has bearing on this bug, I believe.  Check it out:

I have been testing the UW IMAP server (vs. POP3 that I had been using 
and I believe that I hit the same bug (#72928).  Mozilla would lose 
about 25% of the messages whenever I moved (or copied) a block of 
messages to an IMAP server folder.

I noticed a burst of warnings in the syslog from the IMAP server when 
this happened:
   Jan  4 20:16:39 fargo imapd[19707]: port 59507 service init from 
127.0.0.1
   Jan  4 20:16:39 fargo imapd[19707]: Authenticated user=hargen 
host=localhost [127.0.0.1]
   Jan  4 20:16:39 fargo imapd[19705]: Killed (lost mailbox lock) 
user=hargen host=localhost [127.0.0.1]
   Jan  4 20:16:41 fargo imapd[19709]: port 59509 service init from 
127.0.0.1
   Jan  4 20:16:41 fargo imapd[19709]: Authenticated user=hargen 
host=localhost [127.0.0.1]
   Jan  4 20:16:41 fargo imapd[19707]: Killed (lost mailbox lock) 
user=hargen host=localhost [127.0.0.1]

I believe that what is happening is that Mozilla is opening a new IMAP 
connection for each message to move, and each new connection kills the 
previous one in the course of removing a "stale" lock.  (IMAP refers to 
this as the "kiss of death" protocol.  It ought to work quite well to 
deal with the case of trying to read mail from home after leaving a 
client running at the office.  But multiple concurrent active sessions 
won't work!)

So it looks like there are perhaps two Mozilla bugs:
 1) It should never open two (or more) read/write connections to the 
same mailbox!
 2) It should abort a move operation if its connection is terminated 
before completion.

I'm not involved with Mozilla development - just a potential new IMAP 
user.  So if this analysis sounds OK, could you pass this on the the 
appropriate folks?

Thanks.

--Bill

P.S. I'm using Mozilla 0.9.4 that came with SuSE 7.3.


Notice that he is experiencing this bug with Linux and I am having trouble in 
Win32, so one can (?) assume that the bug is not in the platform code but is 
actually in the main branch.  Seems like MozillaMail is just forgetting that 
the handle it has on IMAP is fresh and is trying to recycle it every time.  
Wouldn't that bash a server with a lot of clients?  

Comment 24

[scottputterman]

Adding David and Navin.

Comment 25

[David :Bienvenu]

I suspect this bug has morphed a bit (unless Ben left out many salient details
in his report). Michael Semich's comments make a lot of sense, but only apply
when move/copying multiple messages from local mail or another imap server to
the UW server, and not when move/copying messages from one folder on the same UW
server to another folder on the same server, or move/copying a single message.
So my comments only apply to what Michael has discovered.

For the reasons Michael describes, we go to great lengths to avoid opening
multiple simultaneous connections to any imap folder (not just UW folders), and
we do try to re-use connections. Generating a client-side imap protocol log
would be very helpful here.
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap - if you
could generate a log, edit out any sensitive info, and post it to this bug, that
would be very helpful. Thanks.

Comment 26

[David :Bienvenu]

we're definitely not using the already active connection for the second copy,
and we're issuing a select after each append, which we shouldn't be doing. I'll
investigate.

Comment 27

[David :Bienvenu]

We're not making multiple connections to the same folder. We're selecting the
folder, then, later, we're doing an append on the folder; waiting for the append
to finish, then, using the same connection, selecting the folder. This causes
the UW server to complain that the folder is open by another process, which of
course, it's not - it's open with the same process and already selected. The
problem is that we forgot the folder was in the selected state when we did the
append (and we should be doing the second select at all, but that's a different
problem).

Comment 28

[Michael Semich]

Attachment: IMAP dataloss testcase log

Don't spam me :)

Comment 29

[Michael Semich]

OK, I made a test bed to lay the logs out on and produced some output.  I 
created a local folder and called it test_source, filled it with short 
(identical) messages, and moved them en masse to the IMAP server folder 
test/test_target.  Of the 50 originals, only 37 made it across.  The log is 
about 350k, so it is included as an attachment.  I don't know much about the 
nuts and bolts of IMAP, but there is some strange output in the log.  
Incidentally, there is no way to see the status of the IMAP transfer, so I 
watched my firewall to see when mozilla stopped transferring (is there a bug 
out on that? :) 

Mike

Comment 30

[David :Bienvenu]

Attachment: proposed fix

this patch makes it so we don't issue a select on the destination folder if
it's currently selected, after we upload a msg to the folder. 4.x had the same
check, but it was lost at some point in 6.x

Comment 31

[David :Bienvenu]

Cavin, can I get a review for this? thx.

Comment 32

[David :Bienvenu]

Comment on attachment 63935 [details] [diff] [review]
proposed fix

better fix coming

Comment 33

[David :Bienvenu]

Attachment: better fix

This is a better fix - we need to try to use a connection that's already in the
selected state for this folder, in case we want to select it. And, we don't
want to select the folder unless we actually need to (which is only in the save
draft/template case).

Comment 34

[Cavin Song]

Comment on attachment 63971 [details] [diff] [review]
better fix

r=cavin.

Comment 35

[Scott MacGregor]

Comment on attachment 63971 [details] [diff] [review]
better fix

sr=mscott

Comment 36

[Ben Bucksch (:BenB)]

*** Bug 107233 has been marked as a duplicate of this bug. ***

Comment 37

[David :Bienvenu]

fix checked in - append url should run with connection that's in the selected
state for that folder, if any, and we'll no longer issue a select on every
append (for imap servers without uidplus).

Comment 38

[Gregg Meehan]

Verified on Linux and Win2K, looks good