261668 - POP3, "Fetch headers only": Mails disappear from inbox

Status: VERIFIED FIXED

(Thunderbird :: Mail Window Front End, defect)

Description

[Michael Wenger]

User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows ME) Opera 7.53  [de]
Build Identifier: TB version 0.8 (20040925)

Hi there!
For some reason, mails disappear from inbox when using a POP3-Account with 
"Fetch headers only" enabled.
'Disappear' means that the mail is not displayed anymore although it is 
physically available. You can find it using 'Search messages...'.
I have taken a screenshot from such a constellation:
http://wenger.home.t-link.de/net/Mail-disappeared.png

After applying 'Compact This Folder' to inbox, the mail suddenly appears again.

I took two sets of snapshots of the files inbox and inbox.msf.
The first set in the folder 'before compact' contains the files, when the test 
mail was not displayed.
Then I applied 'Compact This Folder' to inbox. After that TB displayed all 
messages correctly. The files representing this state are located in the folder 
'after compact'.

Reproduced on WinXP and WinME using TB version 0.8 (20040925), downloaded from 
here:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-0.8/
thunderbird-win32.zip

With kind regards,
Michael and Torsten (who tested a lot)


Reproducible: Always
Steps to Reproduce:
1. create new POP3-Account and activate "Fetch headers only"
2. send a mail to this account and fetch it completely
3. step into another account or mail-folder
4. return to the inbox of the test-account
5. now, the test mail should disappear from the inbox
Actual Results:  
The test mail appeared from inbox, but I could find it using 'Search messages...
'.

Expected Results:  
It should have displayed the message.

Comment 1

[Michael Wenger]

Attachment: snapshots of inbox and inbox.msf

Comment 2

[Michael Wenger]

It seems to be very difficult to reproduce the bug using an existing profile.
So, the steps to reproduce should be:

Steps to Reproduce:
0. create an new profile
1. create new POP3-Account and activate "Fetch headers only"
2. send a mail to this account (from another account) and fetch it COMPLETELY 
(which means: first fetch just the header and then fetch the complete mail using 
the link in the header-only mail)
3. step into another account or mail-folder
4. return to the inbox of the test-account
5. now, the test mail should disappear from the inbox
Actual Results:  
The test mail disappeared from inbox, but I could find it using 'Search 
messages...'.

Expected Results:  
TB should have displayed the message.

Reproduced on TB version 0.8 (20041015).

Regards,
Michael

Comment 3

[Schorsch]

Same Problem here.

Hope this but would be CONFIRMED and coul'd be fixed. 

Many THX

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

*** This bug has been marked as a duplicate of 265553 ***

Comment 5

[Howard Chu]

(In reply to comment #2)
> It seems to be very difficult to reproduce the bug using an existing profile.
> So, the steps to reproduce should be:
> 
> Steps to Reproduce:
> 0. create an new profile
> 1. create new POP3-Account and activate "Fetch headers only"
> 2. send a mail to this account (from another account) and fetch it COMPLETELY 
> (which means: first fetch just the header and then fetch the complete mail using 
> the link in the header-only mail)
> 3. step into another account or mail-folder
> 4. return to the inbox of the test-account
> 5. now, the test mail should disappear from the inbox
> Actual Results:  
> The test mail disappeared from inbox, but I could find it using 'Search 
> messages...'.

I reproduced this bug using a new POP account (default profile) as described
above, using my current (somewhat old) build of Mozilla 20041002. However, if I
fetch the mail using the Get Selected Messages option, the bug does not occur,
so it appears to be specific to the processing of the POP URL in the stub. I'll
update my source to today's version and see if it still occurs. I at least
should be able to identify the area of code that's causing this.

I'm commenting here because I don't believe anyone is sure what 265553 is really
about. The behavior reported by Henrik in 265553 is "works as designed" but
there hasn't been any clear method to reproduce the problem reported by the
original poster.

Comment 6

[Howard Chu]

Attachment: Patch for nsPop3Sink.cpp

The problem is caused by the FindPartialMessages/CheckPartialMessages
functions. Apparently nsPop3Protocol removes the current message from it's
UIDL, and so it gets deleted by CheckPartialMessages. This patch fixes it by
bypassing the FindPartialMessages check. (As I noted before, this bug only
affects the specific-UIDL fetch operation, not the DownloadOffline operation.)

I don't believe this patch is the correct fix, but I'm posting it here just as
a starting point.

Probably the correct fix is not to run FindPartialMessages at the beginning of
a download. Instead we should combine that step with the current
CheckPartialMessages step, so it all happens after the download completes. Then
we won't inadvertently step on partial messages that were actually fully
retrieved in the current download.

Comment 7

[Howard Chu]

Attachment: better fix

Ok, the real problem is that there was redundant code in nsLocalMailFolder.cpp
that also deleted the header for a specific-UIDL fetch. This patch consolidates
the FindPartialMessages/CheckPartialMessages functions as described above, and
eliminates the redundant delete from nsLocalMailFolder.

Strange that this bug was so difficult to reproduce; it should have tripped
every time you used the "Click Here" link. I always use the "Get Selected
Messages" command so I didn't run into it before...

Comment 8

[Howard Chu]

By the way, I don't believe this bug is at all related to 265553, since this one
is specific to Headers-Only and 265553 is not.

Comment 9

[David :Bienvenu]

Comment on attachment 164170 [details] [diff] [review]
better fix

I'll try this patch right away - we'd want to get it into tbird .9

Comment 10

[David :Bienvenu]

Comment on attachment 164165 [details] [diff] [review]
Patch for nsPop3Sink.cpp

this one worked fine for me.

Comment 11

[David :Bienvenu]

Comment on attachment 164170 [details] [diff] [review]
better fix

this patch didn't work for me - it worked for Howard, but not me - trying to
figure out why, but the earlier patch works fine...

Comment 12

[David :Bienvenu]

re-opening to mark fixed.

Comment 13

[David :Bienvenu]

fixed on branch and trunk.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It's not finally fixed. We have to create a small fix for updating the
messagepane. When the user downloads a not existing partial message, it will be
removed. The threadpane is updated correctly but we miss to do it for the
messagepane. The content is still visible and you can try to download it again.
I also get a js exception when doing this:

Error: [Exception... "Component returned failure code: 0x80004003
(NS_ERROR_INVALID_POINTER) [nsIMessenger.messageServiceFromURI]"  nsresult:
"0x80004003 (NS_ERROR_INVALID_POINTER)"  location: "JS frame ::
chrome://messenger/content/msgMail3PaneWindow.js :: SelectMessage :: line 1604"
 data: no]
Source File: chrome://messenger/content/msgMail3PaneWindow.js
Line: 1604

Comment 15

[herbert kauer]

My starting point is that TB has downloaded only the headers on a mail.

1) In comment #14 (for me) a situation is described, in which TB tries
   to download a mail and TB can connect to the mailserver and is 100% sure
   that the mail does not exist anymore.
   In this case, if the mail is removed from threadpane, also the message-
   pane has to be updated accordingly.

   Another possible solution for me would be to keep the mail in threadpane
   and only to modify the mail contents with a new text:
   "mail deleted on server before content downloaded". 

2) In contradiction to 1) I had a situation in which TB tried to download
   a mail, but it could not connect to the mailserver.
   TB 0.9 removed the mail from threadpane, but messagepane kept unchanged.
   (as described in comment #14).
   But now the mail has to be kept unchanged both in threadpane and
   also messagepane in order to try the downlaod again.

Comment 16

[Christian Ege]

David, Howard,

although I cannot reproduce this bug anymore since the patch was implemeted,
there are some evidences, that it still exists. In the German TB Forum we get
ongoing complaints about lost emails. Nearly all of them seem to be related to
the fetch headers only option while the emails can be 'restored' by compressing
the folder. Unfortunately there is still no known procedure to reproduce it.

However, we are wondering if this could be because the patch was based on
Howard's first approach but not on the better one, he provided a bit later. As
Howard mentioned in comment #7 the real root cause was not caught by the first
fix. What do do think about it?



 

Comment 17

[Hans Bauer]

Hi all,

i got a problem downloading emails with large attachments using tb v1.0RC1. The
download seems to occur but no new mail is shown. This happens as well when
trying to download the whole mail as well as first downloading the header and
next the content. Compressing the incoming folder does not help. 

I reported this bug at "https://bugzilla.mozilla.org/show_bug.cgi?id=273369".
Maybe the bugs are related.

Thanks and hope this helps
Hans Bauer

Comment 18

[Pelle Goeg]

This proble still exist:
version 1.0 (20041206)

Just lost 300 messages!

Comment 19

[Howard Chu]

(In reply to comment #15)
> My starting point is that TB has downloaded only the headers on a mail.
> 
> 1) In comment #14 (for me) a situation is described, in which TB tries
>    to download a mail and TB can connect to the mailserver and is 100% sure
>    that the mail does not exist anymore.
>    In this case, if the mail is removed from threadpane, also the message-
>    pane has to be updated accordingly.

I agree, the messagepane needs to be fixed like this.
 
>    Another possible solution for me would be to keep the mail in threadpane
>    and only to modify the mail contents with a new text:
>    "mail deleted on server before content downloaded". 

This approach would annoy me, it would leave a lot of useless stub messages in
my mailbox. I definitely would not go with this solution.
> 
> 2) In contradiction to 1) I had a situation in which TB tried to download
>    a mail, but it could not connect to the mailserver.
>    TB 0.9 removed the mail from threadpane, but messagepane kept unchanged.
>    (as described in comment #14).
>    But now the mail has to be kept unchanged both in threadpane and
>    also messagepane in order to try the downlaod again.

I have seen this problem a number of times, and I've found the problem - the
popstate gets updated even on an aborted connection. If the connection aborted
before all messageIDs/UIDs could be received, then the popstate that is written
out will be incomplete and missing a lot of messages. After this comes the bit
of code that deletes message headers if their corresponding ID is missing from
the popstate. That runs, deleting a bunch of headers that really should have
been left alone. This problem bit me a dozens of times while I was traveling and
using unreliable dialup connections. I commented out the popstate update call
for the abort, but that's a bit of an extreme fix. It would be better to skip
the update only when we know that the incoming message list is incomplete. If we
got past the phase of reading the message list, and are into the phase of
actually retrieving messages, then it would be safe to update the popstate.

Comment 20

[Howard Chu]

Attachment: Fix header deletion due to aborted connection

There didn't seem to be an existing state variable that I could use to reliably
determine that we'd entered POP3_GET_MSG state, so I added the list_done flag.

Comment 21

[Howard Chu]

Attachment: Fix message pane update for deleted header

This addresses problem 1 in comment #14

Comment 22

[Howard Chu]

Just noticed that I never replied to this. The second patch had some other
problems with the database, and the one that got checked in worked best.

(In reply to comment #16)
> David, Howard,
> 
> although I cannot reproduce this bug anymore since the patch was implemeted,
> there are some evidences, that it still exists. In the German TB Forum we get
> ongoing complaints about lost emails. Nearly all of them seem to be related to
> the fetch headers only option while the emails can be 'restored' by compressing
> the folder. Unfortunately there is still no known procedure to reproduce it.
> 
> However, we are wondering if this could be because the patch was based on
> Howard's first approach but not on the better one, he provided a bit later. As
> Howard mentioned in comment #7 the real root cause was not caught by the first
> fix. What do do think about it?

Comment 23

[bugzilla]

The bug is still there in the 1.0 build of mozilla. i've found non jonk marked 
as junk in the junk folder. klicking on fetch message the message disappears. 
everytime i click again, another message from the junk folder disappears. there 
is *no* way to recover them, at least via the ui. compact folers or search 
won't help 

Comment 24

[David :Bienvenu]

Comment on attachment 173107 [details] [diff] [review]
Fix message pane update for deleted header

if you add a method to an interface, you need to change the interface-id, using
one from uuidgen - here's one: ca3be30c-b850-4fec-bdd9-131c39e74805

Other than that, this looks OK.

Comment 25

[Howard Chu]

Attachment: Same as 173107, with new UUID

OK, inserted the new UUID from #24

Comment 26

[David :Bienvenu]

Comment on attachment 177885 [details] [diff] [review]
Same as 173107, with new UUID

thx, Howard.

Comment 27

[chris hofmann]

Comment on attachment 172921 [details] [diff] [review]
Fix header deletion due to aborted connection

a=chofmann

Comment 28

[chris hofmann]

Comment on attachment 177885 [details] [diff] [review]
Same as 173107, with new UUID

a=chofmann