Closed Bug 405440 Opened 17 years ago Closed 16 years ago

IMAP cache broken if the message download is not finished due to user interaction (offline-use case, even after fix of 386514)

Categories

(Thunderbird :: General, defect)

defect
Not set
major

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: patrick.chaleat, Assigned: Bienvenu)

References

Details

(Keywords: verified1.8.1.12)

Attachments

(1 file)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9
Build Identifier: 2.0.0.9 (20071031)

This bug can be reproduce more easily on slow network and with desactivation of mime parts on demand, either with mail.imap.mime_parts_on_demand=false, either with a big value for mail.imap.mime_parts_on_demand_threshold

Reproducible: Always

Steps to Reproduce:
On a folder with messages available in offline mode.
With message pane, clic on a new and big message. You can see the download with the progress bar . Before the end, clic on an other message. Then clic to the precedent message. You can't see it.
And what you do, you will never see it.
Actual Results:  
The message with donwload not finished will never be re-downloaded

Expected Results:  
The message with donwload not finished will be re-downloaded.

The only way to get the mail is to move it to another imap folder, Rebuild the index is a solution also, but with re-download of all the messages in the folder
Sounds DUP of Bug 384819
If DUP, please recommend better bug summary for bug search by troubled people.

> Steps to Reproduce:
> On a folder with messages available in offline mode.

"Offline-Use" folder only problem?
No problem if non-offline-use folder, when download failure?

When mail.imap.mime_parts_on_demand=true, "partial then corrupted"(due to error while download) usually occurs only on attachment.
Does problem of "partial then corrupted" occur on whole mail, if mail.imap.mime_parts_on_demand=false?
> "Offline-Use" folder only problem?
> No problem if non-offline-use folder, when download failure?

No, we can't reproduce problem on test with non-offline folder. But in production, we always use offline-use folder.

> When mail.imap.mime_parts_on_demand=true, "partial then corrupted"
> (due to error while download) usually occurs only on attachment.

We found this bug because we had an other bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=405437
An this bug only occurs with attachment.

For this bug, it seems also to only occurs with attachment.
But with our setting, when we have the problem, we have a "blank" message and when we clic on the message corrupted, Thunderbird continue to display the previous displayed message.

But i make again myself some test today with the 2.0.0.9, it seems to be more difficult than with previous version to have message not present at all. The problem was found with version 1.5 and we test the first version of the 2.0.0.0 -> 2.0.0.4
We are going to re-check with this older versions...

However, i manage to break the cache going offline (without downloading the messages) during download one message. The message is not blank in the case, the attachment is incomplete. But it's not a big problem, it can't happen often.
I have made some more tests with several versions.
The bug is easy to reproduce in 2.0.0.0 and 2.0.0.5
It's more difficult in 2.0.0.5 and 2.0.0.9
Perhaps the correction of the bug
https://bugzilla.mozilla.org/show_bug.cgi?id=386514
make a difference.
But, it's still easy to get the bug in our production settings of Thunderbird ...
We investigate again ...
(In reply to comment #3)
> It's more difficult in 2.0.0.5 and 2.0.0.9
> Perhaps the correction of the bug 386514 make a difference.

Sound same problem as Bug 386514 (verified1.8.1.5 means resolved by 2.0.0.5).
(Bug 384819 I mentioned in Comment #1 seems to be DUP of Bug 386514) 

> But, it's still easy to get the bug in our production settings of Thunderbird

Which Tb version is used in production system?
If version before Tb 2.0.0.5(i.e. Bug 386514 is not resolved yet), it's reasonable, because usual user, including me, usually does do operation like Bug 386514 or Bug 384819 frequently.
> The bug is easy to reproduce in 2.0.0.0 and 2.0.0.5
> It's more difficult in 2.0.0.5 and 2.0.0.9

I wanted to say:
The bug is easy to reproduce in 2.0.0.0 and 2.0.0.4
It's more difficult in 2.0.0.5 and 2.0.0.9

> Sound same problem as Bug 386514 (verified1.8.1.5 means resolved by 2.0.0.5).
> (Bug 384819 I mentioned in Comment #1 seems to be DUP of Bug 386514) 

Yes, the description of the Bug 384819 is very very similar to this one.


> Which Tb version is used in production system?
Our official production client is still an old version of 1.5 (30000 clients).

With a fresh install of TB 2.0.0.5 and 2.0.0.6 on a default profile with mime part on demand off, i can't reproduce the problem. This is good.

But, in a production configuration (set by an extension by reading the LDAP) on a TB 2.0.0.5 and 2.0.0.6, it is possible to reproduce the problem. We have to see why. The correction can be ineffective in our configuration or perhaps there is another problem with our choice of configuration.

I will do more tests...
(In reply to comment #5)
> With a fresh install of TB 2.0.0.5 and 2.0.0.6 on a default profile with mime
> part on demand off, i can't reproduce the problem. This is good.
> But, in a production configuration (set by an extension by reading the LDAP) on
> a TB 2.0.0.5 and 2.0.0.6, it is possible to reproduce the problem.

Remaining problem is possibly Bug 92111(MS Exchange case) / Bug 390795(Non MS Exchange case).
Get IMAP protocol log(NSPR log), and check real protocol level flow.
See Bug 402793 Comment #1 for getting NSPR log. 
1 - way to reproduce
create imap account
create (or use) a test folder
mark it for offline use
put 2 messages : 1 big (2Mo attachment) - 1 short
add pref:
mail.imap.mime_parts_on_demand_threshold : 5000000
Show message pane

2 - identify the problem
Open TB (2.0.0.9)
Select test folder
Reindex test folder and compact it
Select big message
Interrupt downloading by selecting the short message
The short message is displayed
Selecting again the big message won't downloading it (or restart), message pane still display the short one.

3 - code analysis
When downloading is finish, message is marked for offline use (with this configuration).
Mark is done in 'nsMsgDBFolder::EndNewOfflineMessage' method by calling 'mDatabase->MarkOffline' 
(mailnews\base\util\nsMsgDBFolder.cpp).
But in our tests, 'MarkOffline' call refer to the big message when downloading the short one has finished!
(m_offlineHeader refer to the big message).

4 - applying a code correction
whe simply add these line:
m_offlineHeader=nsnull;
at the begining of the method 'nsImapMailFolder::EndOfflineDownload' ('mailnews\imap\src\nsImapMailFolder.cpp')
which seems to be called for the big message before the load of the short message start.

5 - testing
Whe can't reproduce the problem with this modification to the code.
Interesting - I wonder why AbortMsgWriteStream isn't getting called. This looks like a good fix, though. I'll try to drive it in.
This is the fix as proposed by Patrick, more or less. I'll go ahead and land it when I get a chance.

It looks very simple, but there are a few things to be aware of:

This method gets called whenever any imap url finishes running, not just urls that are storing a message for offline use.  So if you somehow had two urls running against the same imap folder, this patch could cause problems. We don't allow two urls to run against the same folder at the same time, so this should be OK.

I was not able in my tests to recreate this problem - AbortMsgWriteStream was called every time, but it's possible there's some situation where it isn't, but m_offlineHeader is still created. So we'll give this a try.
Assignee: nobody → bienvenu
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attachment #297549 - Flags: superreview+
Attachment #297549 - Flags: review+
OS: Windows XP → All
Hardware: PC → All
fixed on the trunk, thx, Patrick.
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
*(Version) unspecified -> 2.0, per comments so far.

Patrick/Philippe, could you try to reproduce with recent Trunk nightlies ?
Possibly, one before the patch and one after...
Version: unspecified → 2.0
Do we want this for the branch too?
I'm sure that Patrick would love to see a branch-based build w/ this fix, whenever that happens.
Yes, we need this patch !
files:
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2008-01-23-04-mozilla1.8/thunderbird-2.0.0.12pre.en-US.win32.installer.exe
.../2008-01-22-04-mozilla1.8/...
.../2008-01-20-04-mozilla1.8/...
has been tested.
The problem has been reproduced.
Comment on attachment 297549 [details] [diff] [review]
patch as proposed by Patrick

the way to get the patch in is to request approval for the next 1.8.1 release (the code freeze for 1.8.1.12 is coming up soon, so it might not make it into that one  but that's up to the release drivers)
Attachment #297549 - Flags: approval1.8.1.12?
Comment on attachment 297549 [details] [diff] [review]
patch as proposed by Patrick

approved for 1.8.1.12, a=dveditz for release-drivers
Attachment #297549 - Flags: approval1.8.1.12? → approval1.8.1.12+
fixed for 1.8.1.12, thx, Patrick.
Keywords: fixed1.8.1.12
Patrick, can you verify this fix using the RC build for Thunderbird 2.0.0.12? It is up at ftp://ftp.mozilla.org/pub/thunderbird/nightly/2.0.0.12-candidates/rc1/. 
We have already check the rc1. It is ok.
We are now waiting for the final release.
Thank you for all
The RC and the final release are the same bits. We only built the one RC and are shipping it today. I'm going to mark this as verified then.
Status: RESOLVED → VERIFIED
Adding some words in bug summary, for ease of distinction between Bug 386514 and this bug, and for ease of search.
Summary: IMAP cache broken if the message download is not finished due to user interaction → IMAP cache broken if the message download is not finished due to user interaction (offline-use case, even after fix of 386514)
I don't see how this is fixed.  In version 2.0.0.21, if the email is viewed before it's finished downloading from IMAP and stored to an offline folder, the attachment is broken and will NOT re-download.  It may be localized to Windows only as I can't re-produce this on Mac release.
(In reply to comment #25)

See DUP'ed Bug 373821 & Bug 417470. Problem reporters in these bugs say "problem don't occur after fix of this bug". See also "Status: VERIFIED FIXED" and "Keywords: verified1.8.1.12" of this bug.

Really same problem as this bug's original problem? Please note that "same external symptom" doesn't always mean "completely same problem (same condition, same cause, same required solution)". Kenny Lerma, if you can recreate your problem, get IMAP log and check real IMAP level flow. And if new problem with similar symptom or regression of this bug, open separate bug, with referring to this bug, please.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: