Closed Bug 506877 Opened 15 years ago Closed 15 years ago

incomplete email download (partial emails and file attachs)

Categories

(Thunderbird :: Message Reader UI, defect)

Product:
Thunderbird
Component:
Message Reader UI
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 468595

People

(Reporter: dbucher, Unassigned)

References

Details

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; fr-FR; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 GTB5 (.NET CLR 3.5.30729)
Build Identifier: version 2.0.0.22 (20090605)

This bug happens for a long time in my Mozilla :
a lot of emails are downloaded partially from IMAP server.

The result is that most file attachs are corrupt because they are uncomplete/partial. Even worse, you never know if there are more fileattachs that you don't see.

When forwarding the mail to someone else, it forwards the broken file attach.
But when MOVING the mail on the mail server into another folder, the message could be valid or not depending on the time Thurnderbird waited to download the message.

But the message is ALWAYS valid in webmail / on the server.

On my opinion it is a timing problem :
When Thunderbird displays a message BEFORE it has finished downloading it, it stops the download, and therefore corrupts the message...

The corruption has always been "partial download", nothing else. Like the download was "cut in the middle".

On the same server, other colleagues have no problem.

Reproducible: Sometimes

Steps to Reproduce:
1. When receiving a big email, I do not wait it to arrive but :
2. I immediately click on it while Thunderbird is still downloading
3. Normally (not always) the email is then corrupt.
Actual Results:  
Email is incomplete / corrupt :
Some file attaches are missing, or some file attach is "cut in the middle" (partial download)

Expected Results:  
Show a valid email (not corrupt ;-)

To correct the problem (it's not 100% working but 70%) :
1. I move the email in another folder on the same server
2. I WAIT (VERY IMPORTANT) Thunderbird to finish all activities
3. Normally (but not always) the email is complete
4. Anyway I can always get the complete email on server via webmail
(Q1) POP3? IMAP? If IMAP, offline-use=Off? offline-use=On?

(Q2) Problem like next known issue? 
> Bug 92111 Microsoft Exchange IMAP: Attachments not fully downloaded (due to incorrect/smaller RFC822.SIZE by MS Exchange, which is RFC violation)
> Bug 390795 -  attachments corruption for IMAP account (truncated due to estmated/incorrect RFC822.SIZE)

(Q3) If IMAP and offline-use=On, next issue is already known.
  If download of mail data fails at mid of download of mail data,
  it'll never be re-downloaded again until rebuild-index, or until mil is
  downloaded again as new(different) mail.
  (move to other folder, then move back to folder)
  This case?
My answers :

(A1) IMAP
offline-use=On (I don't use offline much but "This folder is selected for offline usage")

(A2) No it's not Bug 92111. Our server is cyrus imapd server, and not everyone using it has the same problem. Only me.
About bug 390795, yes it's very near, not all comments apply though...

(A3) Yes it's this case... Is there a bug about it ? (I extensively searched bugs before posting) Or a solution ? [Thanks a lot, by the way]
(In reply to comment #2)
> (A3) Yes it's this case... Is there a bug about it ?
>      (I extensively searched bugs before posting)

I don't know specific bug. I only know that some(greater than a few) bug reports like yours exist. In such bugs, I couldn't see important findings other than "partial" and "rebuild-index is required"(e.g. Bug 468595). AFAIR, I couldn't see "when/how becomes partial", "detail of partial" etc. in bugs reports for "partial download in offline-store".

Bug 501253 is only exception, although offline folder or not is still unclear.
  
I could see data like next in Cache during duplication test of Bug 498473. 
> --Boundary_(ID_SjJJqft3zJ3V7JlRrinaFA)
> Content-type: application/pdf;
>(snip)
> 
> This body part will be downloaded on demand.
> --Boundary_(ID_SjJJqft3zJ3V7JlRrinaFA)
And opener of Bug 501253 reported next.
> <div class="moz-text-plain" wrap=true graphical-quote=true style="font-family: -moz-fixed">
> <pre wrap><BR><HR WIDTH="90%" SIZE=4><BR>This body part will be downloaded on demand.</pre></div>

Bug 498473 : Non offline folder. Data like above was seen in Cache direcory.
             It's mainly by download-on-demand. "Partial(unable to open PDF)"
             was caused by this bug. See attached data to Bug 501253.
Bug 501253 : Unable to view PDF attachment.
             Offline folder or not is still unclear.

Another case is Bug 501851(not "partial" issue).
I could produce Bug 501851(whole mail data corruption because of wrong pointer). 

Do you still have partially downloaded mail data in offline-store?
If yes, can you check mail data in offline-store?
 1. Copy offline-store file to a file under POP3(real or dummy) account or
    "Local Folers"
 2. Restart Tb, and "Rebuild-Index" of the copied local mail folder.
 3. Copy mail which is partial to a new local mail folder.
 4. Check mail source of the "partial" mail.
I also see thunderbird 3.0b3pre sometimes displaying a message (from IMAP) before it was fully downloaded. It then displays that the messagebody will be downloaded when needed upon demand, but doesn't do so - I expect because it already thinks it has cached the full message. There is no way to "reload" it. Only chance: Move message via webmail, go to that folder and tbird will think the message is new and redownload.

There seems to be something terribly broken. What makes tbird think that (I assume) when a download of a message failed it still marks it as "has been cached fine"?
As I wrote in Bug 468637 Comment #10, I could produce partial mail data in
offline-store using test mail attached to Bug 501253, without utilizing known bug.
In that case, "partial mail data in offline-store" is for JPEG file, then data for HTML nor PDF is not held in offline-store file. So phenomenon of "can't find the pdf file attachment"(phenomenon Bug 501253 reports) is also observed in addition to "This body part will be downloaded on demand."(phenomenon Bug 468637 reports).

Simple mail structure? Or not-so-well formed mail like test mail of Bug 501253(see Bug 468637 Comment #10 for structure of mail)?
Note:
Test procedure of Bug 468637 Comment #10 step (7) is an emulation of your next STR in Comment #0.
> Steps to Reproduce:
> 1. When receiving a big email, I do not wait it to arrive but :
> 2. I immediately click on it while Thunderbird is still downloading
> 3. Normally (not always) the email is then corrupt.
Bug 468637 Comment #10 step (8) is emulation of usual/not-so-busy user's behaviour.
(In reply to comment #0)
> But when MOVING the mail on the mail server into another folder, the message could be valid
> or not depending on the time Thunderbird waited to download the message.

It's not surprising. (1) Mail data on IMAP server is not corrupted, and "move/copy of mail to other folder of same IMAP account" is requested by IMAP command of "uid xxx copy target-folder", and copy operation is executed at IMAP server. (2) After move/copy by Tb, header of copied mail is usually fetched immediately after move/copy, because Tb already knows existence of new mail in target folder. And, if auto-sync=on & offline-use=on, download by auto-sync is usually executed immediately, or, sooner or later. And, if move/copy by Tb user, target folder is not opened at any Tb's window usually(opened folder is move/copy source folder usually). So event like next usually doesn't occur after move/copy of only several mails by user.
> Steps to Reproduce:
> 1. When receiving a big email, I do not wait it to arrive but :
> 2. I immediately click on it while Thunderbird is still downloading
(In reply to comment #4)
> There is no way to "reload" it.

There are two ways.
 a) Rebuild-Index of Tb.
 b) Move the mail to other folder of same IMAP account by Tb, then move back.
    (Read Comment #7)
    This is same as your "Move message via webmail" but simpler.
Since a) forces re-download of all mails, b) is practical, although b) may produce small problem in threaded view because b) changes UID of the mail("Order Received" column value is UID of mail when IMAP).
(In reply to comment #3)
> Do you still have partially downloaded mail data in offline-store?
> If yes, can you check mail data in offline-store?
>  1. Copy offline-store file to a file under POP3(real or dummy) account or
>     "Local Folers"
>  2. Restart Tb, and "Rebuild-Index" of the copied local mail folder.
>  3. Copy mail which is partial to a new local mail folder.
>  4. Check mail source of the "partial" mail.

I must admit I don't understand how to do this ?

In my Thunderbird profile I have :
* A directory called "ImapMail" containing a lot of directories called with the server name of my email accounts
* In these directories I have :
  * files called like the IMAP folder (with .MSF extention and with no extention)
  * In all folders I have lots and lots of HUGE nstmp, nstmp-1, ... nstmp-14... files ?
For your information I tried "fetch_by_chunks". After some days I cannot say if it works better. It seems to work a little better, but at least not always, I still have problems like this :

One of the problem (is it the same bug or another one ?) is the following :
1. When dragging received attached files from the message to Windows Desktop I get an error message (Thunderbird Windows popup window) : "Cannot move <name of file>: It is being used by another person or program. Close any programs that might be using the file and try again"
2. But when I do "Save as" instead of drag&drop it works and files are valid...

I found no bugs about this, except :
http://forums.mozillazine.org/viewtopic.php?f=39&t=598205&start=0
https://bugzilla.mozilla.org/show_bug.cgi?id=433849
(In reply to comment #9)
> I must admit I don't understand how to do this ?

File to hold offline data of IMAP has same structure as file for local mail folder (Unix Mbox format. Search Google for "Unix Mbox").
So, if you copy file named ...\ImapMail\imap.gmail.com\Inbox (not Inbox.msf) to ...\Mail\Local Folders\Inbox-IMAP, you can see the "Inbox-IMAP" as local folder of "Inbox-IMAP" owned by pseudo account of "Local Folders".
If you define dummy POP3 account with hostname(server name)=dummy.pop3.server without Global Inbox use, you can see "Inbox-IMAP" as local folder of "Inbox-IMAP" owned by dummy POP3 account of "dummy.pop3.server" by copying the file to ...\Mail\dummy.pop3.server\Inbox-IMAP.



> 
> In my Thunderbird profile I have :
> * A directory called "ImapMail" containing a lot of directories called with the
> server name of my email accounts
> * In these directories I have :
>   * files called like the IMAP folder (with .MSF extention and with no
> extention)
>   * In all folders I have lots and lots of HUGE nstmp, nstmp-1, ... nstmp-14...
> files ?
(In reply to comment #9)
> * In all folders I have lots and lots of HUGE nstmp, nstmp-1, ... nstmp-14... files ?

I can say nothing about what happened on your PC, but I can say that it can occur  during usual use(no crash, no freeze, no kill of Tb, ...), if "Compact Folder(s)" is interfered by external program such as auto-backup, file scan of anti-virus software.
See bugs listed in Dependency Tree for meta Bug 498274.
If you frequently experience "partial offline-store data" problem, it may be a result of interfere by external program, because program such as auto-back keeps offline-store-file open for log time if big mbox. It'll interfere Tb's file writing.

(In reply to comment #10)
> For your information I tried "fetch_by_chunks". After some days I cannot say if it works better.

It's perfect workaround of "IMAP server lies about data length". It can do nothing for partial data in offline-store, even though it may reduce frequency of partial offline-store data problem.
I got something even worse, dont know if related - I have, in my inbox, an e-mail that starts (in source code view) with no headers, but:
This is a multi-part message in MIME format.

--=-------------12495995561580099052
Content-Type: text/plain; charset=iso-8859-15
Content-Transfer-Encoding: quoted-printable

(email text)

--=-------------12495995561580099052
Content-Type: application/octet-stream;
 name=(attachment name)
Content-Disposition: attachment;
 filename=(attachment name)
Content-Transfer-Encoding: base64

(base64 encoded data, about 1.2k)

AND THEN, after quite exactly 2 KB of total data taken from the middle of the mail:
Headers, body again including the part that is also on top, complete e-mail with encoded attachment. Attachment below (>100k) decodes correctly when decoded manually.

Using Gmail via imap on tb3beta3. Offline storage enabled. Was switching between the mail that was being downloaded and other one, back and forth quite quickly multiple times. The mail is completely unuseable for regular users, as thunderbird does not find the headers. Trying to keep it as a testcase. Looks quite bad to me.
Flags: blocking-thunderbird3?
No longer blocks: 468595
Depends on: 468595
This should be fixed in b4, for newly downloaded messages - please let me know if you see newly corrupted messages in b4.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → DUPLICATE
Flags: blocking-thunderbird3? → blocking-thunderbird3-
I have a similar problem with POP3 email.
Fetching the email into an empty Inbox sometimes results in the second email (or subsequent) having the correct text in the listing in the Inbox, but points to the content of the preceding email. It started about a year ago & has carried on since (first on an XP system & then on W7) a second XP system exhibits similar issues.
For a while now I've suspected a timing/locking issue in TB. Last night the symptoms strongly suggested that TB wasn't waiting until some process or other had completed.
Last night it wasn't the occasional message getting corrupted it was about 5 occurrences out of 30 emails, and it wasn't just one message getting corrupted either it was two or three per occurrence (i.e. message 2 & message 3 indexed message 1's content)
The thing which was different about last night is that I was getting the emails after my nightly backup dumps had run, so there were long delays in programs responding as stuff was brought back in from the swap file.
After deleting Popstat.dat & re-fetching the emails everything worked just fine.
On previous occasions I've monitored the LAN, so I know the emails are fine as they come in from the POP3 server.
You need to log in before you can comment on or make changes to this bug.