817212 - Attachments are not fully downloaded using IMAP with google

Status: RESOLVED DUPLICATE of bug 647562

(MailNews Core :: Backend, defect)

Description

[Wah]

Attachment: Example of mismatching attachment size

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
Build ID: 20121024073032

Steps to reproduce:

Attempted to save an attachment downloaded using IMAP through gmail. This has happened to jpg and ptf (Pro Tools Project) files.

I automatically download attachments.


Actual results:

The attachment listed in the message was of a smaller size than that as shown in gmail. When I save the attachment using thunderbird, the file is corrupted or in the case of the Jpg the bottom part of the image was missing.

If I download the attachment from Gmail through the web browser, it is fine.


Expected results:

Thunderbird should have downloaded the full attachment.

Comment 1

[Thomas D. (:thomas8)]

Same/similar was reported in Bug 818733.

Comment 2

[Eric Moore]

I don't have this problem with my Gmail IMAP account with 17.0.6 or the 24.0a1 (2013-06-12) Daily build but I have mail.imap.mime_parts_on_demand set to false because I've occasionally ran into similar problems in the past. Could you retry after setting mail.imap.mime_parts_on_demand false using the config editor and restarting Thunderbird, to see if that has any effect? That disables "MIME Parts On Demand" optimizations and fetches the entire message (including any attachments) when you open a message in a remote folder.

http://kb.mozillazine.org/Entire_message_fetched_when_opening_an_IMAP_message

Comment 3

[Eric Moore]

Forgot to check the checkbox for need more information :)

Comment 4

[Wah]

Hi Eric

My Computer is broken at the moment (Motherboard Warranty Issue) - Will get to it once I get the Mobo back and re-install windows... might be a while though sorry.... (Not to mention the problem may disapear with a new install)

I will keep you posted

Cheers

Comment 5

[WADA:World Anti-bad-Duping Agency]

> Bug summary : Attachments are not fully downloaded using IMAP with google

What is evidence that your problem was acturely/surely caused by flaw in Tb's program code?

Please note that here is bugzilla.mozilla.org where to report Tb's actual bug==actual flaw in Tb's code to developers, and to ask them to resolve actual Tb's bug. Here is never place to simply report to someone about funny phenomena for you what you saw while you are using use Thunderbird（i.e. Here is never support forum.)

Comment 6

[Wah]

Hi WADA

I assumed it was a bug in underbird as thunderbird did not behave as you would expect. I thought a repeatable "funny phenomena" in a program is a bug?

where else would I get support for this issue?

My recent comment is to let Eric know that I will look at it once my computer is up And running again. It took 6 months to get a suggestion, I was being polite by letting Eric know that it may take some time before I can test his solution.

Please try and be constructive and helpful.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to Wah from comment #6)

Funny phenomenon for you, phenomenon in Tb what is different from your expectation, is not always Tb's bug.
  Server problem, router problem, network related problem, addon's bug,
  anti-virus problem, mail sender's problem(malformed mail),
  proxy server problem, local proxy problem(usually by anti-virus
  software), ...
  can produce problem.
So, such issues should be surely ruled out before report pheomenon you saw as "Tb's bug" to Tb developer.
Is such problem already ruled out?

The reason why I can't think "Tb's bug".
I also use Gmail IMAP with Seamokey and Thunderbird daily, with mail.server.serverXX.fetch_by_chunks=true, mail.imap.mime_parts_on_demand=true (i.e. no bypass of problem like Bug 92111). 
But I don't experience problem like "partial attachment part(shorter attachment part)" with Sm/Tb, with Gmail IMAP. And, I don't see such problem report like "partial attachment part(shorter attachment part)" with Gmail IMAP in bugzilla.mozilla.org, although I know Tb's bug like "2048 bytes of top part of mail + entire mail data is saved as mail data by Tb" which may occur with any IMAP server.
Many of problem like "partial attachment part(shorter attachment part)" was; 
  Server returns wrong(smaller) RFC822.SIZE(for example, Bug 92111)
AFAIK, Gmail IMAP didn't have such problem.
Example of known(already fixed) bug of "far smaller attachment data than original is saved".
  Base64 encoded "Thisbodypart..." is saved(Bug 774217)
However, if this kind of problem, phenomenon like "in the case of the Jpg, the bottom part of the image was missing" can't occur.

> where else would I get support for this issue?

Help forum of Mozilla messaging company, Support forum of MozillaZine, ...
Help for "what kind of data is needed to report bug at B.M.O" is also obtained at such forum.
Help for "problem determination procedure" is also obtained at help or support forums.

Have you checked message source shown by Tb?

Actually shorter data than original data held at server? If so, last part of mail? Or mid part of multipart?
Actually non malformed mail or non broken data?
  Example: 
  Some mail app sends standalone "=" line in quoted-printable.
  ...=[CRLF]...[CR+F]=[CRLF] (... is valid quoted-printable string)
  Tb saves the malformed =[CRLF] at end as "=" according to RFC.
  Some mailers ignore the malformed =[CRLF] at end upon save.
  Then, someone says Tb's bug at B.M.O.
Is such malformed mail or broken data case surely ruled out?

As seen in bug 823838 and bugs in dependency tree for that bug("top 2048 byte * entire mail" case I previously referred), Tb actually has problem around fetching mail data, although such known bug can't produce phenomenon like "in the case of the Jpg, the bottom part of the image was missing".
Because your case is IMAP case, it's affected by many elements than POP3 case, such as auto-sync and Offline-Use=On/Off, Memory/Disk Cache is intentionally disabled or not if no auto-sync or Offline-Use=Off, if Offline-Use=Off, setting around Tb's behavior in fetching mail data(mime-parts-on-demand, fetch-by-chunk, ...), proxy sever is used or not(if non-SSL, mail-scanner of anti-virus software does port-scan, but if SSL, local proxy or SSL tunnel is usually used), ...

What is your setting in your environment?

Comment 8

[Wah]

Hi Wada

I would assume that it would not be mail senders bug, as there was no problem in GMail with the attachment, just in Thunerbird.

It could be antivirus, but I didn't think of that. I was using Avast AntiVirus... I could try disabling it to see if that makes a difference.

My Account was setup for Offline Use with Auto-Sync (I think)... again, my computer is in parts at the moment and I won't be able to check this out for a few weeks until I get it back together.

I tried searching google/forums for an answer before posting here and could not find anything. I thought the bug was in accordance with the bug writing guidlines...  If further information etc. is needed then why isn't it shown on this website rather than a seperate forum?

I was just after some help and do not know the inner workings of config settings behind Thunderbird... If there are other things to try that may be causing the problem (Antivirus etc) then I will try it and if it works will close this off. 

The JPG Attachment was showing half an image becuse of missing data at the end of the attachment. This depends on interlacing etc. but half an image will display if only half the data is there (As an example).

Thanks for your help

Comment 9

[WADA:World Anti-bad-Duping Agency]

Question about condition when problem occurs.
Do you usually open newly arrived mail immediately and problem occurs when such immediate open happens?
- Usuallyy click new mail at thread pane just after new mail arrived.
- Thread pane is sorted i reversed order, and new mail is shown at top,
  and newest mail is always shown at message pane.
Or problem occurs on already downloaded mail by aut-sync?

Problem on mail moved/copied mail by message filter?

Permanent problem on a mail? Or one time failure?
Because auto-sync is enabled and Offline-Use=On in your case, entire mail data is already stored in offline-store file by Tb, and because Tb thinks held data is clean, Tb never issues fetch again for the mail. So, I think, once problem occurs on a mail, problem is reproducible any time with the mail.
Is it right?

Do you see your problem on *copy* or *original*" of the mail in other Mbox of Gmail IMAP?

Try following, if you can still see problem on a mail, or when you see problem next time, please.
(1) Call "mail on which problem occurred" "mail-X in Mbox-X of UID=UID-X".
    Create Tag of Tb(call Tag-X).
    Add Tag-X to the "mail-X in Mbox-X of UID=UID-X".
(2) Restart Tb, to surely see added Tag in [Gmail]/All Mail.
    Open [Gmail]/All Mail, search mail which has Tag-X.
    (this is original of mail-X)
Does your problem occur on the *original* mail in [Gmail]/All Mail too?

If problem doesn't occur on original in [Gmail]/All Mail, check difference between two mails, please.
- save mail-x in Mbox-X to email-X.eml
- save original mail [Gmail]/All Mail to email-Y.eml
What is difference between two .eml files? "loss of bottom part only in email-x.eml"?
If so, difference of file size == 2048?
(2048 bytes by previewText is not written to offline-store file, but size is counted, so bottom 2048 bytes of entire mail data is not written to offline-store file...)

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Wah from comment #8)
> I would assume that it would not be mail senders bug, as there was no
> problem in GMail with the attachment, just in Thunerbird.

"Malformed mail or not"(or sender side bug or not) is never assumption. "Malformed mail or not " is always determined by "message source" and "mail related RFCs".

"No problem in other mailer" only is never sufficient evidence of "mail is never malformed". At least, Gmail(Web mail) has quirks for some kind of malformed mail, for example, torelance with misuse of multipart/mixed for correct multipart/related, which Tb doesn't have. Gmail displays multipart/mixed mail as if multipart/related mail, but Tb displays multipart/mixed mail as multipart/mixed mail and displays multipart/related mail as multipart/related mail. Creator of such mail of "misuse of multipart/mixed" always says "Gmail correctly shows his mail, so this is Tb's bug".

Please surely rule out "malformed mail" case including "misuse of message header" or "misuse of parameter of message header".

Comment 11

[Josiah Bruner [:jsbruner]]

This is most definitely not a front end bug. So setting to general since the exact issue is yet to be discovered. 


As for this debate you guys seem to be having:


As long as Wah can not reproduce a similar problem on other clients (Outlook), then this is surely our bug. Filling a bug for further testing is completely allowed here, so there is no need to make Wah feel like he is lacking knowledge. Even if this is caused by a server problem, we could probably handle the error better. Now of course, the more info Wah would like to add, the faster this bug will be fixed (If it turns out to be one) Thanks! 


Respectively,
Josiah

Comment 12

[Wayne Mery (:wsmwk)]

Wah, please test  setting mail.imap.mime_parts_on_demand false using the config editor and restarting Thunderbird, to see if that has any effect and post your results here.

Comment 13

[Wayne Mery (:wsmwk)]

(if valid, this is either backend or imap)

Comment 14

[WADA:World Anti-bad-Duping Agency]

Same mail data pattern as bug 647562?
If such case, and if "attached image file" with "View/Display Attachments Inline", and if "Offline-Use=On with auto-sync enabled", "partial imade data may happen.

Comment 15

[Wah]

Hi Wayne

To be honest, I had completely forgotten about this as it has been too long between getting this computer up and running, and also I have installed a new version of Thunderbird and primarily use gmail as an email interface now.

I made the change and have checked the file that originally had the issue and it is fine.
However, this computer is a new build and I now have Thunderbird 24.2.0 installed. (Due to above mentioned Motherboard... it was replaced with a different model)

To be honest, this could have fixed it, but due to the version change and the computer rebuild, I can't be 100% sure on what caused the issue.

I changed the setting back to True, re-emailed the file to myself and it wasn't an issue.

I haven't experienced the issue again as I am using gmail and not really using Thunderbird except for downloading a local cache of my email to my computer for backup purposes.

Thanks for your time

Wah

Comment 16

[Wayne Mery (:wsmwk)]

Reporter reports WFM.
Assuming bug 647562 addresses comment 11, so duping