824972 - Imap: TB truncates message with attachment: Cannot open some attached pdf files (but they can be opened correctly from Gmail)

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: IMAP, defect)

Description

[MQW]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0
Build ID: 20121128204232

Steps to reproduce:

I can't open some pdf files from attached files in Thunderbird, but I can open them from my account Gmail in Firefox. When I open them from Thunderbird with Adobe Reader, I often get an error "The file is damaged and could not be repaired."

I read a lot of threads but none works. For example, I tried using the extension OpenAttachmentByExtension but it doesn't work. I tried removing the mimeTypes.rdf but still can't open some pdf. I also tried uninstalling and re-installing Adobe Reader and Thunderbird. And I tried changing mail.server.default.fetch_by_chunks to false in Config Editor, but still nothing.

I read here : https://bugzilla.mozilla.org/show_bug.cgi?id=659355 that the bug was fixed with the new version 17.0, so I don't understand why I still can't open some pdf files. The problem appears suddenly since about a month.

Comment 1

[Thomas D. (:thomas8)]

Maison... (how about adding a shorter nickname in bugzilla instead of your long email address ;)

We need a testcase message, otherwise there's not much we can do.
- Select an affected msg in TB
- Ctrl+U to view source
- save page as testcase.eml
- "add an attachment" (testcase.eml) to this bug (see link above)

I recall a number of open/unconfirmed bugs around PDF, many are poorly or malformed messages.

Comment 2

[Thomas D. (:thomas8)]

Pls remove private data (email addresses etc.) before adding the testcase!

(In reply to Thomas D. from comment #1)
> Maison... (how about adding a shorter nickname in bugzilla instead of your
> long email address ;)
> 
> We need a testcase message, otherwise there's not much we can do.
> - Select an affected msg in TB
> - Ctrl+U to view source
> - save page as testcase.eml

- edit testcase.eml with text editor and remove private data as necessary!

> - "add an attachment" (testcase.eml) to this bug (see link above)
> 
> I recall a number of open/unconfirmed bugs around PDF, many are poorly or
> malformed messages.

Comment 3

[MQW]

Attachment: Testcase1.eml: Example of an affected message as downloaded by TB (truncated somewhere within attached file Noel_END5.pdf)

Comment 4

[MQW]

(In reply to MQW from comment #3)
> Created attachment 696268 [details]
> Testcase of an infected message

(nickame added, sorry ^^)

I just added the testcase in attachment, I hope I've removed all private data. Let me know if you want another testcase.

I first think that the PDF was corrupted but I can open it directly from gmail, and there are several PDF I can't open from Thunderbird so...

Thanks !

Comment 5

[Thomas D. (:thomas8)]

(In reply to MQW from comment #4)
> (In reply to MQW from comment #3)
> > Created attachment 696268 [details]
> > Testcase of an infected message
> (nickame added, sorry ^^)

thanks! :)

> I first think that the PDF was corrupted but I can open it directly from
> gmail, and there are several PDF I can't open from Thunderbird so...

MQW, I examined the testcase.

I suspect that the message is truncated, because it does not have a closing boundary:

start boundary:
> Content-Type: multipart/mixed;
>  boundary="------------010402000706000102040001"

so there there should be the following closing boundary which is missing from source of testcase1 (attachment 696268 [details]):
> --------------010402000706000102040001--

When I save the attachment (Noel_END5.pdf) to hard disk, it has the following size:
237.478 Bytes

MQW, when you save Noel_END5.pdf from Gmail to your hard disk, can you tell me the file size of the correctly working file which can be opened?

If possible, attach Noel_END5.pdf to this bug for comparison purposes.

Comment 6

[MQW]

Attachment: Noel_END5.pdf (Full file originally attached to Testcase1.eml)

Comment 7

[MQW]

(In reply to Thomas D. from comment #5)
> I suspect that the message is truncated, because it does not have a closing
> boundary:
> 
> start boundary:
> > Content-Type: multipart/mixed;
> >  boundary="------------010402000706000102040001"
> 
> so there there should be the following closing boundary which is missing
> from source of testcase1 (attachment 696268 [details]):
> > --------------010402000706000102040001--

I'm not an expert in mails so ok I trust you ^^

> 
> When I save the attachment (Noel_END5.pdf) to hard disk, it has the
> following size:
> 237.478 Bytes
> 
> MQW, when you save Noel_END5.pdf from Gmail to your hard disk, can you tell
> me the file size of the correctly working file which can be opened?

When I save the correct file, it has this size : 3463 Ko

> If possible, attach Noel_END5.pdf to this bug for comparison purposes.
(In reply to MQW from comment #6)
> Created attachment 696959 [details]
> Correct PDF Noel_END5

Just added =)

Plus : I just had the same problem on an other PC this morning, it never happened before on this one.

Comment 8

[Thomas D. (:thomas8)]

CC Wada, perhaps this is a dupe of something he's tracking.

(In reply to MQW from comment #7)
thanks MQW, that's very helpful.

Size of attached file Noel_END5.pdf (attachment 696959 [details]) when saved to disk:
-   237.478 Bytes when saved from TB (broken, see reporter's attachment 696268 [details])
- 3.545.268 Bytes when saved from Gmail webmail (working)

iow, when TB downloads the msg, the attachment is truncated (or more precisely, the message is truncated somewhere within the mimepart of the attachment).

Comment 9

[Thomas D. (:thomas8)]

MQW, just to be on the safe side, can you confirm this:
- TB account receiving broken messages is set up as *IMAP* account?
- TB account receiving broken messages is a gmail email address?

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Thomas D. from comment #8)
> CC Wada, perhaps this is a dupe of something he's tracking.

Suspected phenomenon is one like Bug 92111 with "fetch body[]" and similar phenomenon with "fetch body[m.n] for a part".
"This kind of problem or not" is easily be determined by IMAP log and by quick log check by problem reporter himself(returned data length value as RFC822.SIZE and actuallly returned data length).

Why such IMAP log is not checked by bug opener yet in this bug?

Comment 11

[MQW]

(In reply to Thomas D. from comment #9)
> MQW, just to be on the safe side, can you confirm this:
> - TB account receiving broken messages is set up as *IMAP* account?
Yep, the account is set up as IMAP

> - TB account receiving broken messages is a gmail email address?
At first, yes. But last friday (2012-12-31), I had the same problem on another computer with my proper adress mail hosted by 1&1.

WADA, I admit I hadn't understand all you said. Do I have to do something ?

Comment 12

[WADA:World Anti-bad-Duping Agency]

> -   237.478 Bytes when saved from TB (broken)
> - 3.545.268 Bytes when saved from Gmail webmail (working)

Delta of file size == 2307790
If (a) RFC822.SIZE for fetch body[] == NN-2307790(where NN=entire mail data size), or if (b) RFC822.SIZE for fetch body[m.n] == 237478 (should be 3545268), phenomenon similar to Bug 92111 may happen.
Following in comment #5 by Thomas D. is a characteritics of problem like Bug 92111 when (a).
> so there there should be the following closing boundary which is missing
> from source of testcase1 (attachment 696268 [details]):
> > --------------010402000706000102040001--

To rule out such problem, check of message source which Tb obtained and check of IMAP log will be needed.

Checking of content of "corrupted 237478 Bytes file" is also needed.
If the "corrupted 237478 Bytes file" is top part of "correct 3545268 Bytes file", similar problem to Bug 92111 is suspected.
But if it's not, your problem can't be similar problem to Bug 92111. 

Because delta of 2307790 looks too large for wrong RFC822.SIZE, and because many of "problem due to wrong RFC822.SIZE" is already worked around by fixing of Bug 92111 and follow-up bugs, it may be problem like bug 815012(see also bugs in Dependency tree for that bug).

In any case, "fetch of mail from IMAP server is done correctly or not" is important, and it is easily known by checking IMAP log.

Comment 13

[Thomas D. (:thomas8)]

Attachment: Noel_END5.pdf (truncated at 237.478 Bytes of target file size on disk, after incomplete msg download by TB)

This is the truncated (and hence corrupted) Noel_END5.pdf file attached to Testcase1.eml, which is the incomplete msg download by TB (attachment 696268 [details]).

Comment 14

[Thomas D. (:thomas8)]

(In reply to WADA from comment #12)
> Checking of content of "corrupted 237478 Bytes file" is also needed.
> If the "corrupted 237478 Bytes file" is top part of "correct 3545268 Bytes
> file", similar problem to Bug 92111 is suspected.
> But if it's not, your problem can't be similar problem to Bug 92111. 

File comparison of
- Noel_END5.pdf (original file, 3.545.268 Bytes, attachment 696959 [details]), and
- Noel_END5.pdf (truncated file,  237.478 Bytes, attachment 699590 [details])
shows that the truncated file is indeed identical with top part of original file, so it has just been cut off (because the msg was not fully downloaded, as evidenced by missing final boundary), but not otherwise corrupted.

(In reply to WADA from comment #12)
> > -   237.478 Bytes when saved from TB (broken)
> > - 3.545.268 Bytes when saved from Gmail webmail (working)
> 
> Delta of file size == 2307790
> If (a) RFC822.SIZE for fetch body[] == NN-2307790(where NN=entire mail data
> size), or if (b) RFC822.SIZE for fetch body[m.n] == 237478 (should be
> 3545268), phenomenon similar to Bug 92111 may happen.

Wada, RFC822.SIZE returns size of *mime-encoded* msg data, right?
But file sizes of original and truncated file (3.545.268 and 237.478 Bytes) are size of *decoded* data after saving files to disk. So I think the following statement is not right:

> RFC822.SIZE for fetch body[m.n] == 237478 (should be 3545268)

Instead, if I understand this correctly:

a) The truncated size of *mime-encoded* part body[m.n] is *not* 237478 Bytes, but whatever the *mime-encoded* size is for the 237.478 Bytes of truncated file after decoding & saving to disk.

b) RFC822.SIZE for fetch body[m.n] should correctly return whatever the *mime-encoded* size is for 3.545.268 Bytes of original file (not 3.545.268 Bytes of decoded file size).

Comment 15

[Thomas D. (:thomas8)]

(In reply to WADA from comment #10)
> "This kind of problem or not" is easily be determined by IMAP log and by
> quick log check by problem reporter himself(returned data length value as
> RFC822.SIZE and actuallly returned data length).

Objection, Sir ;) Especially for users "New to Bugzilla" (like reporter MQW), creating & retrieving of IMAP log will neither feel "easy" nor "quick", not to mention analysing such an animal...

> Why such IMAP log is not checked by bug opener yet in this bug?

dito, and I guess because I myself am not very familiar yet with those logs so I didn't request it from reporter ;)

So Wada, thanks for pointing us to the need for the IMAP log.

(In reply to MQW from comment #11)
> WADA, I admit I hadn't understand all you said. Do I have to do something ?

MQW, I think WADA is requesting you to enable TB logging for your IMAP activities and then make the log file available to us for analysis (not sure about privacy concerns, pls remove private data before attachig to this bug, or just send to both WADA and me via private mail). The instructions how to get the log are found here:

https://wiki.mozilla.org/MailNews:Logging

or perhaps here:

http://email.about.com/od/mozillathunderbirdtips/qt/et_mail_log.htm
(but you'd probably want logging level IMAP:5 where they say IMAP:4)

Comment 16

[MQW]

I just send to you two by private mail two logs from the two computers I previously mentionned. One for the gmail adress, the other for the adress hosted by 1&1.

I took a look at the logs but I didn't understand much.

Tell me if you need anything else =)

Comment 17

[WADA:World Anti-bad-Duping Agency]

Correction, to avoid going to wrong direction.
 RFC822.SIZE for fetch body[m.n]
 == size after base64 encode of 237478 bytes data(if decoded, 237478 bytes)
 should be size after base64 encode of 3545268 bytes data(if decoded, 237478)
My point is:
  Actual saved file size is 237478 bytes, even though original is 3545268 bytes.
  RFC822.SIZE should de number corresponds to original 3545268 bytes,
  instead of number smaller than it or larger than it.

Why such problem is suspected is; attached .eml is as follows.
> Content-Type: multipart/mixed;
>  boundary="------------010402000706000102040001"
>
> This is a multi-part message in MIME format.
> --------------010402000706000102040001
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Content-Transfer-Encoding: 7bit
>
>
>
> --------------010402000706000102040001
> Content-Type: application/pdf;
>  name="Noel_END5.pdf"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment;
>  filename="Noel_END5.pdf"
>
> JVBERi0xLjYNJeLjz9MNCjcgMCBvYmoNPDwvTGluZWFyaXplZCAxL0wgMzU0NTI2OC9PIDkv
> (snip, dala lines are removed)
> (last part of PDF attachment. Bottom of .eml file.  
> Wg1lWarVm61uf+J6zrBrNmq1agj1jo2tBxFcFGWWiMMgb7BXDLB29FYhCuk6uDliCQSD3sIo
> rnpVIN8AGRYTGl26XvLdeVy9Pq+cR/4mvz22UtefpyQcKqpL7QYzC9
>
Boundary for next part(--------------010402000706000102040001) nor close boundary(--------------010402000706000102040001--) is not seen in mail data. This indicates that data after bottom line in .eml file is not sent from server, or is not received or not saved by Tb.

(In reply to MQW from comment #16)
> I took a look at the logs but I didn't understand much.

Because big attachment case and because imap:5 writes mail data in log, log data is usually huge.
Do first following to see IMAP flow such as;
  uid m:* flags
  uid xx fetch body[header.fields] ( ... )
  Offline-Use=Off : uid xx fetch body[bodystructure], uid xx fetch body[m.n]
  Offline-Use=On  : uid xx fetch body[]
(1) Create new profile, define one IMAP account only 
(2) Do following for minimum configuration.
Disable Global Search and Indexer, disable automatic new mail check, disable automatic mail deletion/purge, disable IDLE command use, set max cached connection=1(Server Settings/Advanced)
(2) Create a folder for test(call FolderX) with same Offline-use=On/Off setting as your daily use(Folder Properties/Synchronization)
(3) Edit your .eml file, remove all data lines in application.pdf part except first line, add close boundary line, save.
(4) Drag&Drop(import) the .eml file to FolderX. Terminate Tb, with account selected(to force no IMAP activity upon restart)
(5) Restart Tb with IMAP logging enabled.
(6) Clink FolderX(oppen FolderX) where only one test mail is contained.
    "Repair Folder", view the mail, save pdf if required(size==0, or not saved)
(7) Terminate Tb, and edit log file by Text editor.
    Because minimum log, flow of fetching mail data is visible.
    For IMAP command/response, see http://tools.ietf.org/html/rfc3501
(8) After it, delete many log lines for attached pdf data from original IMAP log except top line and bottom line(data itself is irrelevant and is not needed, unless counting of actual sent bytes in log data is needed), remove/replace private data from log file, remove irrelevant data such as LSUB, LIST etc. If file size is reduced, log file is visible/readable.
Note: because data is sent like following format, sent data size can be obtaied from log without detailed log for actual attached data.
  {LL}data_lines where LL is sending data length

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to MQW from comment #16)
> the other for the adress hosted by 1&1.

Log when your problem is reproduced?
If not, it's useless, except for seeing ordinal behaviour of your server...

Comment 19

[Wayne Mery (:wsmwk)]

MQW, do you still see this when using version 45?

Comment 20

[Phoenix]

Resolved per whiteboard