1654995 - Thunderbird 79.0b2 corrupting large attachments because of chunking

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Peter Robinson]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36

Steps to reproduce:

I have checked e-mails with large > 2Mb and small < 100kb attachments, large

Actual results:

Large attachments are corrupted dwg files unreadable, pdf files partial or unreadable, jpg files display part of image. Same files downloaded direct from Yahoo are unaffected. Small files seem to be OK, don't have time or inclination to try multiples of various size files to ascertain problem cut off, but definately only happened since update on 16th July 2020

Expected results:

I would have hoped that all files would be received un-corrupted

Comment 1

[Jorg K (CEST = GMT+2)]

but definately only happened since update on 16th July 2020

Related to bug 1580480?

Comment 2

[Magnus Melin [:mkmelin]]

Peter, so this is on beta? Do you see it with 78.0?

Comment 3

[Jorg K (CEST = GMT+2)]

Well, Peter filed it for 79.0b2.

Comment 4

[gene smith]

Reporter Peter, I assume this is an imap account? If so, are you letting tb store your emails locally (the default) or keeping message bodies only on the server?
If stored locally, are the problem emails just now received or are they messages that have been present on the system and possibly opened OK before?
Would you characterize your network connection to your imap server as generally "fast" or "slow"?
Please look in the config editor and tell me the values you see for:
mail.imap.chunk_size = ?
and
mail.imap.min_chunk_size_threshold = ?
If they are bold font, try resetting them back to default and restart tb and see if that helps.
If that doesn't help, set mail.server.default.fetch_by_chunks to false and restart tb. Does that help?

Comment 5

[Peter Robinson]

Problem occurs on download, not with locally stored e-mails, fast connection 100Mbs, sadly new to all this so not sure how to get values you ask, but never had this problem until upgrade to 79.0b2 on the 16th so clearly associated with upgrade or it's interaction with other programs (security is Microsoft 10 own virus protection.

Comment 6

[gene smith]

Actually I've been seeing the same problem and thought it was caused by something else I was working on. When the chunk size is changing (an old feature that was accidentally disabled years ago and recently revived, see Bug 1580480), the wrong offset into the message is sometimes calculated for the next chunk. It uses the new chunk size as the offset instead of the previous chunk size. This causes pieces of the message to be left out when chunk size increases, or causes pieces to be fetched again when chunk size goes down. Once the chunk size stabilizes to a constant value the problem goes away.
You can work around this by disabling chunking in the config editor. It's under advanced preferences or options: mail.server.default.fetch_by_chunks to false and restart tb.
This will fix any new messages but you may have to repair your folder to fix existing messages. Another way, instead of right-click/properties/repair folder is to copy the bad messages to another folder and then back to the original folder; do these fixes with chunking disabled.
A patch to fix this shouldn't be too hard. (Famous last words :).)
Thanks for reporting this!

Comment 7

[gene smith]

Attachment: Bug1654995_fix-for-chunk-offset-on-size-variance-v1.patch

This fixes the problem.

Comment 8

[Peter Robinson]

Problem fixed with Gene Smith's cure, thanks.

Comment 9

[Magnus Melin [:mkmelin]]

Comment on attachment 9166104 [details] [diff] [review]
Bug1654995_fix-for-chunk-offset-on-size-variance-v1.patch

Review of attachment 9166104 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks Gene! r=mkmelin

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1da0226308d3
Chunk starting offset wrong after m_chunkSize changes. r=mkmelin

Comment 11

[Jorg K (CEST = GMT+2)]

I assume bug 1580480 caused this?

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to Jorg K (CEST = GMT+2) from comment #11)

I assume bug 1580480 caused this?

Yes. We are going to back that out of the esr and not ship it until these both go through another round of beta.

Comment 13

[Wayne Mery (:wsmwk)]

Comment on attachment 9166104 [details] [diff] [review]
Bug1654995_fix-for-chunk-offset-on-size-variance-v1.patch

Approved for beta - 79.0b3

Comment 14

[Wayne Mery (:wsmwk)]

Rob, this will also need a release note

Comment 15

[Rob Lemley [:rjl]]

Thunderbird 79.0b3:
https://hg.mozilla.org/releases/comm-beta/rev/8223d0c9a40e

Comment 16

[Jorg K (CEST = GMT+2)]

Comment on attachment 9166104 [details] [diff] [review]
Bug1654995_fix-for-chunk-offset-on-size-variance-v1.patch

Can we get this on the way to ESR at some stage together with bug 1580480?

Comment 17

[Wayne Mery (:wsmwk)]

Comment on attachment 9166104 [details] [diff] [review]
Bug1654995_fix-for-chunk-offset-on-size-variance-v1.patch

[Triage Comment]
Approved for esr78

Needs bug 1580480

Comment 18

[Rob Lemley [:rjl]]

Thunderbird 78.2.0:
https://hg.mozilla.org/releases/comm-esr78/rev/80fab2f75fbc