1675914 - Complete message fetch occurs with imap parts on demand, no offline store and non-inline attachment, >=77.0b1

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[gene smith]

While working on bug 1673093 I noticed that

• a simple message with, for example, a one line body
• a fairly large not-inline attachment (e.g., a zip file)
• with mail.server.default.mime_parts_on_demand set true
• no offline store

did not result in the imap accesses that I expected to occur, i.e., first fetch just the body and then when the attachment link is opened, fetch just the attachment.

This works as expected with version 60 and with a daily build from 2019-3-31. On all version beginning with the first 68 beta up to the current trunk(<--incorrect, see below) there is unexpected network traffic: The body is fetched and then the whole message (including the attachment) is fetched. Then when the attachment link is opened, the attachment is fetched again.

This is probably a regression that was introduced with the 2019-4-4 commit for bug 1345167: https://hg.mozilla.org/comm-central/rev/875a13f010ee plus several follow-up commits. (this didn't cause this bug).

I reported another regression on bug 1345167 here: bug 1579341. The symptoms are similar but there were other issues and I didn't notice the exact problem I'm seeing now when I tested the fix here: bug 1579341 comment 30. (This is n/a)

Fetch of imap parts on demand is now disabled by default so this extra traffic will probably not be noticed by most users. However, some users run with non-default settings and, especially on slower links and/or huge attachments, will notice the problem.

Comment 1

[Magnus Melin [:mkmelin]]

Personally I wonder if we shouldn't just stop supporting all the nonsensical configurations of "not downloading it all". Mails are pretty small in comparison to all other data people use. I'm not sure what the use case for saving the off 70kBs are when the next second people still go online a single Facebook page not even counting the pics/videos is multi mega byte.

Comment 2

[gene smith]

In general I guess you are right but the example I've been working with, the subject of bug 1673093, has a trivial main email body and a huge attachment, 25M base64 encoded. If a user with no offline store just looks at the email all 25M are fetched when it is opened. Then if they open the attachment the 25M are downloaded again.

I don't think all mail servers allow as quick access as popular and public sites like facebook.

Interesting too, in the current tb code with "fetch on demand" disabled, it still gets temporarily switched back on in a certain case:
https://searchfox.org/comm-central/rev/7cf487b2cc2783ff71b0f2d4c4690af3307561d7/mailnews/imap/src/nsImapService.cpp#2483

So I think a properly functioning "fetch on demand" is still a useful feature for some users. (Of course, if everyone used offline store this issue would not exist.)

Comment 3

[gene smith]

I checked closer and bug 1345167 and its follow-ups didn't cause this bug.

I did some bisecting to see where this bug started. It was OK with 76.0b3 and bad with 77.0b1. Looking closer at daily builds, the behavior was OK on Apr 22 2020 and bad on Apr 24 2020. I couldn't tell about Apr 23 because it crashes; there was a bug reporting bustage. Also, in this range there were problems with the Config Editor not coming up when clicked but probably not related to this bug.)

Here is the info from the daily .txt files in this range:

2020-04-21--095819 (OK - but Config Editor doesn't come up)
https://hg.mozilla.org/comm-central/rev/bf38a1ba950eff2e665d28358157c53ff44cf732
https://hg.mozilla.org/mozilla-central/rev/2fd61eb5c69ce9ac806048a35c7a7a88bf4b9652

2020-04-22--105014 (OK - but Config Editor doesn't come up)
https://hg.mozilla.org/comm-central/rev/542fb0021c53f7d462c07f07419d80664c4f0e33
https://hg.mozilla.org/mozilla-central/rev/d8eecc663784c8463af1d2bc3f91f8078c7e1940

2020-04-23--105100 (Crashes)
https://hg.mozilla.org/comm-central/rev/d010d86cedeb754677886254f697dd775a0127ea
https://hg.mozilla.org/mozilla-central/rev/47426d145e246fa1924fbda83a8ecb0d25a6f606

2020-04-24--173214 (Bad - Config Editor OK I think)
https://hg.mozilla.org/comm-central/rev/274dd8d1e772b19acd32ff1d575a1c5939a0dd5b
https://hg.mozilla.org/mozilla-central/rev/be13cb1076bc1923240b3ec450cf177d2b4ad42b

2020-04-25--165103 (Bad)
https://hg.mozilla.org/comm-central/rev/99b6256ea0bf453e35ca224c3c14b285a274ba52
https://hg.mozilla.org/mozilla-central/rev/d8a8178627c4fab05f1eb46ee40f55db207c7c18

I checked with "hg diff -r XXX -r XXX" the comm-central changes and don't see anything obvious that would cause this bug.

Comment 4

[gene smith]

Attachment: moz.configure.diff (enable openPGP stuff triggers this bug)

I looked again at this and still can't find exactly what in the diffs is causing this bug. However, I did noticed that it occurred when the openPGP feature was enabled in the daily and beta release. If I disable the openPGP stuff in moz.configure (see attachment) and rebuild, the bug no longer occurs and fetch by parts works as it should when it is enabled:

(Correct behavior)
Click message
Fetches just the very short message body
Click/open attachment
Fetch just the 25M attachment

So something about the openPGP is causing this but I don't know what. When PGP enabled:

(Incorrect behavior)
Click message
Fetches the very short message body, then fetches the whole message (25M)
Click/open attachment
Fetches the 25M attachment

In both cases fetch by parts is true. Also, the message is just plain text (not encrypted).
So for lack of a more specific commit, I'm marking this as regressed by bug 1628424

Comment 5

[Wayne Mery (:wsmwk)]

The dependency list for bug 1628424 is small - https://bugzilla.mozilla.org/showdependencytree.cgi?id=1628424&hide_resolved=0
Perhaps https://mzl.la/3pzAy5A will catch what you seek

Comment 6

[gene smith]

Kai, Maybe you can figure this out since I don't know the openpgp related code. While working on another bug (bug 628646) I'm still seeing what I describe in comment 0 above. An imap url occurs and requests a full body fetch of the message even when fetch parts on demand is enabled. When I first wrote up this bug, I noticed that disabling openpgp fixed the problem (see comment 4 above) but I could never find anything in your code that caused it. But now a couple years later I'm still seeing it.

The bug goes away with ESR 91 if I set pref mail.openpgp_enable to false. However, that pref is no longer in 102 and later version and not in daily. But in daily, in the file mail/base/content/msgMail3PaneWindow.js, if I comment out the line initOpenPGPIfEnabled(); it fixes the problem and full message fetch doesn't occur.

So maybe you have some idea where in the openpgp or related code that the full message fetch is being requested and, if so, is it really needed?

This may be related to the requirement that mime_parts_on_demand now defaults to false for encryption (see Bug 1629292). However, a user not using encryption should still be able to set this true and have parts/attachments fetched on demand and not always have the full message fetched.

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to gene smith from comment #2)

So I think a properly functioning "fetch on demand" is still a useful feature for some users. (Of course, if everyone used offline store this issue would not exist.)

Agree it should be supported. It may NOT be the norm, but one of the attractions of Thunderbird is it provides performant features in low bandwith/high latency situations. [Modulo security requirements]

Comment 8

[Kai Engert (:KaiE:)]

Yes, the OpenPGP code enforces full downloading. The reason is, if we fetch only a portion, and attempt to verify or decrypt the message, we would run into a failure, and would have to report in the UI that the message is bad, being unable to decrypt, or having a bad signature. We cannot properly process an OpenPGP message with only a subset available.

If you wanted to enable partial download with OpenPGP, we'd have to invent mechanism that prevents OpenPGP status display and OpenPGP process, whenever only a subset of the message is available. You'd need related UI, to inform the user about this partial/incomplete state. And you'd need UI, e.g. a button, that would allow the user to trigger the full download and full process and status display.

Comment 9

[Magnus Melin [:mkmelin]]

(In reply to Wayne Mery (:wsmwk) from comment #7)

Agree it should be supported.

I'm not so sure mime_parts_on_demand gives actual over all value. That you, in some cases, didn't download/cache a few 100KB (or maybe even a couple of M) isn't too much of a concern for most people. Just opening a web page will usually download more. People aren't on modem speeds anymore. For situations of high latency the user experience of having it all downloaded before hand is probably way better.