1840943 - Message headers missing in message preview for some messages (rarely, and random). Displays in message list.

Status: RESOLVED FIXED

(Thunderbird :: Message Reader UI, defect, P1)

Description

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

+++ This bug was initially created as a clone of Bug #1819916 +++
We made a fix for bug 1819916 but it's not ideal. There's some discussion in that bug and I put some things I discovered in comment 25.

Let's see if we can make a better fix.

Comment 1

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

Attachment: Bug 1840943 - Avoid closing IMAP mock channels until OnStopRequest completes. r=BenC

Removes the gist of https://hg.mozilla.org/comm-central/rev/10277c24518637c05ec21f303d7fad5a324e3c0f
and implements something similar (and some cleaning up) in the problematic IMAP code.

Comment 2

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

Gene, does this handle the problematic messages you had in the other bug okay?

Comment 3

[gene smith]

Applied the patch and still looks good on my large test messages while also showing the original reporter's smaller test message without missing stuff. So looks like a good fix.

Off topic, but when I open one of my large messages (about 100Mb) there is no progress or progress bar graph in the status bar. Initial load via imap takes a while and there is no progress feedback while waiting for the message and attachments to completely appear like there was with 102. Has this been intentionally changed or is it a bug?

Comment 4

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

Yeah, what happens when messages load slowly sure isn't great. It's not intentional, but it's not been a priority either. I can't think of a specific bug number for it.

Comment 5

[gene smith]

OT;
I'm not so concerned about the slow loading (kind of expect it since these are huge messages even when loading from offline store or cache). What I'm saying is the status bar progress feedback is missing. I'll do a search and see if it's been reported and, if not, file a new bug.
Edit: Looks like someone reported it here: Bug 1832149

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/b3523ca25ec5
Avoid closing IMAP mock channels until OnStopRequest completes. r=gds

Comment 7

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

This fix didn't really do the job, and I don't want to clone the bug again for the next patch.

Comment 8

[Chris Andrichak]

Attachment: no-headers-w-attachment.txt

pretty sure i only see this on messages with attachments, like this one. TB117b6. Attachment is a PDF if that matters.
Doesn't matter if you try View -> Headers -> All or Normal.

Comment 9

[gene smith]

I see the randomly missing headers in preview when testing with message in comment 8. It occurs both with and without mbox offline store. Didn't try it with maildir. Tested just with locally built daily 118.0a1.

Observation that may or may not be relevant: When I open very large emails (e.g., 100Mb) from storage (mbox or disk cache) the body appears first and then the header. With older versions (110 or earlier, pre-SN) the header appeared first and then then body. It's not noticeable with small or average size emails.

Comment 10

[Wayne Mery (:wsmwk)]

There are many other reports of this. Linking some examples from https://mzl.la/3YqhoAS, there may be more

Comment 12

[quadronom]

It definitely got worse. In 118.0b2 the problem is now persistent for certain emails for me and it does not recover anymore.
Hopefully this will lead you to the cause now.
Tip for everybody else struggling: If you are forwarding the email you can open and download the attachments at least from the mail editor window...

Comment 13

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #10)

There are many other reports of this. Linking some examples from https://mzl.la/3YqhoAS, there may be more

I think the last two "See Also:" listed above are the same as my observation in comment 9 above:

When I open very large emails (e.g., 100Mb) from storage (mbox or disk cache) the body appears first and
then the header. With older versions (110 or earlier, pre-SN) the header appeared first and then then body.
It's not noticeable with small or average size emails.

Also from comment 9:

I see the randomly missing headers in preview when testing with message in comment 8. It occurs both with and without mbox offline store. Didn't try it with maildir. Tested just with locally built daily 118.0a1.

When the header is missing, the list of attachments at the bottom is also missing which is consistent with other reports.

Note to self: Test messages are in imap://gene@localhost/111

Comment 14

[gene smith]

I wrote in comment #3:

Applied the patch and still looks good on my large test messages while also showing the original reporter's smaller test message without missing stuff. So looks like a good fix.

Maybe I didn't test it enough but problem still occurs with reporter supplied test messages (the one attached by Chris Andrichak above and the one attached by pmuiruri over at bug 1819916).
I reversed the patch for this bug, https://phabricator.services.mozilla.com/D182428?download=true, and tested again and don't see the problem at all after opening the messages many times from disk cache. I re-applied the patch and I see the problem again (header and attachments often don't appear when message opened, but not always, when toggling between the two messages).

So it appears that the patch originally landed at bug 1819916 still works better even if it's not "ideal".

Comment 15

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

(In reply to gene smith from comment #14)

So it appears that the patch originally landed at bug 1819916 still works better even if it's not "ideal".

I've long thought that but I've been too busy to deal with it. I'm going to back out the patch from this bug until I have a better fix.

Comment 16

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/comm-central/rev/ecf0198f0dab
Backed out changeset b3523ca25ec5 until a better fix can be found.

Comment 17

[gene smith]

Ok, sound like the best thing to do at this time.
I know you are busy but you might want to take a look at this related bug over at bug 1851871. It deals with the fact that the message header in preview doesn't appear until after the whole body is completely displayed.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1b43b03bab01
Fix up test that changed between the patch landing and back out. rs=me

Comment 19

[Jaap van der Leer]

I have a similar observation. An email form a mailing list takes very long before the correct header is displayed, when external content is allowed, ca 30 seconds. The body displays instantaneously. In the mean time, the previous shown header is shown. If external content is not allowed, ther header is drawn immediately.
I do have supporting files to clarify, which I seem not be able to upload.

Comment 20

[gene smith]

(In reply to Jaap van der Leer from comment #19)

I have a similar observation. An email form a mailing list takes very long before the correct header is displayed, when external content is allowed, ca 30 seconds. The body displays instantaneously. In the mean time, the previous shown header is shown. If external content is not allowed, ther header is drawn immediately.
I do have supporting files to clarify, which I seem not be able to upload.

Thanks.
Under "Attachments" above there is a "Attach New File" button that should allow you to attach/upload the file(s). If it's an email file you can name it with an ending .eml. I suspect there is a size limit but don't know what it is. If you can't attach, you can try to email it to me using my profile address.

Comment 21

[Jaap van der Leer]

Attachment: 2023-09-08_17-55-41 Incorrect header afte allowing external content, switching to other email and back.png

This is a mailing-list entry, and 2 screenshots. The first with no external content allowed, the second with external content allowed. After about 30 seconds, the header of the second preview becomes as the same as the first one is. So, it is more of a performance problem than a functional problem, as far as I can see.

Comment 22

[Jaap van der Leer]

Attachment: testmail.eml

Comment 23

[Jaap van der Leer]

Attachment: 2023-09-08_17-50-45 correct message header.png

Comment 24

[gene smith]

Jaap, What you are seeing is definitely the same as bug 1851871. I should have had you attach the files over there. I'll add more info there.

Comment 26

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

Attachment: Bug 1840943 - Remove requests from their load group only after parsing finishes. r=BenC,gds

This old code in nsImapCacheStreamListener::OnStopRequest removes the request from the load
group as soon as reading it from the cache has finished. Removing it from the load group causes
the STATE_STOP event to happen and the UI tries to display the headers and attachments.

However reading the file from the cache isn't what gets us the header and attachment data. We pass
the file via a stream converter to the MIME parsing code and that reads the data. In the failure
cases, the parsing code hasn't finished before STATE_STOP so the UI has no data to use.

This wasn't a problem before we rewrote the UI, because back then we got the data directly from
the MIME parser via a series of hacks that don't work any more. So I think this bug has existed
the whole time, it's just never been a problem before.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c83ce313ced3
Remove requests from their load group only after parsing finishes. r=BenC,gds

Comment 28

[Wayne Mery (:wsmwk)]

Comment on attachment 9353031 [details]
Bug 1840943 - Remove requests from their load group only after parsing finishes. r=BenC,gds

[Triage Comment]
Approved for beta

Comment 29

[Daniel Darnell [:dandarnell]]

Thunderbird 118.0b5:
https://hg.mozilla.org/releases/comm-beta/rev/7ccd98b8d687

Comment 30

[Wayne Mery (:wsmwk)]

All,

If at all possible today or Sunday, please post your results here after testing patches using either:

• beta download - includes instructions
• nightly 119.0a1 build download - beta link above has detailed testing instructions

Comment 31

[Wayne Mery (:wsmwk)]

118.0b5 helped Bug 1841539 Specific mail doesn't show message headers and attachments (TypeError: can't access property "timing", window.performance is null)

Comment 33

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #30)

All,

If at all possible today or Sunday, please post your results here after testing patches using either:

• beta download - includes instructions
• nightly 119.0a1 build download - beta link above has detailed testing instructions

I'm only seeing the fix being effective in the 119.0a1 version. The beta link doesn't seem to fix the problem and , if anything, makes it worse.

Also, the topicbox link say this will be fixed: "very [s]low loading of message headers, i.e. body loads long before subject, contact, date". Actually, the fix in daily/nightly is only for sometimes missing headers and attachment list. Slow or delayed showing of message headers is another bug that still needs to be fixed, bug 1851871.

Comment 34

[Wayne Mery (:wsmwk)]

Yeah, I oversimplified my messaging. In part because I wanted people to report if the still see a problem.

Comment 35

[gene smith]

I'm only seeing the fix being effective in the 119.0a1 version. The beta link doesn't seem to fix the problem and , if anything, makes it worse.

Somehow this got left out of the beta 115.0b5 merge:
https://searchfox.org/comm-central/rev/933a56297d54b40a9fab147e93eefe226e413f77/mail/base/content/msgHdrView.js#550-552
This got put back into daily with the backout of comment 16. Not sure why this wasn't seen by beta. (I don't know how betas are put together.)
Apparently that delay/timeout is needed to completely fix the bug since when I remove those lines from my daily I see the problem again.

Edit: I thought I tested this with the changes from Bug #1819916 completely removed, but apparently I didn't since the SetTimeout(resolve) change from Bug #1819916 is needed to fix the bug.

Comment 36

[Geoff Lankow (:darktrojan) [PTO, back 9 Feb]]

Urgh, that's messy. I forgot to have the original patch from this bug backed out on beta.

Although, I was still intending to remove that piece, and developed most of the current patch without it (the removal is in a patch for another bug I haven't finished yet). Still, I'm not seeing any of the usual problems on central (with or without the original fix) or beta.

Comment 37

[Wayne Mery (:wsmwk)]

Comment on attachment 9353031 [details]
Bug 1840943 - Remove requests from their load group only after parsing finishes. r=BenC,gds

(I want to be certain we don't miss this for 115.3.0)

Comment 38

[Wayne Mery (:wsmwk)]

Comment on attachment 9353031 [details]
Bug 1840943 - Remove requests from their load group only after parsing finishes. r=BenC,gds

[Triage Comment]
Approved for esr115 (here goes...)

Comment 39

[Wayne Mery (:wsmwk)]

correcting that 118 beta is not fixed (or partially unfixed)

Comment 40

[Daniel Darnell [:dandarnell]]

Thunderbird 115.3.0:
https://hg.mozilla.org/releases/comm-esr115/rev/df315718979f