1888790 - subject is all question marks and email sometimes is corrupt because of it (pop3)

Status: RESOLVED FIXED

(Thunderbird :: Message Reader UI, defect)

Description

[chris.fogel]

Steps to reproduce:

This is 125.0b2 version.

I did nothing. Certain emails come in and for some reason appear to have a carriage return inserted in the subject line (when viewing raw data file). When there is a carriage return in the Subject line, it breaks all the rest of the headers and sometimes the email as a whole.

I have been fixing it by closing Thunderbird, opening the mail file, removing the carriage return and saving. But I'm not sure if it's an actual issue coming from the sender, or Thunderbird is not handling the carriage return properly on receipt.

It appears randomly, mostly from zendesk emails, so I have not been able to email myself and forcefully replicate.

Actual results:

Subject:

Expected results:

Subject: A really long subject line here that should not have wrapped in the header

Comment 1

[chris.fogel]

Attachment: thunderbirdsubject.jpg

The characters did not print in the ticket, so I have attached a sample of the subject as it appears in the UI

Comment 2

[chris.fogel]

Attachment: thunderbirdrealsubject.jpg

And here is a copy of the real subject from the raw file

Comment 3

[Francesco]

Looks like folder corruption, bug 1872849, there is no reason why ASCII text should display with the <?> character. Also see bug 1881761.

Comment 4

[chris.fogel]

Thank you for the pointers. I have made the modifications and repaired the 3 inboxes that were receiving the emails with the issue. I will monitor this week, I'm sure I'll get another ticket from zendesk that should trigger it.

Comment 5

[chris.fogel]

These items did not seem to fix the issue. Certain new emails from zendesk come in and still get the <?> subject line....

If I either a) repair folder again or b) edit the raw file and delete the carriage return in the subject line, it corrects just the ones already arrived.

But the next email that comes in from that source, just <?> again...

Comment 6

[Francesco]

Are you using POP or IMAP?

Comment 7

[chris.fogel]

I am using POP.

Comment 8

[Francesco]

Not really good news. If I understand it correctly, bug 1872849 is about IMAP. Can you paste the header lines which are corrupted, that is the carriage return in the Subejct: header.

Comment 9

[chris.fogel]

Oh, interestingly, this one corrupted and doesn't have a carriage return...

Subject: [Update] 501678: New UI Preview | Doesn't Auto-Refresh View

Comment 10

[chris.fogel]

Attachment: thunderbirdheader.txt

Attached is the full exact headers from the raw file...

Comment 11

[Magnus Melin [:mkmelin]]

(In reply to Francesco from comment #8)

If I understand it correctly, bug 1872849 is about IMAP.

So far yes, but since we don't yet know the exact problem, it's possible POP3 could be affected as well.

Comment 12

[Ben Campbell]

(In reply to chris.fogel from comment #5)

These items did not seem to fix the issue. Certain new emails from zendesk come in and still get the <?> subject line....
If I either a) repair folder again or b) edit the raw file and delete the carriage return in the subject line, it corrects just the ones already arrived.
But the next email that comes in from that source, just <?> again...

That sounds like something different to Bug 1872849 - it seems to me that the mbox file is actually OK in this case, given that you can use "repair folder" to fix it.
From what I can tell, the incoming emails are valid. It's OK for headers to be broken up over multiple lines, as long as the continued lines begin with whitespace (which does seem to be the case here, looking at bug 1888790 comment #2).

Bug 1872849 seems to be more to do with message writes to the mbox file being truncated or interleaved, which "repair folder" would not help.

My suspicion would be that the problem is in the header parsing code rather than the mbox storage code.

Comment 13

[Wayne Mery (:wsmwk)]

This is 125.0b2 version.

So you would have been on 125.0b1 for 7-10 days.
Would you say you definitely did not see this problem when using 125.0b1 and earlier?

(removing relationship to bug 1872849, because that problem started months ago)

Comment 14

[chris.fogel]

I have seen it earlier, at least a month or two if not more, but never took the time to investigate because at first I thought it was 1 specific ticket that was the issue and not Thunderbird, but then recently I started seeing other tickets from other zendesk systems come in...and decided to look. I have been on beta track for years, so it's not a recent move there...

Comment 15

[Ramona Gavrilescu]

Hello,

I have managed to find a regression range for this issue.

It would seem the last good build was from 2023-11-25 and the first bad build was from 2023-12-04.

The bisection returned: ```2024-05-10T11:00:13.666000: INFO : Narrowed integration regression window from [249bf489, f46382a0] (3 builds) to [c25819f5, f46382a0] (2 builds) (~1 steps left)
2024-05-10T11:00:13.685000: DEBUG : Starting merge handling...
2024-05-10T11:00:13.686000: DEBUG : Using url: https://hg.mozilla.org/comm-central/json-pushes?changeset=f46382a0d7a9145e49fc8c096df4bc8c5351e14e&full=1
2024-05-10T11:00:13.686000: DEBUG : redo: attempt 1/3
2024-05-10T11:00:13.686000: DEBUG : redo: retry: calling _default_get with args: ('https://hg.mozilla.org/comm-central/json-pushes?changeset=f46382a0d7a9145e49fc8c096df4bc8c5351e14e&full=1',), kwargs: {}, attempt #1
2024-05-10T11:00:13.694000: DEBUG : urllib3.connectionpool: Resetting dropped connection: hg.mozilla.org
2024-05-10T11:00:14.532000: DEBUG : urllib3.connectionpool: https://hg.mozilla.org:443 "GET /comm-central/json-pushes?changeset=f46382a0d7a9145e49fc8c096df4bc8c5351e14e&full=1 HTTP/1.1" 200 None
2024-05-10T11:00:14.575000: DEBUG : Found commit message:
Bug 1719121 - Part 2/2, Update tests for mbox refactor. r=darktrojan

Differential Revision: https://phabricator.services.mozilla.com/D190227

2024-05-10T11:00:14.575000: DEBUG : Did not find a branch, checking all integration branches
2024-05-10T11:00:14.581000: INFO : The bisection is done.
2024-05-10T11:00:14.582000: INFO : Stopped```

If the information provided above are not sufficient here are the push log from the second to last build.

Comment 16

[Francesco]

Based on the regression range, it's bug 1890230, folder corruption. Does the message display correctly if you repair the folder?

Comment 17

[Ramona Gavrilescu]

Yes after the repair the message is displayed correctly.

Comment 18

[Ben Campbell]

I can replicate this bug (finally!)
Still haven't nailed down the exact replication steps, but I found a test message from another bug that causes it:

https://bugzilla.mozilla.org/show_bug.cgi?id=1880867#c3

1. load that message into my local pop3 server (I have one other tiny test message in the inbox too)
2. start TB with a fresh profile
3. enter the connection details for my local pop3 server
4. click on Inbox to see the list of messages
5. see that that message has a borked subject in the GUI (looks fine in the mbox file)

Comment 19

[gene smith]

I see it too with that message (pointed to in comment 18) when putting it into a not-fresh profile for POP3 server. On first POP3 get new messages it crashed with this console message:
[Parent 717225, Main Thread] ###!!! ASSERTION: Expected 4 hex digits for flags.: 'MsgIsHex(mozstatus->value, 4)', file /home/gene/mozilla2/comm/mailnews/local/src/nsParseMailbox.cpp:1167
After that it was OK except for the bad UTF8 char indicator on each subject char: <?> <?> ...
FWIW, it looks fine in the Inbox for IMAP for the same server (dovecot account accessed via IMAP and POP3 for test purposes).
Repair also fixes the message in POP3 inbox.

Comment 20

[Ben Campbell]

OK, it's the message header parser (nsParseNewMailState) getting tripped up by an unescaped "From " line in the body of the message, and thinking a new message has started.
Should be an easy fix, but the message header parser is pretty convoluted so I just need to pick my way through it carefully.
Doesn't affect IMAP - that uses the header parser in a different way. Of course. Sigh.

More justification for rewriting the header parsing code. See Bug 1876407.
POP3 is a particularly annoying case, as the message filtering is all mixed in with the parsing code, whereas imap has it separated out.

Comment 21

[Ben Campbell]

Attachment: Bug 1888790 - Fix nsParseNewMailState to ignore "From " lines in message body. r=mkmelin

There was leftover code in nsParseNewMailState from the everything-is-an-mbox days.
This meant messages lines beginning with "From " in the body would flip it back out
into header-parsing mode, causing database fields to be filled with rubbish.
This particularly affected POP3 mail. IMAP uses a different class for header parsing.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5dd17b198727
Fix nsParseNewMailState to ignore "From " lines in message body. r=mkmelin