2063740 - POP3 download hangs indefinitely on messages containing a non-standard "Content-Length" header (AliExpress/Amazon SES notification emails)

Status: UNCONFIRMED

(MailNews Core :: Networking: POP, defect)

Description

[Damir Sel]

Attachment: aliexpress-pop3-hang-redacted.txt

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36

Steps to reproduce:

1. Configure a POP3 account (tested with Yahoo Mail, pop.mail.yahoo.com:995, OAuth2 auth)
2. Receive/have present in the mailbox an email from notice.aliexpress.com (relayed via Amazon SES) that includes a "Content-Length" header in addition to standard MIME headers
3. Click "Get Messages" in Thunderbird

Actual results:

Thunderbird hangs indefinitely on "Downloading message X of Y" for the specific message. No further messages download. The account becomes stuck until Thunderbird is restarted, at which point it hangs again on the same message when retried. Error Console shows:

pop3.server1.X: NetworkTimeoutError: a Network error occurred Pop3Client.sys.mjs:381:18
pop3.server1.X: SecurityError info: Pop3Client.sys.mjs:441:20
pop3.server1.X: SecurityError cert chain: legacy.pop.mail.yahoo.com; ... Pop3Client.sys.mjs:446:22

Deleting the specific message from the mail server (via webmail) allows the remaining queue to download normally.

Expected results:

Thunderbird should ignore any non-standard "Content-Length" header and rely solely on the POP3 dot-stuffed terminator (\r\n.\r\n) for message framing, per RFC 1939. A malformed or non-conformant message should not be able to deadlock the entire POP3 download queue — it should time out and move to the next message, or download the message correctly regardless of the extraneous header.

Comment 1

[Wayne Mery (:wsmwk)]

This problem didn't exist for you in version 152?

Comment 2

[Damir Sel]

(In reply to Wayne Mery (:wsmwk) from comment #1)

This problem didn't exist for you in version 152?

The problematic email actually came from Aliexpress, and I didn't ordered from them for quite some time, so I can't tell if I had same problem with ver 152. But this is the first time I experience such a glitch with TB.

Comment 3

[gene smith]

Are you sure the problem was caused by "content-length" header? I can't find any place in mail code where tb cares about that header so it should just be ignored when fetched from pop3 server.

I edited as new your attached email and saved it as a draft and "content-length" is present. I tried sending it from 2 accounts but when received "content-length" was stripped off at 2 pop3 accounts. So unable to cause a received message to have that header and can't say for sure that tb doesn't choke on it.

Comment 4

[gene smith]

The only other recent issue I can find regarding content-length is this: Bug 2055384. Don't think this is related unless you are trying use an mbox file imported from Mutt.

Comment 5

[cm99]

I have been able to reproduce this problem on demand and captured both Thunderbird POP logging and the TCP traffic.

Environment

Yahoo POP server: pop.mail.yahoo.com:995
Yahoo server identifies itself as jpop-0.1
Authentication: XOAUTH2

Reproduction

I have a specific Ticketmaster message that consistently causes the failure. If I move this message into the Yahoo Inbox using Yahoo Webmail and then retrieve mail with Thunderbird, POP retrieval hangs on that message. If I move the same message out of the Inbox using Webmail and retrieve again, the POP session completes normally.

With the problem message in the Inbox, Yahoo reports:

STAT
+OK 309 64289969

LIST
...
307 152431

Thunderbird then requests:

RETR 307
+OK 152431 octets.

Yahoo begins sending the message, but the RETR response never completes. Thunderbird eventually reports:

NetworkTimeoutError: a Network error occurred
Done with status=0x804b000e
Connection closed.

I captured this same transaction with tcpdump. In the failed TCP/TLS connection, the Yahoo server sent only 78,053 bytes of TCP payload total before becoming idle. That byte count includes the TLS handshake and all previous POP responses (CAPA, authentication, STAT, LIST, UIDL, etc.), as well as the partial RETR.

After approximately 100 seconds with no further substantial data from Yahoo, Thunderbird timed out and closed the connection.

Therefore Yahoo cannot have transmitted the complete 152,431-byte RETR response. This does not appear to be a case where Yahoo transmitted the complete message and Thunderbird subsequently failed to parse it; the server stops transmitting during the RETR.

Controlled test

I then moved only the offending Ticketmaster message out of the Yahoo Inbox and immediately repeated Get Messages.

Yahoo then reported:

STAT
+OK 308 64137538

The difference in mailbox size is exactly 152,431 bytes, matching the size Yahoo had reported for the offending message.

No new messages needed downloading during this second test. Thunderbird completed the POP session normally:

QUIT
+OK Server signing off.

The control TCP connection completed in about 1.8 seconds without the long server-side stall.

I have also encountered the same behavior with PetSmart messages, so it is not specific to Ticketmaster.

I previously examined the raw source of failing messages because Content-Length had been suggested as a possible trigger. However, I also have successfully downloaded messages containing Content-Length, including a successful Ticketmaster message from the same sender/infrastructure with essentially the same MIME structure and similar size. Therefore the presence of Content-Length by itself does not appear sufficient to trigger the failure.

At this point the evidence appears to indicate that Yahoo's jpop-0.1 server begins a RETR successfully but, for certain messages, stops transmitting partway through the message and leaves the connection open. Thunderbird then waits until its network timeout.

I can provide the Thunderbird POP debug log and/or sanitized packet-capture statistics if useful. I have retained the offending message and can reproduce the failure by moving it back into the Yahoo Inbox.

Comment 6

[Magnus Melin [:mkmelin]]

Yes, please attach logs to this bug. And message.

Comment 7

[cm99]

Need to sanitize the logs and message first. Will work on that.

Comment 8

[Max Emig [:maxe]]

Could you try to reproduce the failure with any other POP3-client or does that work? I checked the code, we do not care for Content-Length in POP3 at all.

Comment 9

[gene smith]

I dropped the "redacted aliexpress" message attached at comment 0 into my imap yahoo inbox and it retrieves successfully into my matching pop3 yahoo inbox with no hang.
Yes, the log and the message source should be helpful.
Another thing you might try, if offending message in comment 5 above is not private, maybe you could forward it to my yahoo test account at [e-mail] and I will fetch it using my yahoo pop account and see what happens. I don't know if there has to more than one message in the fetch to see a problem. So you might send a couple short message, then forward the "bad" message and then then send a couple more short messages.

Comment 10

[cm99]

Attachment: cm99-bugzilla_evidence.zip

Sanitized version of the test email and logs for issue in downloading tickermaster email from Yahoo Pop3

Comment 11

[cm99]

The apple mail client on my iPhone was able to download the messages. Posted the sanitized version of the email and logs.

Comment 12

[gene smith]

The "failing" file, when dropped into my imap yahoo inbox is retrieved successfully at the corresponding pop3 yahoo account. Does it fail for you if you do a similar experiment using the failing file?
I can't really decode the .pcap file since I don't have the keys. So not sure exactly how many bytes are retrieved before the 99.999 second timeout occurs. Can you tell at your end how many bytes or what percent of the bytes are sent back from the server in response to the RETR?

Comment 13

[Worcester12345]

(In reply to Max Emig [:maxe] from comment #8)

Could you try to reproduce the failure with any other POP3-client or does that work? I checked the code, we do not care for Content-Length in POP3 at all.

Sounds like it is time to start caring, or convince "the other side" to do something.

Comment 14

[Max Emig [:maxe]]

(In reply to Worcester12345 from comment #13)

(In reply to Max Emig [:maxe] from comment #8)

Could you try to reproduce the failure with any other POP3-client or does that work? I checked the code, we do not care for Content-Length in POP3 at all.

Sounds like it is time to start caring, or convince "the other side" to do something.

Well, users can switch to IMAP and for now we still have no trace on how a working vs non-working POP session looks like and couldn't reproduce.

Comment 15

[gene smith]

(In reply to cm99 from comment #11)

The apple mail client on my iPhone was able to download the messages.

Is iPhone app using pop3? If it uses imap it may not tell us anything.

I asked:

Can you tell at your end how many bytes or what percent of the bytes are sent back from the server in response to the RETR?

I looked closer at your encrypted .pcap and I see that about 75,000 bytes are sent by yahoo after RETR is sent before the stream stops and TB times out the connection. This pretty much matches your comment above where the total size is supposed to be 152,431 but you only see 78,053 sent. So for some reason yahoo is only sending about 1/2 of the message.

This appears to be a yahoo issue. To decrypt the .pcap file I think I will need to receive a key log file from you and a new .pcap file from the same session.
p/s: I wrote this yesterday and forgot to send it.

Comment 16

[cm99]

(In reply to gene smith from comment #15)

(In reply to cm99 from comment #11)

The apple mail client on my iPhone was able to download the messages.

Is iPhone app using pop3? If it uses imap it may not tell us anything.

I asked:

Can you tell at your end how many bytes or what percent of the bytes are sent back from the server in response to the RETR?

I looked closer at your encrypted .pcap and I see that about 75,000 bytes are sent by yahoo after RETR is sent before the stream stops and TB times out the connection. This pretty much matches your comment above where the total size is supposed to be 152,431 but you only see 78,053 sent. So for some reason yahoo is only sending about 1/2 of the message.

This appears to be a yahoo issue. To decrypt the .pcap file I think I will need to receive a key log file from you and a new .pcap file from the same session.
p/s: I wrote this yesterday and forgot to send it.

I confirmed that the iPhone accesses the Yahoo account using IMAP. Thunderbird on the Mac is using POP3.

I still have the original offending Ticketmaster message stored in another Yahoo folder. I will work on the suggested experiment of importing a copy of the original message into the Yahoo Inbox via IMAP and then seeing whether that newly imported copy can be retrieved successfully via POP3. After that I can also move the original known-failing Yahoo message back into the Inbox to confirm that it still fails.

I can also make another packet capture of the failure along with an SSL/TLS key log if Thunderbird can be configured to generate one. Since I can reproduce the failure, obtaining a matching PCAP and key log should be possible.

Comment 17

[gene smith]

I can also make another packet capture of the failure along with an SSL/TLS key log if Thunderbird can be configured to generate one.

Yes it can. I generate the key log by running tb from command line with environment variable SSLKEYLOGFILE defined like this:
SSLKEYLOGFILE=~/.sslkeylogfile <optional path to tb>/thunderbird
And in wireshark under Edit->Preferences...->Protocols->TLS browse to or enter path to .sslkeylogfile in box labeled (Pre) Master-Secret log filename, e.g., /home/gene/.sslkeylogfile.