1788102 - Thunderbird incorrectly renders articles with \r\n line endings

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: NNTP, defect)

Description

[Kamil Jonca]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0

Steps to reproduce:

1. in user.js file I turned on: user_pref("mailnews.nntp.jsmodule", true);
2. then I tried to view some newsgroup articles. It is worth noting that, these articles are internal for organization and they mostly in html, sometimes huge (serveral MB), sometimes with attachments.

Version 102.1.2, first offered to channel users on August 9, 2022

Actual results:

1. most of articles were unreadable with first reading (thunderbird shows them as "raw html") then when try to view second time - thesy show correctly, but some was stil rendered as raw html.
2. some articles have no body displayed
3. cannot do "repair folder" - message about other action on folder

Expected results:

messges should rendered correctyl. (As a point of reference we can assume old functionality)

Comment 1

[Magnus Melin [:mkmelin]]

Do you have any public place we could reproduce this?

Comment 2

[Kamil Jonca]

Unfortunately not. I will try to prepare some such articles and push them. But it probably will take some time.

Comment 3

[Kamil Jonca]

Well, I am completely confused now.
I tried to prepare some test articles.
I save article (with gnus) in eml format
removed Message-id and then try to post with mailpost.
reposted article was properly rendered with thunderbird (I thnk)
I tried to compare both artcles but they are pretty much the same (except of course injection info, msgid, xref, etc)
most notable difference is that old have header:
"Mime-Version: 1.0"
and second one
"MIME-Version: 1.0"
but some other articles with "Mime-Version: 1.0" are also properly rendered.
Stranger and stranger.

Comment 4

[Kamil Jonca]

Well finally I thing I got cause (I do not know if this is bug in thunderbird)
I wrote simple perl script which connect to my local news server and tried to get raw article.
(This is sometimes callled "wire format" - \r\n newlines)
when I took a look it seems that article itself has "wire format"

for example (this is excerpt from xxd)

00000ba0: 7320 6973 2061 206d 756c 7469 2d70 6172  s is a multi-par
00000bb0: 7420 6d65 7373 6167 6520 696e 204d 494d  t message in MIM
00000bc0: 4520 666f 726d 6174 2e0d 0d0a 2d2d 3d3d  E format....--==
00000bd0: 3d3d 3d3d 3d3d 3d3d 3d3d 3d3d 3d34 3235  =============425
00000be0: 3731 3739 3934 3030 3132 3934 3731 3838  7179940012947188
00000bf0: 3d3d 0d0d 0a43 6f6e 7465 6e74 2d54 7970  ==...Content-Typ
00000c00: 653a 206d 756c 7469 7061 7274 2f61 6c74  e: multipart/alt
00000c10: 6572 6e61 7469 7665 3b0d 0d0a 2062 6f75  ernative;... bou
00000c20: 6e64 6172 793d 222d 2d2d 2d2d 2d2d 2d2d  ndary="---------
00000c30: 2d2d 2d30 3230 3130 3830 3930 3130 3130  ---0201080901010
00000c40: 3930 3530 3730 3430 3530 3222 0d0d 0a0d  90507040502"....

please take look at double "0d" in some places
When I removed 0d's at end of lines and repost article it was readable by thunderbird.
next it seemed that removing "0d" at mime-related lines (definitions, boundaries, etc) was enough.

So I do not know if this is bug in thunderbird (probably not), but accoring to https://en.wikipedia.org/wiki/Robustness_principle :) it would be VERY useful if thunderbird can render these articles. At least as option in config.

Comment 5

[Kamil Jonca]

Attachment: get-msg.pl

Comment 6

[Kamil Jonca]

So in fact there are three issues:

1. lack of rendering/wrong rendering messages with "\r\n" line endings
2. big articles are initially displayed as "raw" and only the second time are rendered correctly
3. cannot do "repair folder" - (probably where are big messages) .
   Creating example is quite difficult as most of free (=without login) servers prohibit creating messages with attachments
   Will try.

Comment 7

[max]

Kamil, do you have (or can you create) a message that causes a problem and that you can add here as an attachment?

I saw problems with some messages as well. Comparison of full messages might reveal if it is the same problem or different ones.

I have not dug deep enough into related RFCs and I would expect that additional \x0d should be harmless and should belong to content of the MIME part of the message.

Comment 8

[Kamil Jonca]

Attachment: 1.raw

Comment 9

[Kamil Jonca]

Attachment: 2.raw

Comment 10

[Kamil Jonca]

Comment on attachment 9307694 [details]
1.raw

This article is rendered properly

Comment 11

[Kamil Jonca]

Comment on attachment 9307695 [details]
2.raw

This article is not rendered properly.

Comment 12

[Kamil Jonca]

I attached 2 (1.raw and 2.raw)files. They differ only with line endings in body.
Both of them can be injected to local hierarchy by command like:

formail -I "Message-ID:" < ~/tmp/2.raw|formail -a message-id: |/usr/lib/news/bin/mailpost  kjonca.test

Comment 13

[Kamil Jonca]

I changed title because during preparing I found problem with line endings, and this is easily reproductible.
Original problem has to wait when someone (or me) will be able to create reproductible case.

Comment 14

[Wayne Mery (:wsmwk)]

Reporter, do you still see this issue when using the latest version?
What information can you add to clarify the steps to reproduce this issue?

Comment 15

[Kamil Jonca]

At 102.9.1 (64-bit) (this is the latest on debian sid) bug still exists.
Ad. reproduction: I think that https://bugzilla.mozilla.org/show_bug.cgi?id=1788102#c12 is rather clear - do you need additional info?

Comment 16

[Hartmut Welpmann [:welpy-cw]]

After inserting an \r at the end of the first (From) line of each raw file and copying them to Mail/Local Folders, they were accessible as folders in Thunderbird. Running mozregression revealed that this was actually fixed in Bug 1719121. So also 2.raw should work on ESR 128 and newer releases.