User-agent error when posting to NNTP due to RFC5536 violation of Tb (user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...")

RESOLVED FIXED in Thunderbird 40.0

Status

MailNews Core
MIME
--
major
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Gavin Lambert, Assigned: Magnus Melin)

Tracking

({regression})

Thunderbird 40.0
regression
Bug Flags:
in-testsuite +

Thunderbird Tracking Flags

(thunderbird38+ fixed, thunderbird39 fixed, thunderbird40 fixed)

Details

(Whiteboard: [regression:TB??])

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

3 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36

Steps to reproduce:

Tried to post a message to news.jrsoftware.org (Inno Setup Forums).


Actual results:

Alert: A News (NNTP) error occurred: 437 No colon-space in "user-agent:" header
Sending of message failed.


Expected results:

Sending of message should have succeeded.
This was working earlier today, before I upgraded Thunderbird.  (I don't recall what the prior version I was using was.)
(Reporter)

Comment 1

3 years ago
On an almost completely unrelated note, if I select-all & copy the text in the editor window, it pastes (anywhere, including another editor window) without newlines.  This also seems like a new bug.
Severity: normal → blocker
(Reporter)

Comment 2

3 years ago
Sorry, I didn't mean to post that last comment, I ended up creating a new bug for it.  (Didn't realise that the Save Changes button at the top of Bugzilla also submitted comments.)

Anyway, what I meant to post instead was that I confirmed that restarting Thunderbird did not fix it, but downgrading to Thunderbird 31.5.0 (current release channel) and then re-sending the saved Draft of the unsendable post completed successfully.

I tagged this as "blocker" because it renders Thunderbird completely unable to post NNTP -- at least to that server.  I haven't tested if other servers are more tolerant.
This is user-agent: header by Tb 39.0a1 at mozilla.test of news.mozilla.org ( [CRLF] = 0x0D0A, [SP] = 0x20
> user-agent:[CRLF]
> [SP] Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1[CRLF]
If rule of header folding of RFC5322(RC2822), this is absolutely correct and it's identical to "user-agent:[SP]Mozilla...".

Rule of RFC 822 is used, or News server doesn't expect folding before first SP after ":"?

Possible workaround. Shorter user agent
  general.useragent.override = Mozilla/5.0 (Android; rv:39.0) Gecko/20100101 Tb/39.0a1
Length in ascii is same in 39.0a1 and 31.5.0. "user-agent:" vs. "User-Agent:" war in Tb?
> user-agent:  Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1
> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0
(Assignee)

Comment 5

3 years ago
This must be why I had problems posting to a news group. (Didn't get any error message though.)
Severity: blocker → major
Component: Untriaged → MIME
Keywords: regression
OS: Windows 7 → All
Product: Thunderbird → MailNews Core
Hardware: x86 → All
Summary: User-agent error when posting to NNTP → User-agent error when posting to NNTP(user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...")
(Assignee)

Updated

3 years ago
Status: UNCONFIRMED → NEW
tracking-thunderbird38: --- → +
Ever confirmed: true
FYI.
"user-agent: instead of ordina; User-Agent:" is bug 1138220.
See Also: → bug 1138220

Updated

3 years ago
Keywords: regressionwindow-wanted
Whiteboard: [regression:TB??]
(Reporter)

Comment 7

3 years ago
(In reply to WADA from comment #3)
> This is user-agent: header by Tb 39.0a1 at mozilla.test of news.mozilla.org
> ( [CRLF] = 0x0D0A, [SP] = 0x20
> > user-agent:[CRLF]
> > [SP] Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1[CRLF]
> If rule of header folding of RFC5322(RC2822), this is absolutely correct and
> it's identical to "user-agent:[SP]Mozilla...".
> 
> Rule of RFC 822 is used, or News server doesn't expect folding before first
> SP after ":"?
> 
> Possible workaround. Shorter user agent
>   general.useragent.override = Mozilla/5.0 (Android; rv:39.0) Gecko/20100101
> Tb/39.0a1

Looking at RFC 5322 myself, it says that the field-body may be folded, but that the header name and colon are not part of the field-body.  To me this means that the first space after the colon is not a valid folding point.

Also for reasons of sanity (and because some agents truncate at folding points rather than unfolding) it should *always* be the case that headers are written as long as possible before folding.  It should therefore always be an error to fold at the first position.

> user-agent:[CRLF]
> [SP] Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1[CRLF]

This should ideally be:

> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101[CRLF]
> [SP]Thunderbird/39.0a1[CRLF]

in my opinion, if the order of tokens must be maintained.  (However it would be better if the Thunderbird token came first, given possible truncation at the folding point.)
(In reply to Gavin Lambert from comment #7)
> To me this means that the first space after the colon is not a valid folding point.

From perspective of rule in RFC5322(RFC2822), I believe "space after :, which is first character of field-body" is FWS.
It may be different in RFC822, and old news servers may use RFC822, and obsolete format in old mails is permitted.
I also think "first letter should be capital", "single space should be placed after :", "folding shouldn't be done at space just after :", when Tb generates header.
Header in NNTP was defined by rfc5536.
https://tools.ietf.org/html/rfc5536#section-3.2.13
3.2.13. User-Agent
   user-agent      =  "User-Agent:" SP 1*product [CFWS] CRLF
   product         =  [CFWS] token [ [CFWS] "/" product-version ]
   product-version =  [CFWS] token
3.2.14. Xref
   xref            =  "Xref:" SP *WSP server-name
                      1*( FWS location ) *WSP CRLF

In NNTP, "SP just after :" is mandatory, and "folding before it" is perhaps never permitted.
Apparently, this bug is RFC violation by Tb. RFC5322 folding is wrongly applied to header of NNTP by Tb.
Summary: User-agent error when posting to NNTP(user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...") → User-agent error when posting to NNTP due to RFC5536 violation of Tb (user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...")
(Assignee)

Updated

3 years ago
Blocks: 959309
Keywords: regressionwindow-wanted
(Assignee)

Comment 10

3 years ago
Created attachment 8581292 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

I think there were two problems here. 
 1) It should never be a preferred breakpoint directly after the header name and colon.
 2) when calculating if it would go over the soft margin the length header name (and colon) wasn't accounted so it would actually wrap when the value was over 78ch, not when the line was over 78ch
Assignee: nobody → mkmelin+mozilla
Status: NEW → ASSIGNED
Attachment #8581292 - Flags: review?(Pidgeot18)
(In reply to Magnus Melin from comment #10)
>  2) when calculating if it would go over the soft margin the length header
> name (and colon) wasn't accounted so it would actually wrap when the value
> was over 78ch, not when the line was over 78ch

When following useragent.override,
   user_pref("general.useragent.override", "Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd eeee/39.0a1");
folding occurrs.
   user-agent:[CRLF]
   [SP]Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd eeee/39.0a1[CRLF]
If shorter useragent.override, 
   user_pref("general.useragent.override", "Mozilla/5.0 Gecko/20100101 Thunderbird/39.0a1");
folding doesn't occur.
   user-agent::[SP]Mozilla/5.0 Gecko/20100101 Thunderbird/39.0a1[CRLF]
Byte length is caluculated on "UTF16 binary" then folding is executed?
Folding at 78 bytes is needed in message header? It's for message body with format=flowed or wrraplength=72 or 76, isn't it?
(Assignee)

Comment 12

3 years ago
(In reply to WADA from comment #11)
> Byte length is caluculated on "UTF16 binary" then folding is executed?
> Folding at 78 bytes is needed in message header? It's for message body with
> format=flowed or wrraplength=72 or 76, isn't it?

I'm not sure what you're asking but length is calculated on the string length in JS. 
I don't change the margins in this patch.
(In reply to Magnus Melin from comment #12)
> length is calculated on the string length in JS. I don't change the margins in this patch.

user-agent: Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd/39.0a12
Header length = 78 byres(80 bytes including CRLF) => not folded

user-agent: Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd/39.0a123
Header length = 79 byres(81 bytes including CRLF) => folded at ":" like next.
user-agent:
 Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd/39.0a123

Sorry for my confusion.

Why message header folding at 80 bytes(including CRLF) is mandatory?

Following is first line of an actual Received: header of bug mail from B.M.O. Length = 94 bytes(excluding CRLF).
Received: from jobqueue1.bugs.scl3.mozilla.com (jobqueue1.bugs.scl3.mozilla.com [10.22.82.41])

http://tools.ietf.org/html/rfc5322#section-2.2.3 "Long Header Fields" never says "folding at 78 bytes is MUST nor SHOULD".
http://tools.ietf.org/html/rfc5322#section-2.3 "Body" says "folding at 78 bytes is SHOULD", but it's for message body instead of field-body in message header.
(Assignee)

Comment 14

3 years ago
(In reply to WADA from comment #13)
> Why message header folding at 80 bytes(including CRLF) is mandatory?

It's a soft limit. Probably not strictly necessary, but useful the same way as for body. Say someone inspects headers and their editor wraps the text where it shouldn't, leading to confusion.
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1150792
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1151448
Comment on attachment 8581292 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

Review of attachment 8581292 [details] [diff] [review]:
-----------------------------------------------------------------

Again, I'd like to see tests added here.
Attachment #8581292 - Flags: review?(Pidgeot18) → review-
(Assignee)

Comment 18

3 years ago
Created attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

With tests.

Re the test_EncodeMimePartIIStr_UTF8.js change:
"Subject: MXR now displays links to Github log & Blame for Gaia/Rust/Servo" is 73 long
Attachment #8581292 - Attachment is obsolete: true
Attachment #8591336 - Flags: review?(Pidgeot18)
Comment on attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

Review of attachment 8591336 [details] [diff] [review]:
-----------------------------------------------------------------

I'd rather see the tests added to JSMime tests, but I can tackle that myself later.
Attachment #8591336 - Flags: review?(Pidgeot18) → review+
(Assignee)

Comment 20

3 years ago
https://hg.mozilla.org/comm-central/rev/107a975915e9 -> FIXED
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
status-thunderbird38: --- → affected
status-thunderbird39: --- → affected
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 40.0

Comment 21

3 years ago
Comment on attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):
Attachment #8591336 - Flags: approval-comm-beta?
Attachment #8591336 - Flags: approval-comm-aurora?

Comment 22

3 years ago
Comment on attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

http://hg.mozilla.org/releases/comm-aurora/rev/6c54d11a4ed5
http://hg.mozilla.org/releases/comm-beta/rev/31d8992f40e9
Attachment #8591336 - Flags: approval-comm-beta?
Attachment #8591336 - Flags: approval-comm-beta+
Attachment #8591336 - Flags: approval-comm-aurora?
Attachment #8591336 - Flags: approval-comm-aurora+

Updated

3 years ago
status-thunderbird38: affected → fixed
status-thunderbird39: affected → fixed
status-thunderbird40: --- → fixed

Comment 23

3 years ago
Didn't fix Bug 1151448 in Thunderbird 38 or 41, as I discovered yesterday, when using 38.0b6 to reply to a post in m.support.thunderbird, with a cross post and Followup-To m.general on news.mozilla.org. First time I recall trying that since posting the bug report. Sorry, I forgot the problem existed.

Also tried 38.0.1 RC (build1), 38.0b4, 38.0b5 and 41.0a1, with a test user and clean profiles for each test on Kubuntu 15.04.
You need to log in before you can comment on or make changes to this bug.