Closed Bug 1143569 Opened 9 years ago Closed 9 years ago

User-agent error when posting to NNTP due to RFC5536 violation of Tb (user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...")

Categories

(MailNews Core :: MIME, defect)

defect
Not set
major

Tracking

(thunderbird38+ fixed, thunderbird39 fixed, thunderbird40 fixed)

RESOLVED FIXED
Thunderbird 40.0
Tracking Status
thunderbird38 + fixed
thunderbird39 --- fixed
thunderbird40 --- fixed

People

(Reporter: mozbugs, Assigned: mkmelin)

References

Details

(Keywords: regression, Whiteboard: [regression:TB??])

Attachments

(1 file, 1 obsolete file)

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36

Steps to reproduce:

Tried to post a message to news.jrsoftware.org (Inno Setup Forums).


Actual results:

Alert: A News (NNTP) error occurred: 437 No colon-space in "user-agent:" header
Sending of message failed.


Expected results:

Sending of message should have succeeded.
This was working earlier today, before I upgraded Thunderbird.  (I don't recall what the prior version I was using was.)
On an almost completely unrelated note, if I select-all & copy the text in the editor window, it pastes (anywhere, including another editor window) without newlines.  This also seems like a new bug.
Severity: normal → blocker
Sorry, I didn't mean to post that last comment, I ended up creating a new bug for it.  (Didn't realise that the Save Changes button at the top of Bugzilla also submitted comments.)

Anyway, what I meant to post instead was that I confirmed that restarting Thunderbird did not fix it, but downgrading to Thunderbird 31.5.0 (current release channel) and then re-sending the saved Draft of the unsendable post completed successfully.

I tagged this as "blocker" because it renders Thunderbird completely unable to post NNTP -- at least to that server.  I haven't tested if other servers are more tolerant.
This is user-agent: header by Tb 39.0a1 at mozilla.test of news.mozilla.org ( [CRLF] = 0x0D0A, [SP] = 0x20
> user-agent:[CRLF]
> [SP] Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1[CRLF]
If rule of header folding of RFC5322(RC2822), this is absolutely correct and it's identical to "user-agent:[SP]Mozilla...".

Rule of RFC 822 is used, or News server doesn't expect folding before first SP after ":"?

Possible workaround. Shorter user agent
  general.useragent.override = Mozilla/5.0 (Android; rv:39.0) Gecko/20100101 Tb/39.0a1
Length in ascii is same in 39.0a1 and 31.5.0. "user-agent:" vs. "User-Agent:" war in Tb?
> user-agent:  Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1
> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.5.0
This must be why I had problems posting to a news group. (Didn't get any error message though.)
Severity: blocker → major
Component: Untriaged → MIME
Keywords: regression
OS: Windows 7 → All
Product: Thunderbird → MailNews Core
Hardware: x86 → All
Summary: User-agent error when posting to NNTP → User-agent error when posting to NNTP(user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...")
Status: UNCONFIRMED → NEW
Ever confirmed: true
FYI.
"user-agent: instead of ordina; User-Agent:" is bug 1138220.
See Also: → 1138220
Whiteboard: [regression:TB??]
(In reply to WADA from comment #3)
> This is user-agent: header by Tb 39.0a1 at mozilla.test of news.mozilla.org
> ( [CRLF] = 0x0D0A, [SP] = 0x20
> > user-agent:[CRLF]
> > [SP] Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1[CRLF]
> If rule of header folding of RFC5322(RC2822), this is absolutely correct and
> it's identical to "user-agent:[SP]Mozilla...".
> 
> Rule of RFC 822 is used, or News server doesn't expect folding before first
> SP after ":"?
> 
> Possible workaround. Shorter user agent
>   general.useragent.override = Mozilla/5.0 (Android; rv:39.0) Gecko/20100101
> Tb/39.0a1

Looking at RFC 5322 myself, it says that the field-body may be folded, but that the header name and colon are not part of the field-body.  To me this means that the first space after the colon is not a valid folding point.

Also for reasons of sanity (and because some agents truncate at folding points rather than unfolding) it should *always* be the case that headers are written as long as possible before folding.  It should therefore always be an error to fold at the first position.

> user-agent:[CRLF]
> [SP] Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Thunderbird/39.0a1[CRLF]

This should ideally be:

> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101[CRLF]
> [SP]Thunderbird/39.0a1[CRLF]

in my opinion, if the order of tokens must be maintained.  (However it would be better if the Thunderbird token came first, given possible truncation at the folding point.)
(In reply to Gavin Lambert from comment #7)
> To me this means that the first space after the colon is not a valid folding point.

From perspective of rule in RFC5322(RFC2822), I believe "space after :, which is first character of field-body" is FWS.
It may be different in RFC822, and old news servers may use RFC822, and obsolete format in old mails is permitted.
I also think "first letter should be capital", "single space should be placed after :", "folding shouldn't be done at space just after :", when Tb generates header.
Header in NNTP was defined by rfc5536.
https://tools.ietf.org/html/rfc5536#section-3.2.13
3.2.13. User-Agent
   user-agent      =  "User-Agent:" SP 1*product [CFWS] CRLF
   product         =  [CFWS] token [ [CFWS] "/" product-version ]
   product-version =  [CFWS] token
3.2.14. Xref
   xref            =  "Xref:" SP *WSP server-name
                      1*( FWS location ) *WSP CRLF

In NNTP, "SP just after :" is mandatory, and "folding before it" is perhaps never permitted.
Apparently, this bug is RFC violation by Tb. RFC5322 folding is wrongly applied to header of NNTP by Tb.
Summary: User-agent error when posting to NNTP(user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...") → User-agent error when posting to NNTP due to RFC5536 violation of Tb (user-agent header is folded just after user-agent:, "user-agent:[CRLF][SP]Mozilla...")
I think there were two problems here. 
 1) It should never be a preferred breakpoint directly after the header name and colon.
 2) when calculating if it would go over the soft margin the length header name (and colon) wasn't accounted so it would actually wrap when the value was over 78ch, not when the line was over 78ch
Assignee: nobody → mkmelin+mozilla
Status: NEW → ASSIGNED
Attachment #8581292 - Flags: review?(Pidgeot18)
(In reply to Magnus Melin from comment #10)
>  2) when calculating if it would go over the soft margin the length header
> name (and colon) wasn't accounted so it would actually wrap when the value
> was over 78ch, not when the line was over 78ch

When following useragent.override,
   user_pref("general.useragent.override", "Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd eeee/39.0a1");
folding occurrs.
   user-agent:[CRLF]
   [SP]Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd eeee/39.0a1[CRLF]
If shorter useragent.override, 
   user_pref("general.useragent.override", "Mozilla/5.0 Gecko/20100101 Thunderbird/39.0a1");
folding doesn't occur.
   user-agent::[SP]Mozilla/5.0 Gecko/20100101 Thunderbird/39.0a1[CRLF]
Byte length is caluculated on "UTF16 binary" then folding is executed?
Folding at 78 bytes is needed in message header? It's for message body with format=flowed or wrraplength=72 or 76, isn't it?
(In reply to WADA from comment #11)
> Byte length is caluculated on "UTF16 binary" then folding is executed?
> Folding at 78 bytes is needed in message header? It's for message body with
> format=flowed or wrraplength=72 or 76, isn't it?

I'm not sure what you're asking but length is calculated on the string length in JS. 
I don't change the margins in this patch.
(In reply to Magnus Melin from comment #12)
> length is calculated on the string length in JS. I don't change the margins in this patch.

user-agent: Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd/39.0a12
Header length = 78 byres(80 bytes including CRLF) => not folded

user-agent: Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd/39.0a123
Header length = 79 byres(81 bytes including CRLF) => folded at ":" like next.
user-agent:
 Mozilla/5.0 Gecko/20100101 Thunderbird aaaa bbbb cccc dddd/39.0a123

Sorry for my confusion.

Why message header folding at 80 bytes(including CRLF) is mandatory?

Following is first line of an actual Received: header of bug mail from B.M.O. Length = 94 bytes(excluding CRLF).
Received: from jobqueue1.bugs.scl3.mozilla.com (jobqueue1.bugs.scl3.mozilla.com [10.22.82.41])

http://tools.ietf.org/html/rfc5322#section-2.2.3 "Long Header Fields" never says "folding at 78 bytes is MUST nor SHOULD".
http://tools.ietf.org/html/rfc5322#section-2.3 "Body" says "folding at 78 bytes is SHOULD", but it's for message body instead of field-body in message header.
(In reply to WADA from comment #13)
> Why message header folding at 80 bytes(including CRLF) is mandatory?

It's a soft limit. Probably not strictly necessary, but useful the same way as for body. Say someone inspects headers and their editor wraps the text where it shouldn't, leading to confusion.
Comment on attachment 8581292 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

Review of attachment 8581292 [details] [diff] [review]:
-----------------------------------------------------------------

Again, I'd like to see tests added here.
Attachment #8581292 - Flags: review?(Pidgeot18) → review-
With tests.

Re the test_EncodeMimePartIIStr_UTF8.js change:
"Subject: MXR now displays links to Github log & Blame for Gaia/Rust/Servo" is 73 long
Attachment #8581292 - Attachment is obsolete: true
Attachment #8591336 - Flags: review?(Pidgeot18)
Comment on attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

Review of attachment 8591336 [details] [diff] [review]:
-----------------------------------------------------------------

I'd rather see the tests added to JSMime tests, but I can tackle that myself later.
Attachment #8591336 - Flags: review?(Pidgeot18) → review+
https://hg.mozilla.org/comm-central/rev/107a975915e9 -> FIXED
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 40.0
Comment on attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):
Attachment #8591336 - Flags: approval-comm-beta?
Attachment #8591336 - Flags: approval-comm-aurora?
Comment on attachment 8591336 [details] [diff] [review]
bug1143569_user-agent-wrapping.patch

http://hg.mozilla.org/releases/comm-aurora/rev/6c54d11a4ed5
http://hg.mozilla.org/releases/comm-beta/rev/31d8992f40e9
Attachment #8591336 - Flags: approval-comm-beta?
Attachment #8591336 - Flags: approval-comm-beta+
Attachment #8591336 - Flags: approval-comm-aurora?
Attachment #8591336 - Flags: approval-comm-aurora+
Didn't fix Bug 1151448 in Thunderbird 38 or 41, as I discovered yesterday, when using 38.0b6 to reply to a post in m.support.thunderbird, with a cross post and Followup-To m.general on news.mozilla.org. First time I recall trying that since posting the bug report. Sorry, I forgot the problem existed.

Also tried 38.0.1 RC (build1), 38.0b4, 38.0b5 and 41.0a1, with a test user and clean profiles for each test on Kubuntu 15.04.
You need to log in before you can comment on or make changes to this bug.