Open Bug 1435903 Opened 4 years ago Updated 1 year ago
E-mail corrupted by Yahoo, Verizon, Bell
South, etc . servers . Was: Use `CTE: 8bit` only when the server is advertising the 8BITMIME support
Initially this but was filed since Thunderbird was sending 8bit encoded mail to Yahoo, Verizon, BellSouth, etc. servers. Whilst that was incorrect and could still be fixed, the situation has changed. Those servers now advertise 8BITMIME capability, see comment #28, yet can't handle anything but UTF-8. Second complaint at Yahoo in Oct. 2018: https://forums.yahoo.net/t5/Sending/Yahoo-SMTP-server-corrupting-e-mail-in-windwos-1252-despite/td-p/556011 (Edit 12-21-2019 by gene smith: This link is no longer accessible. However, an excellent write-up regarding the issues can be found here: https://wiki.mozilla.org/User:Jorgk/8-bit_bytes_and_e-mail_corruption_at_Verizon,_Yahoo,_etc.#All_you_ever_need_to_know_about_8bit_bytes_and_corrupted_e-mail_passing_through_the_servers_of_Verizon.2C_Yahoo.2C_Bellsouth.2C_AT.26T.2C_Sbcglobal_and_others Workaround for affected users: 1) Always use UTF-8 for outgoing e-mail Tools > Options, Display, Formatting tab, Advanced button, Fonts & Encodings, Text Encoding, Outgoing Mail. Also reply in UTF-8: Tools > Options, Display, Formatting tab, Advanced button, Fonts & Encodings, Text Encoding, click: When possible, use the default text encoding in replies. 2) Set pref mail.strictly_mime to true Tools > Options, Advanced, General tab, Config Editor, paste "mail.strictly_mime". You need to use option 2 if UTF-8 sending through Yahoo and friends doesn't work either, which has been the case at times.
+++ This bug was initially created as a clone of Bug #1427636 +++ After a long discussion in bug 1427636 we noticed that Yahoo SMTP servers answer EHLO with 250-8 BITMIME (amongst other things). The correct reply is 8BITMIME. We also noticed that those servers can process UTF-8 when sent with CTE: 8bit but not windows-1252. In fact, all non-ASCII characters get replaced with the Unicode replacement character leading to a garbled display of ï¿½. When the server doesn't advertise 8bit support correctly, we shouldn't send 8bit.
Inspired by bug 1427636 comment #43 I sent two messages via the Yahoo SMTP server, both with ä or similar in windows-1252, once with CTE: base64 and the other with CTE: QP. Both were received correctly. So reading bug 1032302 comment #1 I'm confused now. I'm also confused be the discussion in bug 1379096 where it was requested to set mail.strictly_mime to true by default, which would also fix this bug here.
I think the following is fair to say: - Yahoo's servers send a non-compliant reply "8 BITMIME". - It appears that they try to advertise "8BITMIME" which is confirmed by handling UTF-8 with CTE: 8bit correctly. - We ship out "8BITMIME", and the servers don't issue an error, like the one in bug 1379096. So I don't think TB's behaviour is unreasonable, but the behaviour of the Yahoo servers is unreasonable. 100% strictly speaking, we should ignore the non-compliant response and not send 8bit, but that's what the preference is for. So what's the way forward here?
https://www.limilabs.com/blog/yahoo-smtp-8-bitmime-bug reckons it's a Yahoo bug to advertise the (faulty) 8bit capability incorrectly.
(In reply to Jorg K (GMT+1) from comment #4) > So what's the way forward here? Not a clear one, I'm afraid... (1) We could remove whitespace chars from the advertisement tokens, thus "8 BITMIME" == "8BITMIME"; if Yahoo works correctly for Win1252 with that, we are fine, reasonable effort. However, that's adding an uncertainty by interpreting non-standard responses as standard-compliant ones. (2) We could downgrade 8bit messages to quoted-printable if "8BITMIME" isn't advertised, regardless of the strictly_mime pref: * sounds reasonable at the first glimpse, but * how do we figure out what to send where? - only send qp without 8BITMIME if no 8BITMIME was advertised, 8bit to those which do? - or, sent qp to /all/ if /any/ of the involved MSAs doesn't advertise 8BITMIME? - what will happen if the originating MSA accepts 8bit but an MTA down the road doesn't? * either way, problem is that - while strictly_mime is known in advance - it's only known at the time of sending the message (i.e., not when forming the message body) whether or not we can send a message in 8bit encoding. In the worst case, we need to either prepare two versions of the same message (one qp, the other 8bit), of have two passes over the recipients to figure out the "flavor" to be sent. * thus, implementation sounds expensive to satisfy a fringe case. (3) Keep as is (=WONTFIX) and accept that some people have to manually toggle the qp pref manually. Definitely, I wouldn't like setting strictly_mime to true by default; that's introducing a substantial overhead for encodings primarily or mostly based on 8bit characters (i.e., everything not using latin alphabets, and languages using latin characters with diacritics) for those 99.9% of the cases where 8bit /does/ work.
(In reply to rsx11m from comment #6) > - what will happen if the originating MSA accepts 8bit but an MTA down the road doesn't? See bug 1379096 comment #10 for this case, the proposed (1) or (2) solutions wouldn't help here.
Re. (1): So now since we don't detect "8 BITMIME", we also don't send "8BITMIME"? So let me try that.
OK, detecting "8 BITMIME" and sending "8BITMIME" or "8 BITMIME" doesn't make a difference. BTW, I forgot to mention (from 1427636 comment #37) "... the server changes the mails? This is an absolute no-go" in my comment #4. :-(
I finally got around to reading up on the matter. So bug 1032302 (https://hg.mozilla.org/comm-central/rev/eda5c42f3b4cac18b2334b7f4fc86d194f797571) really only ships out "8BITMIME" is the server advertised it and mail.strictly_mime isn't set. It doesn't change anything in message processing. So in the case of Yahoo, we don't detect the capability and thus don't send out "8BITMIME", but we ship 8bit anyway. Setting mail.strictly_mime has the effect that the message will be prepared in QP: https://dxr.mozilla.org/comm-central/rev/63f09d10244cd7100cb5955a17993160fa180937/mailnews/compose/src/nsMsgSend.cpp#3080 mime_use_quoted_printable_p = strictly_mime; So having mail.strictly_mime at true by default doesn't appear attractive. Since we prepare the message *before* we contact the receiving SMTP server, it would be a lot of change to rebuild the message at that point, see comment #2 "implementation sounds expensive", also see 1032302 comment #2 "possibly downgrading seems an overkill". My apologies to those who working in this area before and already knew all that ;-) Just for clarification of comment #2: > sent qp to /all/ if /any/ of the involved MSAs doesn't advertise 8BITMIME? If I understand it correctly, TB only talks to one server, even if sending to multiple recipients. So as far as I can tell, there is no "multiple" problem. So we could extensively recode the message if it turns out that the server might not accept it. > what will happen if the originating MSA accepts 8bit but an MTA down the road doesn't That's not our problem to fix. We talk to the outgoing SMTP server and only have to satisfy it. Factually, this bug will be a WONTFIX, since we don't have resources to redo the compose/send pipeline, even if it were decided that it was desirable. It would be much more useful to get global mail provider Yahoo to 1) advertise its server properties correctly 2) fulfil its own advertising 3) not mess with content of messages. I know that Masatoshi-san disagrees.
(In reply to rsx11m from comment #7) > (In reply to rsx11m from comment #6) > > - what will happen if the originating MSA accepts 8bit but an MTA down the road doesn't? > > See bug 1379096 comment #10 for this case, the proposed (1) or (2) solutions > wouldn't help here. That should not be a problem. The server could then change the transport encoding to qp or base64. But he should of course not change the charset.
(In reply to rsx11m from comment #6) > (2) We could downgrade 8bit messages to quoted-printable if "8BITMIME" isn't > advertised, regardless > of the strictly_mime pref: > * either way, problem is that - while strictly_mime is known in advance > - it's only known at the > time of sending the message (i.e., not when forming the message body) > whether or not we can send We could query the server for "8BITMIME" when setting up the account and store the result server-related in the prefs.js. Each time we send a new e-mail, we update the flag.
But there are also servers that support 8-bit without any problem, without announcing "8BITMIME". German "GMX" for example.
(In reply to Alfred Peters from comment #12) > We could query the server for "8BITMIME" when setting up the account and > store the result server-related in the prefs.js. > Each time we send a new e-mail, we update the flag. Nice idea. Meanwhile I posted this: https://forums.yahoo.net/t5/Sending/Yahoo-SMTP-server-advertising-8bit-capability-incorrectly-and/m-p/449275
Good morning. I am a user that posted this bug. Today the behavior changed. I am now getting strings of "?" instead of the other funky characters. I checked on the post over at yahoo forums noted above and they show it as solved although there is a post after that disputes that.
It is not surprising at all. They can change their behavior about 8-bit bytes as they like at any time because they say they don't support 8-bit bytes. Do not send 8-bit bytes unless the server explicitly says so, period.
(In reply to Masatoshi Kimura [:emk] from comment #16) > It is not surprising at all. They can change their behavior about 8-bit > bytes as they like at any time because they say they don't support 8-bit > bytes. Do not send 8-bit bytes unless the server explicitly says so, period. To be clear, this comment is not against the reporter, but against Thunderbird developers. You (users) can freely compose your messages using 8-bit bytes. Thunderbird should encode 8-bit bytes as appropreate, but it does not at the moment. This is a bug of Thunderbird.
(In reply to kricks from comment #15) > Good morning. I am a user that posted this bug. Today the behavior > changed. I am now getting strings of "?" instead of the other funky > characters. I checked on the post over at yahoo forums noted above and they > show it as solved although there is a post after that disputes that. I tested this yesterday and I got the "funky" characters. I'm the one who reported it on the Yahoo forum, and yes, it's obviously not solved since now you get ???. Masatoshi-san, as was explained a few times, by the time we get the SMTP response, it's too late to recode the message. The work-around is to set the preference mail.strictly_mime to true.
As of today the problem became much worse, and the responded messages are on the boundary of the complete ineligibility. I hope that this is a temporary state, and somebody got finally a chance to attend to this problem.
Workaround: Set pref mail.strictly_mime to true. Otherwise complain to Yahoo: https://forums.yahoo.net/t5/Sending/Yahoo-SMTP-server-advertising-8bit-capability-incorrectly-and/ There's also an interesting solution idea in comment #12 here. I've just tried Yahoo's servers again and they actually fixed the server response now: telnet smtp.mail.yahoo.com 25 EHLO Yahoo gives: 250-smtp408.mail.ir2.yahoo.com Hello Yahoo [22.214.171.124]) 250-PIPELINING 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-SIZE 41697280 250 STARTTLS Before it was "8 BITMIME". So Thunderbird will now ship 8bit to the server and if that doesn't work, it's 100% Yahoo's fault.
Sorry, wrong link: https://forums.yahoo.net/t5/Sending/Yahoo-SMTP-server-advertising-8bit-capability-incorrectly-and/m-p/509636#M50602 Tested again at Yahoo and still not working after three months :-(
User Story: (updated)
Summary: Use `CTE: 8bit` only when the server is advertising the 8BITMIME support → E-mail corrupted by Yahoo, Verizon, BellSouth, etc. servers. Was: Use `CTE: 8bit` only when the server is advertising the 8BITMIME support
User Story: (updated)
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.