1707178 - Cannot send email messages with Bcc recipients

Status: RESOLVED FIXED

(MailNews Core :: Networking: SMTP, defect)

Description

[Sander]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0

Steps to reproduce:

Send a new mail
Reply on a mail
Forward a mail
...from one email acount out of six. This one (of course the most important one) is not sending anymore after yesterday's update. Receiving is ok. The other 5 accounts (different domains) have the same smtp server and have no problems.

Actual results:

Once I click send, shortly a white window is shown with the message "sending <subject>" (<subject> is the subject), it disappears within one second. And I go back to the composing window.

Expected results:

The mail should have been sended and the composing windows should close.

Comment 1

[Sander]

To try to fix this I tried to make a new Thunderbird account for this email address. Bu that failed to, same bug as this one: https://bugzilla.mozilla.org/show_bug.cgi?id=1706926

Comment 2

[Ping Chen [:rnons] (away)]

Can you please do the following to get some debug logs

1. go to the Config Editor, set both mailnews.send.loglevel and mailnews.smtp.loglevel to All
2. open DevTools and select the Console tab, clear any existing logs
3. send a mail, debug logs should show up in the Console tab
4. copy/paste the logs here

Thanks

Comment 3

[Sander]

1. ok, did that
2. when I click this (also shortcut CTRL SHIFT I) nothing happens...

Comment 4

[Sander]

2. Also tried to restart in debug modus with add-ons off. Not possible to start DevTools. Nothing happens.

Comment 5

[WaltS48 [:walts48]]

I had no problem sending emails and replying to emails when testing the release candidate on Windows 10 and Fedora 33 Workstation.

Just tested forwarding a message from one account to my four other accounts with the addresses in the To: field.

No problem replying to it.

Reporter,

Any addresses in the Cc: and/or Bcc: fields?

Comment 6

[Magnus Melin [:mkmelin]]

You can also open the developer tools through the menu (under Tools).

Comment 7

[Sander]

(In reply to Magnus Melin [:mkmelin] from comment #6)

You can also open the developer tools through the menu (under Tools).

Yes I know that...however I found out that somehow devtools.inspector.remote was on FALSE. I did change that and now the DevTools comes up!

Comment 8

[Sander]

(In reply to WaltS48 [:walts48] from comment #5)

I had no problem sending emails and replying to emails when testing the release candidate on Windows 10 and Fedora 33 Workstation.

Just tested forwarding a message from one account to my four other accounts with the addresses in the To: field.

No problem replying to it.

Reporter,

Any addresses in the Cc: and/or Bcc: fields?

Yes you are right! That was the problem. When there is an address in the BCC it wil not send. Without it it works.
So it gives this error in the log:

GenericSendMessage FAILED: NotReadableError: Could not read file(C:\Users\info\AppData\Local\Temp\nsemail.eml) because it is not UTF-8 encoded

Comment 9

[Sander]

In addition: it's only when the BCC is used. CC works fine.

Comment 10

[Magnus Melin [:mkmelin]]

In that case, I suspect this is a regression from bug 1696659 - https://hg.mozilla.org/comm-central/diff/650df14d8b1f7522fb5e270378d2d057a36c45f3/mailnews/compose/src/MessageSend.jsm. IOUtils.readUTF8 is not very lenient (using raw + then TextDecoder should work)

Comment 11

[Magnus Melin [:mkmelin]]

Though I don't understand why it wouldn't be UTF-8. We don't support sending anything else anymore.

Comment 12

[Sander]

Sorry Magnus, I do not understand. Can I fix it myself?

Comment 13

[Magnus Melin [:mkmelin]]

No, but maybe you have some idea why it wouldn't be in UTF-8?

Comment 14

[Sander]

I have no idea... and: it was clearly no problem, until last week's update...
Can I check some setting somewhere?

Comment 15

[WaltS48 [:walts48]]

I had no problem sending an email with one Bcc: in it from one POP3 account to my other POP3 account and 3 other IMAP accounts.

From: POP3 #1.
To: POP3 #2, IMAP #1
Cc: IMAP #2
Bcc: IMAP #3

Using TB 89.0b1 on Windows 10 and Fedora 33 Workstation.

From the message source:

Subject: Bcc: send test on Windows take 2 after updating to TB 89.0b1
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Comment 17

[Ping Chen [:rnons] (away)]

When the error happens, can you please go to C:\Users\$USER\AppData\Local\Temp and see if nsemail.eml is still there? And if possible, upload the file here.

Comment 18

[Ping Chen [:rnons] (away)]

Never mind about the temp file, seems it is gone. I found a way to reproduce by forwarding a non-UTF8 mail as attachment. Were you doing the same or attaching some files?

Will work on a fix.

Comment 19

[Ping Chen [:rnons] (away)]

Attachment: Bug 1707178 - Fix sending bcc mail with non-UTF8 eml attachments. r=mkmelin

The original encoding of non-UTF8 eml attachments are kept in the final message file, so the final message file is not guaranteed to be UTF-8. This patch changes to use read/write instead of readUTF8/writeUTF8.

Comment 20

[Sander]

(In reply to Ping Chen (:rnons) from comment #18)

Never mind about the temp file, seems it is gone. I found a way to reproduce by forwarding a non-UTF8 mail as attachment. Were you doing the same or attaching some files?

Will work on a fix.

Just plain mails with no attachment...

Comment 21

[Arthur K. (he/him)]

Probably too late to add this but error console throws:

05:47:13.178
Prompter: internal dialogs not available in this context. Falling back to window prompt. Prompter.jsm:1084
set modalType resource://gre/modules/Prompter.jsm:1084
ModalPrompter resource://gre/modules/Prompter.jsm:1040
getPrompt resource://gre/modules/Prompter.jsm:65
CompleteGenericSendMessage chrome://messenger/content/messengercompose/MsgComposeCommands.js:4853
GenericSendMessage chrome://messenger/content/messengercompose/MsgComposeCommands.js:4789
SendMessage chrome://messenger/content/messengercompose/MsgComposeCommands.js:5198
doCommand chrome://messenger/content/messengercompose/MsgComposeCommands.js:995
doCommand chrome://messenger/content/messengercompose/MsgComposeCommands.js:1191
goDoCommand chrome://global/content/globalOverlay.js:101
oncommand chrome://messenger/content/messengercompose/messengercompose.xhtml:1

05:47:13.249
GenericSendMessage FAILED: NotReadableError: Could not read file(C:\TEMP\nsemail.eml) because it is not UTF-8 encoded MsgComposeCommands.js:4861
CompleteGenericSendMessage chrome://messenger/content/messengercompose/MsgComposeCommands.js:4861
AsyncFunctionThrow self-hosted:699

Comment 22

[Arthur K. (he/him)]

So I installed the 89.0b2 candidate and re-tested and sending BCC works. I don't know if your fix made it into this build but just letting you know of my results.

Comment 23

[Arthur K. (he/him)]

Nevermind, false alarm it seems. I tested once more with another account and it still happens. Odd that it succeeded once and the second time it failed. Error console threw the same error about "GenericSendMessage FAILED:" as above. Sorry for the noise.

Comment 24

[clochrua]

Hello

Daily 90.0a1 (27.04.21) (on linux ubuntu 20.10) also cannot send BCC - same symptoms as documented in this report (GenericSendMessage FAILED: NotReadableError: Could not read file(/tmp/nsemail-10.eml) ) except send fails both with and without attachment.

According to your comment 18, Ping, I assume 90.0a2 (28.04.21) will get the UTF8 correction?

If so will update bug report when I've seen 90.0a2.

Comment 26

[Ping Chen [:rnons] (away)]

(In reply to clochrua from comment #24)

According to your comment 18, Ping, I assume 90.0a2 (28.04.21) will get the UTF8 correction?

If so will update bug report when I've seen 90.0a2.

It's not landed yet, hopefully it will be included in today's nightly.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/53c9ee714528
Fix sending bcc mail with non-UTF8 eml attachments. r=mkmelin

Comment 29

[Ping Chen [:rnons] (away)]

Comment on attachment 9218855 [details]
Bug 1707178 - Fix sending bcc mail with non-UTF8 eml attachments. r=mkmelin

[Approval Request Comment]
Regression caused by (bug #): bug 1686636 and bug 1696659
User impact if declined: In some cases, sending bcc mail fails
Testing completed (on c-c, etc.): c-c
Risk to taking this patch (and alternatives if risky): Low, changes only affect sending bcc mail.

Comment 31

[FIlip]

Ok, got same error on my side, but it's not only attachment based. Also the same happen when you use html signature.

Comment 32

[FIlip]

If I delete this part https://pastebin.com/s0fav4YY from my signature, it works fine, but I dont see anything wrong with this code:

Comment 33

[Thomas D. (:thomas8)]

Attachment: HTML signature code triggering the bug per Fllip's comment 31 / comment 32

This is the Fllips's HTML source code from pastebin.

(In reply to FIlip from comment #32)

If I delete this part https://pastebin.com/s0fav4YY from my signature, it works fine, but I dont see anything wrong with this code:

Comment 34

[Thomas D. (:thomas8)]

Current commit message:

Fix sending bcc mail with non-UTF8 eml attachments.

Hi Ping (:rnons), could you clarify what exactly your patch has changed, i.e. the specifics of the subset of messages which will be different after this patch? Is the change really limited to "BCC messages with non-UTF8 file attachments of type .eml", or is it a general change that applies to all BCC messages (with or without attachment, or with UTF8 attachment), or some other subset of messages?

I'm asking because the BCC fail is also reported for messages without attachments... (comment 20, comment 31)

Any idea why a message having HTML of attachment 9219557 [details] in signature would fail, and if your patch fixes that?

Comment 35

[clochrua]

Hello
An update from comment 24:
On TB 90.0a1 (2021-05-01) (64-bit) I can now forward OK to BCC with and without attachments. It will be some days until I can test completely, with address lists from address book, but this test was not working before the update was applied.
With regards to what I posted in comment 24, the problem now seems to be cured.
Thanks Wayne and Ping
Bill

Comment 36

[Alfred Peters [:infofrommozilla]]

(In reply to Thomas D. (:thomas8) from comment #34)

Any idea why a message having HTML of attachment 9219557 [details] in signature would fail, and if your patch fixes that?

The signature contains several  . TB replaces this with the Unicode character U+00A0 ("NO-BREAK SPACE"). That's the 8-bit char.

In the text/plain part this is unavoidable. But TB converts the   also in the HTML part.

Comment 37

[Ping Chen [:rnons] (away)]

(In reply to Thomas D. (:thomas8) from comment #34)

Current commit message:

Fix sending bcc mail with non-UTF8 eml attachments.

Hi Ping (:rnons), could you clarify what exactly your patch has changed, i.e. the specifics of the subset of messages which will be different after this patch? Is the change really limited to "BCC messages with non-UTF8 file attachments of type .eml", or is it a general change that applies to all BCC messages (with or without attachment, or with UTF8 attachment), or some other subset of messages?

I'm asking because the BCC fail is also reported for messages without attachments... (comment 20, comment 31)

Any idea why a message having HTML of attachment 9219557 [details] in signature would fail, and if your patch fixes that?

The commit message was inaccurate, but that's the only way I found to reproduce the problem. The problem was previously we assumed the whole message file was in UTF8, so we incorrectly used readUTF8/writeUTF8. The problems you linked should be also fixed by my patch.

Comment 38

[Wayne Mery (:wsmwk)]

Comment on attachment 9218855 [details]
Bug 1707178 - Fix sending bcc mail with non-UTF8 eml attachments. r=mkmelin

[Triage Comment]
Approved for beta

Comment 39

[Rob Lemley [:rjl]]

Thunderbird 89.0b3:
https://hg.mozilla.org/releases/comm-beta/rev/508aff591f10