968270 - Thunderbird no longer automatically BCC's myself when replying to myself (unless it's the actual sent mail that has the original bcc header set)

Status: VERIFIED FIXED

(MailNews Core :: Composition, defect)

Description

[Luke deGruchy]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36

Steps to reproduce:

1) Configure in Account Settings > {my account} > When Sending messages, automatically: Bcc these email address: {my email address}

2) Find an email in my inbox/folder/smart folder that I previously sent and reply to it.  




Actual results:

The other recipients on the email are in To: fields but my email 

This only applies to messages that I sent, not to messages I received from anyone else, where my compose window DOES correctly contain myself as BCC'd.

This is *only* and issue in Thunderbird 24.3.0.  When I switched back to 24.2.0 I was able to produce the correct behaviour.


Expected results:

As in Thunderbird 24.2.0, my email address is BCC'd automatically when I hit reply.

Comment 1

[rsx11m]

Works for me when replying to a message I've sent myself when using "Reply" only; using "Reply All" adds all other recipients but omits myself as Bcc.

Reproduced with TB 24.3.0 and a recent SM 2.25a2 build on Windows 7, hence appears to be a cross-platform MailNews Core bug.

Tentatively confirming as a likely regression, don't see anything recent filed on this.

Comment 2

[rsx11m]

Interestingly, today's SM 2.24 release isn't affected, thus whatever caused the regression landed on -esr24 but not -beta during the last cycle.

Comment 3

[rsx11m]

My first guess would have been bug 917231, but this checked in too early already not to be picked up by previous releases. Adding Magnus as CC anyway in case he has an idea where this may be coming from.

Certainly irritating for people using BCC-to-self as replacement for the save-in-Sent function, though it still works fine for messages received from others.

Comment 4

[Magnus Melin [:mkmelin]]

Yeah bug 933555 landed for 24.3.0. 
http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#2592 kind of assumed you reply from the sent folder. There is no problem if you reply to a copy of the sent mail, since then the orignal bcc header will be used. 

The behavior now is to preserve what was sent. Suppose the user originally removed the auto-bcc, on the followup supposedly he'd want the same. (He could also have added more bcc addresses.) 

Haven't tested it yet but I think not setting the bcc if empty would fix it. Though then we can't preserve the case where he manually removed the auto-bcc for the original mail.

Comment 5

[rsx11m]

There may be a special case with Gmail anyway, which is moving a copy into Sent Mail on its own, thus won't have a BCC header either.

I think that treating "use default BCC if no BCC present in message" is probably the best that can be done as a trade-off, given that bcc-to-Self is a valid use case for default behavior (on a side note, it's actually encouraged by the UI given that the Cc and Bcc fields in Copies & Folders default to your own address when the boxes are checked for the first time).

Using "not setting the BCC if empty" for other folders than Sent only might restrict the effect, but then a prior Bcc header may be transported to other folders if a message is moved out of the Sent folder for some reason.

Comment 6

[Luke deGruchy]

My opinion is that Thunderbird should always add the BCC that the user has set, even if it duplicates the other fields.  Alternatively, it could work as Magnus suggested, which is to always set the BCC to the default if it's empty.

Comment 7

[rsx11m]

(In reply to Luke deGruchy from comment #0)
> 1) Configure in Account Settings > {my account} > When Sending messages,
> automatically: Bcc these email address: {my email address}

This just bit me today. Not unexpectedly per explanation in comment #4, this not only applies for the bcc-to-myself case but auto-bcc in general to /any/ blind default recipient. So this is a quite unexpected change of behavior that those are silently dropped now and needs to be revisited, preferably within the current release cycle.

Comment 9

[Magnus Melin [:mkmelin]]

Attachment: proposed fix

Comment 10

[Chin]

(In reply to Magnus Melin from comment #4)
> Yeah bug 933555 landed for 24.3.0. 
> http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.
> cpp#2592 kind of assumed you reply from the sent folder. There is no problem
> if you reply to a copy of the sent mail, since then the orignal bcc header
> will be used. 
> 
> The behavior now is to preserve what was sent. Suppose the user originally
> removed the auto-bcc, on the followup supposedly he'd want the same. (He
> could also have added more bcc addresses.) 
> 
> Haven't tested it yet but I think not setting the bcc if empty would fix it.
> Though then we can't preserve the case where he manually removed the
> auto-bcc for the original mail.


I would like to add my experience with this bug. 

When I reply using a copy of the email from my inbox (the copy created by the self BCC setting), it did not work too.

Comment 11

[Magnus Melin [:mkmelin]]

We don't really look at which folder it is, just the presence of the "Bcc:" header matters.

Comment 13

[smithtom947]

(In reply to Luke deGruchy from comment #6)
> My opinion is that Thunderbird should always add the BCC that the user has
> set, even if it duplicates the other fields.  Alternatively, it could work
> as Magnus suggested, which is to always set the BCC to the default if it's
> empty.
I agree. If you've set an auto CC or auto BCC, it should do just that, not
auto CC/BCC except on alternate Tuesdays when Joe is sending it. There is
nothing about that behavior that depends on who is sending the message,
where it's being sent from, or who it's being sent to.

ALSO, there seems to be some confusion about Bcc in the first place. If a
Bcc recipient is passed into an SMTP transport, the rules are to remove
the Bcc addressee from the recipient list and leave the recipient address
only as the envelope address. Any other behavior could be dangerous to your
career or friendships. If you want to see Bcc's in your mail folders, file
the outgoing message directly, don't mail it to yourself.

Comment 14

[Magnus Melin [:mkmelin]]

The patch always adds the auto-BCC, except for when it is a reply to self AND Bcc is already set in the orignal mail. Unless you keep toggling auto-bcc on and off this means your auto-bcc is A) already included, or B) in the original mail you chose to manually exclude it. 
Also note, this only about what fields get pre-filled in the composer - we don't change anything on send.

Comment 15

[Magnus Melin [:mkmelin]]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

Review of attachment 8373531 [details] [diff] [review]:
-----------------------------------------------------------------

Maybe Irving has some cycles to review this?

Comment 16

[smithtom947]

(In reply to Magnus Melin from comment #14)
> The patch always adds the auto-BCC, except for when it is a reply to self
> AND Bcc is already set in the orignal mail.

Note that an auto Bcc or CC is not always to self, or may be to an alternative "self" address (another mailbox or another route), and that it may be a list.

So, does the patch omit it when *the same* Bcc list is already set, or when any old Bcc is already set? I can understand not wanting to stack up repeated duplicate addressees, not that that matters to the SMTP transport, but you wouldn't want to skip adding an auto-Bcc just because one or more different ones were already there.

Also, just to be clear, "reply to self" is defined as what exactly? The original sender address is the same as the current identity's address?

Comment 17

[Magnus Melin [:mkmelin]]

(In reply to smith from comment #16)
> So, does the patch omit it when *the same* Bcc list is already set, or when
> any old Bcc is already set? I can understand not wanting to stack up
> repeated duplicate addressees, not that that matters to the SMTP transport,
> but you wouldn't want to skip adding an auto-Bcc just because one or more
> different ones were already there.

The only reason you would ever have a Bcc header in the original is if you are replying to a mail you sent yourself. You never receive a mail containing a Bcc header.

The patch doesn't check which addresses are in Bcc. The idea is that the followup-mail uses all the same headers as the original. I wouldn't think a case where you change your auto-bcc in between the replies is more common then the case where you manually remove a bcc before sending. 

> Also, just to be clear, "reply to self" is defined as what exactly? The
> original sender address is the same as the current identity's address?

Yes. Except if it's a reply to another one of your identities.

Comment 18

[Chin]

(In reply to smith from comment #13)

> If you've set an auto CC or auto BCC, it should do just that, not
> auto CC/BCC except on alternate Tuesdays when Joe is sending it. There is
> nothing about that behavior that depends on who is sending the message,
> where it's being sent from, or who it's being sent to.

This is exactly, what I am looking for. If we set the BCC in the account settings, it will just do that.

Is this what will be implemented in the fix?

Comment 19

[smithtom947]

(In reply to Magnus Melin from comment #17)
> 
> The only reason you would ever have a Bcc header in the original is if you
> are replying to a mail you sent yourself. You never receive a mail
> containing a Bcc header.

To be precise, the only reason you would ever have a Bcc header in the original is if you composed the message yourself and auto-filed it as it was sent. Correct? I don't see any other way to "reply" other than from a folder other than "Drafts" and no way to get a message into a folder other than to have auto-filed it when sent or to have received it through the mail.

> 
> The patch doesn't check which addresses are in Bcc. The idea is that the
> followup-mail uses all the same headers as the original. I wouldn't think a
> case where you change your auto-bcc in between the replies is more common
> then the case where you manually remove a bcc before sending. 
> 

So the effect of this patch would simply be to preserve all the original headers on any reply and then to add an auto-Bcc if set and if and only if no other Bcc is present. No other criteria. Correct? How does all this reasoning apply to auto-CC? Seems like a violation of the Principle of Least Astonishment if they were to behave differently.

I'm speculating that this funky little dance is a result of somebody who found a lot of duplicate Bcc's in messages in his sent mail folder after a series of replies, and was disturbed by its insufficient neatness. Fair enough. But wouldn't the same be true of auto-CC? And those don't get trimmed off by the SMTP transport.

If the problem to be solved is duplicate headers, shouldn't the solution have had something more directly to do with duplicate headers? Like, for example, when an address - any address - is added to the recipient list by any means, check for and remove any duplicates. Remove Bcc's, CC's, and To's in that order until the recipient is listed only once. 

> > Also, just to be clear, "reply to self" is defined as what exactly? The
> > original sender address is the same as the current identity's address?
> 
> Yes. Except if it's a reply to another one of your identities.

Sorry, I didn't follow that. I get the impression, though, from your earlier explanation, that whether it's a "reply-to-self" or not is irrelevant. It's just merely a fact that the only time you'll see a Bcc is on a message that you yourself have composed (and auto-filed).

Comment 20

[rsx11m]

(In reply to smith from comment #19)
> So the effect of this patch would simply be to preserve all the original
> headers on any reply and then to add an auto-Bcc if set and if and only if
> no other Bcc is present. No other criteria. Correct? How does all this
> reasoning apply to auto-CC?

Bcc and Cc are different cases, given that the Cc headers are always preserved regardless of where the message sent ends up. It is my understanding that the patch for bug 933555 treats Cc headers similarly and just copies them back into the message composed when replying to one of your own messages.

> > > Also, just to be clear, "reply to self" is defined as what exactly? The
> > > original sender address is the same as the current identity's address?
> > 
> > Yes. Except if it's a reply to another one of your identities.

The latter case should be covered by setting the mailnews.reply_to_self_check_all_ident preference.

Comment 21

[Magnus Melin [:mkmelin]]

(In reply to smith from comment #19)
> To be precise, the only reason you would ever have a Bcc header in the
> original is if you composed the message yourself and auto-filed it as it was
> sent. Correct? 

Yes.

> > The patch doesn't check which addresses are in Bcc. The idea is that the
> > followup-mail uses all the same headers as the original. I wouldn't think a
> > case where you change your auto-bcc in between the replies is more common
> > then the case where you manually remove a bcc before sending. 
> > 
> 
> So the effect of this patch would simply be to preserve all the original
> headers on any reply and then to add an auto-Bcc if set, if and only if
> no other Bcc is present. No other criteria. Correct? 

Yes. (Because if Bcc is present, your auto-Bcc would be there too already, unless you manually removed it.)

> How does all this reasoning apply to auto-CC? Seems like a violation of the Principle of Least
> Astonishment if they were to behave differently.

Yes, Cc addresses are also just copied from the original.

> I'm speculating that this funky little dance is a result of somebody who
> found a lot of duplicate Bcc's in messages in his sent mail folder after a
> series of replies, and was disturbed by its insufficient neatness. Fair
> enough. But wouldn't the same be true of auto-CC? And those don't get
> trimmed off by the SMTP transport.

This is not the case at all. We remove duplicates ourselves as necessary.

> If the problem to be solved is duplicate headers, shouldn't the solution

Well, this is a bug, just one case I didn't think of. It has little to do with duplicate headers.

> Sorry, I didn't follow that. I get the impression, though, from your earlier
> explanation, that whether it's a "reply-to-self" or not is irrelevant. It's
> just merely a fact that the only time you'll see a Bcc is on a message that
> you yourself have composed (and auto-filed).

No, this bug only applies to reply-to-self. If it's not a reply-to-self we wouldn't try to use the original Bcc header (which is empty).

Comment 22

[smithtom947]

(In reply to Magnus Melin from comment #21)

Hi Magnus. I'm seriously not trying to give you a hard time. Just maybe uncover some other cases that might not have been thought about before they become a new surprise.

> (In reply to smith from comment #19)
> > How does all this reasoning apply to auto-CC? Seems like a violation of the Principle of Least
> > Astonishment if they were to behave differently.
> 
> Yes, Cc addresses are also just copied from the original.

I'm asking about auto-CC. Like auto-Bcc, but it inserts a CC header instead, which would duplicate any previously inserted auto-CC header. Are auto-CC additions similarly skipped if there is any pre-existing CC header?

> 
> > I'm speculating that this funky little dance is a result of somebody who
> > found a lot of duplicate Bcc's in messages in his sent mail folder after a
> > series of replies, and was disturbed by its insufficient neatness. Fair
> > enough. But wouldn't the same be true of auto-CC? And those don't get
> > trimmed off by the SMTP transport.
> 
> This is not the case at all. We remove duplicates ourselves as necessary.
> 

Not my observation. If that were true, there would be no need to special-case reply-all or auto-Bcc/CC in the first place. Add the auto-Bcc/CC without question and if it's a duplicate it would be removed. But I don't see any evidence that duplicates are being removed. If I send myself mail with 2 identical CC: headers, both identical to the TO: header, I receive 3 headers, not 2, and not 1. So neither Thunderbird nor the SMTP relays are laundering the recipient headers for anything other than blanket removal of Bcc's.

> > If the problem to be solved is duplicate headers, shouldn't the solution
> 
> Well, this is a bug, just one case I didn't think of. It has little to do
> with duplicate headers.
> 

What other reason would there be to do this special-case processing?

Comment 23

[Magnus Melin [:mkmelin]]

Nowadays when To/Cc semantics are preserved on reply, the difference for Reply-All is not that significant. For (single) reply it makes all the difference of course.

Comment 24

[:Irving Reid (No longer working on Firefox)]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

Review of attachment 8373531 [details] [diff] [review]:
-----------------------------------------------------------------

The code looks fine for what it does, but I have to admit I'm concerned about how complex our behaviour is here - we're three bugs deep in 'fixing corner case A causes corner case B to regress'. I don't have a good answer for that, since "take out all the special cases" is not likely to go over well with the users that depend on them. I guess we'll have to keep piling on the regression tests.

::: mailnews/compose/src/nsMsgCompose.cpp
@@ +2589,5 @@
>          {
>            compFields->SetTo(to);
>            compFields->SetCc(cc);
> +          // In case it's a reply to self, but it's not the actual source of the
> +          // sent message, we won't know the BCC header. So set it only if 

Nit: trailing white space

@@ +2775,5 @@
> +            // Remove addresses already in To from Bcc.
> +            RemoveDuplicateAddresses(resultStr,
> +                                     nsDependentCString(_compFields->GetTo()),
> +                                     resultStr);
> +            

White space

Comment 25

[Mark Banner (:standard8)]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

Review of attachment 8373531 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, it feels strange with all the related regressions, but the good thing is we're getting tests in place, so at least we're improving our test infrastructure and less likely to hit issues or more regressions in future. r=Standard8

Comment 26

[Tom Kreutz]

Just a really unhappy user here who wants (and now expects after many years of use) ALL emails to automatically have a Bcc to myself, as configured.  Suddenly I have no record of my own sent emails; it's so frustrating.  I have to write the recipients and ask for a copy of my own email!  It's really embarrassing...

Oh, PLEASE fix this regression!

Comment 27

[rsx11m]

Tom, the patch provided by Magnus has received approval already, thus should land in the development builds. If it's fine there, it will go into the beta and next 24.x release. Thus, the problem is known and in progress, hence no need for further motivation.

Comment 28

[Magnus Melin [:mkmelin]]

https://hg.mozilla.org/comm-central/rev/bef8e5e69c11 -> FIXED

Comment 29

[Magnus Melin [:mkmelin]]

I'd encourage people to try out nightly builds with the fix included - you can get one from here: http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central/

Comment 30

[Magnus Melin [:mkmelin]]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

Review of attachment 8373531 [details] [diff] [review]:
-----------------------------------------------------------------

Maybe Irving has some cycles to review this?

Comment 31

[rsx11m]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

Bug 933555 also landed on comm-aurora at that time, thus SeaMonkey will need this patch on comm-beta to be effective for SM 2.25.

Adding this to the existing branch requests.

Comment 32

[Joachim Herb]

I guess this bug is related to bug 798282: With the fix of this bug, the "automatic" BCC address is not added if there is already at least one BCC address. So if I have e.g. a mail in the Sent folder and want to reply to all and that message includes some BCC addresses but not my "automatic" BCC address a reply to all will not add my "automatic" BCC address. So as said in comment #26 you will lose the new message.

Comment 33

[rsx11m]

That's intended per comment #14. Thus, if you manually removed the automatic Bcc in your original message it won't be in the follow-up message either (preserving the intend in the first instance). Conversely, if you didn't remove the automatic Bcc it will be included in the follow-up. The latter should apply in the default case. If you decided to "lose" the original message you'd also lose the follow-up, this sounds logical to me.

Bug 798282 sounds very similar, would you think that it will be fixed by the patch here? It appears that this case is covered by bug 969358 rather, which in turn is a recent regression, thus shouldn't have happened before bug 933555 was checked in.

Comment 34

[Joachim Herb]

If that is the intended behavior then bug 798282 can be closed as INVALID.
(In reply to rsx11m from comment #33)
> That's intended per comment #14. Thus, if you manually removed the automatic
> Bcc in your original message it won't be in the follow-up message either
> (preserving the intend in the first instance). Conversely, if you didn't
> remove the automatic Bcc it will be included in the follow-up. The latter
> should apply in the default case. If you decided to "lose" the original
> message you'd also lose the follow-up, this sounds logical to me.
> 
> Bug 798282 sounds very similar, would you think that it will be fixed by the
> patch here? It appears that this case is covered by bug 969358 rather, which
> in turn is a recent regression, thus shouldn't have happened before bug
> 933555 was checked in.

Comment 35

[Mark Banner (:standard8)]

https://hg.mozilla.org/releases/comm-aurora/rev/180cac6ac8fe

Comment 36

[Mark Banner (:standard8)]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

Unfortunately this doesn't apply cleanly to beta. Magnus, could you upload a patch that works for Beta & ESR?

Thanks

Comment 37

[Mark Banner (:standard8)]

Comment on attachment 8373531 [details] [diff] [review]
proposed fix

I've managed to fix the merge issues on this:

https://hg.mozilla.org/releases/comm-beta/rev/c6c68dd64727

Comment 38

[rsx11m]

Verified fixed in SM 2.26a2 nightly and 2.25b tinderbox builds (Windows/Linux). Reply All to my own message adds the auto-Bcc address again if no Bcc header is present in the original e-mail.

Comment 39

[rsx11m]

This made it in time for THUNDERBIRD_28_0b1_BUILD1, thus marking as fixed there as well.

Comment 40

[Mark Banner (:standard8)]

https://hg.mozilla.org/releases/comm-esr24/rev/290d7629194a

Comment 41

[rsx11m]

Verified fixed Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 as well.

Comment 42

[Luke deGruchy]

This is verified with the Thunderbird 24.4.0 linux-x86_64 tarball (I do get BCC'd when replying to myself).

However, when I do Help > About Thunderbird the stated version is 24.2.0, so how can I be absolutely sure that this is in fact Thunderbird 24.4.0, and not a mislabeled 24.2.0?

Comment 43

[rsx11m]

In the thunderbird/ installation directory, look at application.ini with a text editor.
It should read something like:

> [App]
> Name=Thunderbird
> Version=24.4.0
> BuildID=20140316xxxxxx
> SourceRepository=https://hg.mozilla.org/releases/comm-esr24
> SourceStamp=a5ac6f0c9785
> ID={3550f703-e582-4d05-9a08-453d09bdfdc6}

If "Version" is wrong here, there was a problem with that build not identifying itself correctly. Another frequently seen issue is add-ons overriding the user-agent string, thus leaving it at an earlier version (at least with the add-on disabled or removed at some point).

Go into Edit > Preferences > Advanced > General tab and click Config Editor, then look for the general.useragent.override preference. If it's set, right-click and select Reset to clear it.

Comment 44

[Luke deGruchy]

This is what I see in application.ini:

Name=Thunderbird
Version=24.4.0
BuildID=20140316131045
SourceRepository=https://hg.mozilla.org/releases/comm-esr24
SourceStamp=a5ac6f0c9785
ID={3550f703-e582-4d05-9a08-453d09bdfdc6}

Config Edit shows me:

general.useragent.enable_overrides;false
general.useragent.site_specific_overrides;true

So thankfully it's just Help > About that's wrong.

Comment 45

[rsx11m]

Hmm, I've just downloaded the official 64-bit tar ball from http://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/24.4.0/linux-x86_64/en-US/ and it shows correctly version 24.4.0 in Help > About. The build seems to be fine, thus no clue why you are seeing this.

Comment 46

[Luke deGruchy]

Odd.... I've relaunched it and now it works.  Help > About shows 24.4.0.

The only thing I can think of is that there was some kind of artifact from my previous installation of 24.2.0.

And I double-checked the BCC bug after confirming Help > About and it's still fixed.

Comment 47

[rsx11m]

Ok, so problem solved. If you or anybody else sees an issue with the About window (again), then please file a new bug for this.