699681 - [Meta] Preset From: upon Reply/Forward is frequently far different from user's expectation, due to forcing X-Account-Key: based identity selection or "Reply-to-self" on any user, by inappropriate mail addr matching/guessing or Reply-To handling)

Status: NEW

(Thunderbird :: General, defect)

Description

[WADA:World Anti-bad-Duping Agency]

Preset From: upon Reply/Forward is frequently far different from user's expectation, due to (a) forcing of X-Account-Key: based identity selection for Global Inbox support on any user, and due to (b) forcing of "Reply-to-Self" on any user even though the feature is for convenience of limited users only.
This produced not-so-small number of unwanted bugs like "Wrong from in reply", "Mail was sent using wrong From:".
This bug is meta bug(tracking bug) for analysis of such bugs and duping of them.

For (a), bug 327713 exists.
For (b), I don't know about existent bug.

According to bug 327713 comment #53, Add-on of "Correct Identity" looks effective for victims of the problem.
> https://addons.mozilla.org/en-us/thunderbird/addon/correct-identity/

Comment 1

[WADA:World Anti-bad-Duping Agency]

Meta bug for general "unexpected identity selection" is found.
xref-ing to that bug via Blocks: field of this bug.
> Bug 228980 : (IdentityPickTracker) [Meta] Identity selection based on recipients should be more correct

Comment 2

[WADA:World Anti-bad-Duping Agency]

Note-1:
If account number is changed at a Tb, phenomenon produced by "identity selection based on X-Account-Key: header" becomes funnier.
Note-2:
Because bug 426651 exists, "identity selection based on X-Account-Key: header" can occur even on mail in IMAP folder. xref bug 426651.

Comment 3

[BayouBill]

> According to bug 327713 comment #53, Add-on of "Correct Identity" looks
> effective for victims of the problem.
> https://addons.mozilla.org/en-us/thunderbird/addon/correct-identity/

As noted in the "Additional information" section of the original Description of bug 684259, installing Correct Identity did not help in that situation.

Comment 4

[WADA:World Anti-bad-Duping Agency]

FYI.
Identity selection for preset From: is done at following code.
http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompose.cpp#1930
> 1930       switch (type)
> 1931       {
> 1932         default: break;
> 1933         case nsIMsgCompType::Reply :
> 1934         case nsIMsgCompType::ReplyAll:
> 1935         case nsIMsgCompType::ReplyToList:
> 1936         case nsIMsgCompType::ReplyToGroup:
> 1937         case nsIMsgCompType::ReplyToSender:
> 1938         case nsIMsgCompType::ReplyToSenderAndGroup:

I think enhancement like next will reduce many user's current confusion.
(a) mailnews.reply_to_self_is_enabled = true/false (default=false)
    If possible, per account option is preferable.
    Global setting & per account setting like retention policy may be better.
(b) mailnews.account_key_based_identity_selection_when_non_Global_Inbox_owner
        = true/false (default=false)
    If possible, per account option is preferable.
    When Global Inbox owner account, this option should be ignored.
    Global setting & per account setting like retention policy may be better.
Use cases of them are independent from use case of "per folder identity", although "per folder identity" can reduce current user's confusion too.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Note-3:
Because bug 270068 exists, "identity selection based on X-Account-Key: header" can occur even on mail which is sent as attachment, saved as .eml, then imported by Drag&Drop of the .eml file. xref bug 270068.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Note-4:
Reply-to-self is affected by next setting.
  mailnews.reply_to_self_check_all_ident = true/false(it looks false is defaulted)
  perhaps next;
    true  : any identity is considered as "self"
    false : identities of selected account only is considered as "self"

Comment 7

[WADA:World Anti-bad-Duping Agency]

Note-5:
Identities are defined in prefs.js like next;
 (account definition)
  mail.account.accountA.identities = idAa,idAb, ... ,idAz
                                     Left most one=Main identity of this account
  mail.account.accountA.server     = serverS
 (mail server definition)
  mail.server.serverS.realhostname/hostname = mail.server.name
  mail.server.serverS.realuserName/userName = login-useid-of-this-mail-account
 (identity definition)
  mail.identity.idAa.useremail = mail-addr-Aa@mail-domain-for-Aa
                   |
  mail.identity.idAz.useremail = mail-addr-Az@mail-domain-for-Az
 (identity's SMTP server choice)
  mail.identity.idAa.smtpServer = smtpP
 (SMTP definition)
  mail.smtpserver.smtpP.hostname = smtp.server.name
  mail.smtpserver.smtpP.username = login-useid-of-this-smtp-account

Comment 8

[WADA:World Anti-bad-Duping Agency]

Tb 8 on Win-XP showed next in Account column.

account1/server1 to account8 is defined. No Global Inbox is used.
  account1/server1 : POP3, not default account
  account2/server2 : POP3, default account
  account3/server3 : Unified Folders
  accountP/serverP : POP3, not default account
  accountS/serverS : IMAP account
  accountL/serverL : Local Folders  
Checked at POP3 accountP/serverP(not server1, not default account, not Local Folders, not Unified Folders), Local Folders, and IMAP account.

                            accountP         accountL         accountS
                            serverP          serverL          serverS
 X-Account-Key: of mail     POP3 folder      Local Folders    IMAP folder 
 
 mail-X : accountN          accountN         accountN         serverS.name of    
   (accountN is defined)    -> serverN.name  -> serverN.name  this IMAP account
   (non Local Folders  )

 mail-Y : accountL          accountL         accountL         serverS.name of    
   (accountL is defined)    -> serverL.name  -> serverL.name  this IMAP account
   (as Local Folders   )    ==Local Folders  ==Local Folders   

 mail-Z : accountC          serverP.name of  serverL.name of  serverS.name of    
   (accountC is        )    this account     this account     this IMAP account
   (  not defined      )                     ==Local Folders   

 mail-W : no X-Account-Key  server1.name of  server1.name of  serverS.name of    
                            account1         account1         this account
     
Note:
Even when existent accountN is hidden account for Unified Folders(smart mail boxes), corresponding serverN.name is used and "Unified Folders" is shown.

I wasn't aware of next until I execute above check.
  account1 or accountX for server1 is always used when no X-Account-Key: header.
This is possibly one of big causes of phenomenon of "preselected reply address is far from user's expectation" in no X-Account-Key: header case.
Some funny phenomena may be no server1/account1 case.

Comment 9

[WADA:World Anti-bad-Duping Agency]

Note.
mail.accountmanager.accounts in above check is as follows (account9999/server9999=Local Folders).
> account2,account4,account9999,account1,account3,account7,account5,account6,account8
So order of accounts is perhaps irrelevant to "always select server1 or account1 if no X-Account-Keys:".

Comment 10

[WADA:World Anti-bad-Duping Agency]

FYI.
Actually WRONG pre-set From: case existed - Bug 397975 (fixed by Tb 13.0).
If [e-mail] is contained in To:, CC:, and if [e-mail] is not defined as Identity but [e-mail] is defined as Identity, [e-mail] is selected as Identity for reply mail due to funny mail address guessing without localpart matching.

Comment 11

[WADA:World Anti-bad-Duping Agency]

FYI.
There are three known/major kinds of "Unwanted/Unexpected pre-set From:".
(1) Due to Tb's identity selection based on identity related Tb's message header,
.   even when user doesn't want it.
    Based on X-Account-Key: header, X-Identity-Key: header.
(2) Due to Tb's "Reply-to-self always even when user doesn't want it".
(3) Due to mail address guessing without localpart matching (Bug 397975).

Comment 12

[brainstuff John Scheb]

I have again a false sender address when I transfer an email received on one account thunderbird send it with an different account as sender 

I have made a screen copy but it is not possible to post it heare. But the contents is, when I klick on "transfer to",  similar to 

sender  [e-mail] 
response to: [e-mail] 

and in the text field the header is 

subject
date
from: [e-mail] 
respone to [e-mail] 


Please take also note that the default account is [e-mail]  ... will say he do not use the default account instead of the original used account

Comment 13

[brainstuff John Scheb]

To say it in a simpla way: when I receive a mail on the account [e-mail] the transfered mail must be send by [e-mail] with reply address [e-mail] .

In the moment thunderbird use a different account to transfer the mail.

Comment 14

[WADA:World Anti-bad-Duping Agency]

Bug 608513 is for user's inconvenience in utilizing current reply-to-self feature. It sounds that mailnews.reply_to_self_check_all_ident=true(defaulted as false) is a solution in his case. So that bug looks "reply to his identity which is associated to different account" case.
Following enhancement may be a solutuion for users who don't want automatic reply-to-self application.
  mailnews.reply_to_self_check_all_ident : Boolean -> Integer
    0 : reply-to-self is disabled
    1 : reply-to-self is enabled, check identities of an relevant account only
    2 : reply-to-self is enabled, check identities of all accounts

Comment 15

[WADA:World Anti-bad-Duping Agency]

FYI.
As for problem due to "X-Account-Key: based identity selection", powerful sord has been implemented.
  Re-define POP3 account
By bug 485839, mail.account.lastKey was introduced, and mail.account.lastKey + 1 is used as account number upon next account definition. So, delete/re-define POP3 account alters account number of POP3 account, then accountN in existent X-Account-Key: header is changed to obsolete.
(1) If default account, select appropriate account as default account,
    in order to avoid problem due to "defaultaccunt=Local Folders" etc.
(2) Delete POP3 account, re-define POP3 account
    => account number is incremented.
    => server number may be altered, and new directory-rel is used.
    Old one : mail.server.serverX.directory-rel = .../<servername>-XX
    New one : mail.server.serverY.directory-rel = .../<servername>-YY
(3) Use mail directory assinged to Old one.
    mail.server.serverY.directory-rel = .../<servername>-XX
    restart Tb.
(4) Select appropriate account as default account
(5) If you start to feel "X-Account-Key: based identity selection" is annoying, do (1) to (4) each time.

Comment 16

[David]

Just fix the #$@% bug. Add an option to ignore X-Account-Key.

Comment 17

[WADA:World Anti-bad-Duping Agency]

Recently, problem around Reply-To: has been reported on Tb 24.
  For example, bug 932774 bug 933377, bug 933555.
Addin "Reply-To:" to ug summary.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Quick summaary of current problems.
(1) Problem due to wrong mail addr matching or inappropriate mail addr guessing.
    Fixed in Tb 13.0 by Bug 397975.
(2) Problem due to "forcing X-Account-Key: based Identity selection, even when Global Inbox is never used by Tb user".
(3) Problem relevant to some other X-...: headers used by Tb.
(4) Problem due to "forcing Reply-To-Self, even when Tb user doesn't want it".
(5) Problem which may be relevant to not-so-appopriate behavior or deffernt-from-user-expectation behavior on Reply-To:, Sender: etc.
(6) Others.

Comment 21

[David]

Surely the idea of multiple accounts is that an account is setup for each 
email identity that the user has, ie Home, work, charity etc. 
Then it make sense to keep the mail relating to that identity in folders 
in the relevant account, ie any work related emails would be kept in the 
work\inbox, or work\sent etc. 
Emails could get there by either downloading from different pop servers, 
or by dragging from another account if all incoming mails are consolidated to one server.

Composing a new message when a certain account is highlighted works correctly, 
ie the correct formatting is selected (text/html) and the correct from address 
is selected. 

When it comes to replying or forwarding a message that is highlighted, 
it should work the same way, BUT IT DOESNT !!

Please correct me if I am wrong, but I believe the solution is blatantly obvious and very simple:
1 Ignore the X-Account-Key, or add an option to ignore the X-Account-Key
2 When replying to a message, or forwarding a message, use the same subroutine that creates a new message. 
   That correctly Opens a new edit window
   Uses the correct formatting 
   Uses the correct signature, CC and Bcc addresses
   Uses the correct From address 

To cater for different setups/users then it would be sensible to have some options 
that the user could select in their preferences.

Comment 22

[brainstuff John Scheb]

(In reply to David from comment #21)
> Surely the idea of multiple accounts is that an account is setup for each 
> email identity that the user has, ie Home, work, charity etc. 
> Then it make sense to keep the mail relating to that identity in folders 
> in the relevant account, ie any work related emails would be kept in the 
> work\inbox, or work\sent etc. 
> Emails could get there by either downloading from different pop servers, 
> or by dragging from another account if all incoming mails are consolidated
> to one server.
> 
> Composing a new message when a certain account is highlighted works
> correctly, 
> ie the correct formatting is selected (text/html) and the correct from
> address 
> is selected. 
> 
> When it comes to replying or forwarding a message that is highlighted, 
> it should work the same way, BUT IT DOESNT !!
> 
> Please correct me if I am wrong, but I believe the solution is blatantly
> obvious and very simple:
> 1 Ignore the X-Account-Key, or add an option to ignore the X-Account-Key
> 2 When replying to a message, or forwarding a message, use the same
> subroutine that creates a new message. 
>    That correctly Opens a new edit window
>    Uses the correct formatting 
>    Uses the correct signature, CC and Bcc addresses
>    Uses the correct From address 
> 
> To cater for different setups/users then it would be sensible to have some
> options 
> that the user could select in their preferences.

@david  ..

I am OK with you, but not with "relating to that identity in folders 
in the relevant account,"  Folders does not have any relation with the senders mail address ... The only criteria is the mail address where I receive a mail ... this must be every time the senders mals and the reply mail addres. 

brainstuff

Comment 23

[Chip Davis]

(See Bug 1076964 for copious documentation.)

Something has changed in this area of the code.

Before 31.1.2 I never had a problem with TB selecting the correct 'From:' identity in a 'Reply' or 'Forward'.  Now, it consistently picks the wrong identity for replies to certain incoming mail.  Not only is the identity not associated with the address to which the email was sent, it's not the default identity of my default account either.

This occurs with mail from any listserver that insert its own address in the 'To:' header, and mail sent 'To: undisclosed-recipients:;'.

In these cases, TB determines the correct address to which the email was sent (I assume from the 'Received:' headers) because the 'X-Account-Key:' header contains the correct account.

Since 31.1.2 it appears that when TB cannot determine the identity from the 'To:' or any 'Delivered-to:' headers, it ignores any 'X-Account-Key:' or 'Envelope-to:' headers AND the default account/identity setting and picks an identity apparently at random.  It is not 'id1' nor is it an identity associated with 'account1'. (I have no idea what the comment "// Still no matches? Give up and pick the first one" means in the code.)

Is there anything I can do to force TB to use the 'X-Account-Key:' or 'Envelope-to:' headers if present?  I haven't gotten a 'Delivered-to:' header since 2012.  Neither it nor the 'Envelope-to:' headers are held in high regard, so why not use the 'X-Account-Key:' information (if present) as a hint for the identity if you have nothing but "pick the first"?

This is obviously a messy area (judging by Comment 18 if nothing else) but the wholesale elimination of 'X-Account-Key:' as a determinant of the proper reply 'From:' identity has already broken a common email occurrence.  Any way to un-screw this pooch?

Comment 24

[Thomas D. (:thomas8)]

Some simple variant of this (along bug 903390) has just bitten me and this SUCKs big time. Can result in MASSIVE privacy violation when TB selects unexpected identity and user doesn't notice before sending.

Comment 26

[Wayne Mery (:wsmwk)]

more potential duplicates