18906 - smart-identity based on currently displayed message

Status: RESOLVED FIXED

(MailNews Core :: Composition, enhancement, P4)

Description

[Alec Flett]

One feature I realized we're missing from the smart-identity code is the
automatic selection of the default identity based on the currently-displayed
message.

The current mechanism just auto-selects the current identity based on the
currently displayed folder. Before this happens, we should be looking at the
"To", "Cc" or "Resent-From" (?) header to see if we can find an identity
there... probably based on e-mail address.

For example, say I have an alias set up so that mail to [e-mail]
goes to my normal inbox for [e-mail]. If I receive mail addressed to
[e-mail], and then reply to it, I should be replying as
[e-mail].

(And of course, this is only if we actually have an identity with the e-mail
address "[e-mail]")

So the fallback order when composing a message should be this:
- look at the e-mail addresses in the "To" or "Cc" headers, and try to find a
relevant identity.
- if that fails, then use the identity associated with the current folder
- if there's no current folder, then just use the default identity.

Comment 1

[Jean-Francois Ducarroz]

Should this be done by message compose or by messenger itself. personnaly I vote
for the second one. But thechnically it my be easier to do it from msgCompose

Comment 2

[Ronald F. Guilmette]

I don't want to rain on this parade, but as I noted in the additional comment
that I just attached to Bug #18882 a moment or two ago, I would classify this
enhancement idea as ``good in theory, impossible in practice''.

Yes, if it were possible, you would definitely like to have your mail client
be smart enough to know which one of your several alias/pseudonym e-mail
addresses the message that you were currently composing a response to was
originally sent to, and you would like the mail client to then assume that
specific personal ``identity'', at least long enough so that it could offer
you a default From: header line that exactly matches that specific identity.

The problem is that there is just no way to reliably get the *real* ``original
recipient address'' out of the message you are responding to, because in
general, it just isn't in there.  Oh sure, you can look in the To: and/or Cc:
headers, and hope that you find the ``original recipient address'' that you are
looking for in one or the other of those two headers, but if the message was
sent to you as a blind carbon (which is quite common, at least for mailing list
messages) then the address you are looking for won't even be there.  And even
in the cases where the address you are looking for *is* in either the To: or
Cc: headers, how do you know which address, in particular, among all of the
recipient addresses inthose two headers is the specific one you're looking for?

The answer is that you simply can't know which one is the ``right'' one.  There
is no way of telling.

Someday in the distant future, all SMTP servers (and also all SMTP clients) will
fully implement ESMTP, and they will all strive to create and maintain ORCPT=
parameters, attaching such parameters to *all* RCPT TO commands used to transfer
mail messages from point A to point B, and THEN, maybe with a little more
cooperation from local delivery agents (which must all be converted over to
using LMTP, so that they also can get at the ORCPT= data) THEN maybe you will
be able to reliably get at the ``original recipient address'' information you
really need to implement the specific enhancement proposed in this Bugzilla
item, but don't hold your breath.  Its a big planet, and there is still an
awful lot of mail software out there that either (a) doesn't speak ESMTP or
that (b) doesn't grok ORCPT= parameters or that (c) groks them but ignores
them.  (I'm doing my part to try to change all this, but its a slow process,
and like I say, its a big planet.)

Speaking of which, does Messenger properly supply the ORCPT= on each RCPT TO
command when it is speaking to an ESMTP-capable mail server?  (I'll wager that
it doesn't... but it ought to.)

The bottom line is just this... the To: and Cc: headers are just (in effect)
body text as far as basic SMTP/ESMTP/LMTP transport is concerned, and lusers
can put any old garbage they want into those header lines.  They can't be relied
on to contain useful information about the true original recipient address, and
it would most probably be a mistake to try to rely on them for that.

Until the whole world starts speaking _proper_ and _full_ ESMTP/LMTP (in
particular, by always supplying and preserving the envelope ORCPT= value
associated with each separate recipient) I think you are pretty much SOL
as regards to implementing the enhancement proposed in this Bugzilla item
in anything approaching a reliable fashion.  (I think that's a darn shame,
because it is basically a great idea, but what are you going to do?  Reality
bites.)

Well, OK, being a bit more pragmatic about this, I'l concede that you _could_
allow the user to specify his own personal list of ``active aliases'', and you
could then just go and look for matches with the elements of that in the To:
and/or Cc: headers (and then, if you find a match, use that as the default From:
address) but just remember that you _may_ find some number of matches OTHER THAN
one, e.g. zero, two, etc.  My advice: Just be prepared to deal with those cases
too.

Comment 3

[Alec Flett]

I think messenger is the right person to pick up this identity.... in the same
place where smart-compose is already happening.

Ronald: most of your comments are based on assuming we don't the user's
identities. We already have this list of "personal aliases" and it's called the
nsIMsgIdentity. We have a list of them stored in the account manager, one or
more stored with each account.... it's simply a matter of looking at all the
nsIMsgIdentities (nsIMsgAccountManager.allIdentities) and intersecting that with
the e-mail addresses in the To/Cc headers of the current message.

Comment 4

[Ronald F. Guilmette]

No, I assure you that I *did* understand that you have a list of identities
stashed away already.  But my comments still stand.  Just doing a search for
each element of that list, in turn, looking for matches within the To: and/or
Cc: header lines will NOT always get you what you want.  In fact, in the
specific case that *I* am most worried about, i.e. responding to mailing
list messages (which are typically sent as blind carbons) when you try to
do the matching process you have described, you will come up with zip,
zero, nada, no matches.  And then what do you do?

Comment 5

[Alec Flett]

bcc is indeed a strange case. I'm not sure what to do about that.

Comment 6

[Ronald F. Guilmette]

Please do not think of blind carbons as being merely a ``strange case'' or a
``corner case'' or even as ``the exception the proves the rule''.  Personally,
I receive far more mail from mailing lists than I do directly from individual
correspondants, and virtually 100% of that mailing list traffic comes to me as
blind carbons.  And I don't believe that I am particularly unusual in this
regard.  If you try to design a solution around the assumption that a
recipient's address WILL appear in either the To: or the Cc: headers, then that
solution is just not going to work a significant percentage of the time.  That's
actually OK, I guess, and in those cases you can just fallback to using some
default address for the recipient (in the From: line you generate when/if he
attempts to reply to some incoming message), and then your scheme will work
adequately well for most responses to ordinary person-to-person messages, but
you should understand up-front that this scheme just won't at all well for
responses to mailing-list messages (which, in my case at least, constitutes a
majority of the e-mail messages I receive on a daily basis).  So in addition to
what you have propsed in _this_ Bugzilla item, there also needs to be some OTHER
separate mechanism that will generate appropriate default From: headers for
responses to (and also original submissions to) mailing lists. That other
(adjunct) mechanism can be, and (in my opinion) should be the support for
plussed addresses proposed in Bug #18882.

Comment 7

[Alec Flett]

Sorry, I meant "strange" like "may be hard to figure out" not like it was a rare
case.
I do think it's important to realize that there are multiple reasons for this
feature, not just mailing lists... your ISP could be giving your family multiple
addresses all directed into the same inbox as one example, so checking To and Cc
are not a lost cause.

Comment 8

[Alec Flett]

Oh, and also realize that MOST users use POP, not IMAP... and only a fraction of
them use cyrus, so the ++ also addresses a fraction of the cases. I'm not
against it, but just so we all have some perspective here.

Comment 9

[Phil Peterson]

Picking identity based on the folder seems like enough for B1

Comment 10

[Steven Grimm]

I'm running qmail as my MTA/MDA, and it uses "-", not "+", for address 
subdivision.  So any solution that sticks "+foldername" on the end of my E-mail 
address will be useless to me.  In addition, qmail's subdivision feature isn't 
just for sticking mail in folders; each subaddress is a separate alias and can 
be handled in whatever way you want. For example, most of my subaddresses go 
straight into my inbox and I simply use them to keep track of which Web sites 
are handing my name out and to be able to shut off delivery to any addresses 
that have been handed out to spammers.

It seems like there's a much more straightforward solution to this whole 
problem, one that'll do what everyone needs as well as keep things simple for 
plussed folder users.  It'd work like this:

1. Keep a list that maps incoming addresses to outgoing addresses.  By default 
each mail identity gets added to the list mapped to itself.  When composing, 
search the list to see if any To: or Cc: address in the current 
message is mapped.  If so, use the corresponding outgoing address.

2. For each folder, keep a default outgoing address.  If the incoming address 
isn't found in the map, use the folder's default address.

3. If there's only one recipient listed, and the user has checked a "reply as 
recipient" option, use that as the sender address.

4. Use the default identity.  (Which ends up just being the default outgoing 
address for the "inbox" folder, probably, so this is really the same as the 
previous item.)

Item 1 alone would solve the whole problem for most cases, but without 2 it's a 
bit unwieldy if you're on mailing lists where a consistent list address doesn't 
appear in the To: or Cc: lines.

How would you use this?  For mailing lists, you simply map, for example, 
[e-mail] to [e-mail].  Presto, the right outgoing 
address, and no MDA-specific address semantics hardwired into Mozilla.

To get even more general about it, use a variant of the filter system rather 
than a simple string for step 1.  Then you could have different addresses 
depending on who you're replying to (reply to Dad from your "family" address 
when he forgets to use it) what's in the subject line (for mailing lists where 
the list name gets prepended to the subject) or what's in MTA-specific header 
lines (for MTAs that indicate which address they're delivering to).

Comment 11

[selmer (gone)]

Syncing with marketing priorities.  These are moving to P4 to connote "Cut here
first".  If you disagree with this priority, please let me know.

Comment 12

[scottputterman]

this should probably be moved off the M16 list since I don't think we will get
to it within the next couple of weeks.

Comment 13

[Michael T. Babcock]

The original recipient address can be assumed to be (in order) in the To:
header, and if it is not, then in the top (most recent) Received: header. 
Making this assumption is just adding an educated guess to the system, but still
a good guess.  If it gets the least bit confused, just default to the current
behaviour.

PS, even if you're a BCC:, your username should be in the Received: headers.

Comment 14

[Jean-Francois Ducarroz]

reassign to varada

Comment 15

[risto.kankkunen]

I have implemented a feature, where I have a regexp pref for each identity that
is matched against all the addresses in current message's To: and Cc: headers to
select the outgoing identity when replying. This bug seems to be close to the
same idea, shall I attach a patch here?

Comment 16

[risto.kankkunen]

Attachment: Patch to select identity by a per-identity reg exp when replying

Comment 17

[risto.kankkunen]

Attachment 55672 [details] [diff] contains a patch I have been using to auto-select the identity
for messages I reply to. The idea is to have a regular expression for each
identity, and if the expression matches that identity is selected.

I used regular expressions, since that was easy to implement and gave me the
flexibility I needed. First, just trying to match the addresses with the
identity's address doesn't work since all addresses I know of contain at least
two aliases. The user makes an identity for the one he prefers, but also the
other aliases should map to that same identity. So we need at least lists of
addresses, reg exps were an easy way to implement this. The second reason to use
re's is that I wanted to easily match plussed addresses.

In practise, you add lines like this to your prefs.js:

  user_pref("mail.identity.id8.reply_match_re",
"^(risto.kankkunen|kankkune)@iki.fi$");

Comment 18

[risto.kankkunen]

I see two partly overlapping ways to smart-select the identity for a message:

1. When replying to a message, select an identity based on the address that was
used to send the message to the recipient. This is basically the idea to match
To&Cc that has been discussed here and what my patch addresses in one way.

2. When replying or composing a new message, select the identity based on the
address the message is going to be sent. If I read correctly, I didn't see this
mentioned in this bug yet. This could be done by adding a field to the address
book entries telling what identity to use when sending messages to that address.

#1 is good for situations, where you have a role that reaches an unlimited
number of people. E.g. you have a help desk identity: you cannot list all the
addresses this identity should be used for, but you know that if you receive a
message by this mail alias, you should reply with that identity. It's also
easier to manage.

#2 is good, when the set of recipients is known, e.g. the identity you use for
sending mail to your friends, or to particular mailing lists. The good thing
here is that you don't need to use info from the msg you reply to, so you could
use this even when composing, not just replying. Plus it doesn't matter if you
got the msg via a bcc.

I think #1 & #2 would augment each other nicely and would take care of the bcc
problems mentioned here. Comments?

Comment 19

[scottputterman]

moving to 0.9.9 until after perf work is finished.  

Comment 20

[Jack Tanner]

It's too bad that this is languishing in futured purgatory (although, given the
comments on this bug, I understand that this is certainly hard to implement).
How about a dumb way (rather than the smart ways being discussed) to solve this,
at least temporarily: let the user associate a list of aliases with a mail/news
account. (i.e. account [e-mail] has the alias [e-mail].)
Then, when composing the e-mail, the user can select that alias (and the
corresponding server settings that come for free) from the "From:" dropdown.

This would simplify the workaround I've been using so far -- when I want to send
using a different From: line, I go to the account properties, make the change,
and then cross my fingers to remember to undo this after sending.

A way to make this a little smarter is to have the default alias be the same as
the folder name, but that's a separate can of worms.

Comment 21

[(not reading, please use [e-mail] instead)]

taking all of varada's bugs.

Comment 22

[Josh Steiner]

in reply to the very old comment #4, the original report handles this situation:

"So the fallback order when composing a message should be this:
- look at the e-mail addresses in the "To" or "Cc" headers, and try to find a
relevant identity.
- if that fails, then use the identity associated with the current folder
- if there's no current folder, then just use the default identity."

this would be terrific.

Comment 23

[Ben Bucksch (:BenB)]

*** Bug 226594 has been marked as a duplicate of this bug. ***

Comment 24

[Scott MacGregor]

Attachment: updated fix to leverage Bug #226580

this patch may still need a little more work but it works for mail accounts
using the email address as the hint for figuring out which identity to choose
by default.

Comment 25

[Scott MacGregor]

Comment on attachment 137525 [details] [diff] [review]
updated fix to leverage Bug #226580

this routine can still be improved but I think it's a good start.

One thing it doesn't do good job of yet is mailing lists where none of your
identity email addresses are going to appear in the headers.

Comment 26

[David :Bienvenu]

Comment on attachment 137525 [details] [diff] [review]
updated fix to leverage Bug #226580

nit: I don't think you need to initialize js vars to null.

The searching of the hint string for the email address is just a string search
- that's good enough for now but it would be more correct to parse out the
addresses and compare that way, so that a@b won't match freda@bc

Comment 27

[Scott MacGregor]

Checked in. We can discuss ideas for what to do about mailing lists in a
separate bug.

Comment 28

[Ben Bucksch (:BenB)]

Filed bug 228980 about bienvenu's comment 26.

Comment 29

[Joe Infla]

*** Bug 187578 has been marked as a duplicate of this bug. ***

Comment 30

[Mike Cowperthwaite]

*** Bug 81281 has been marked as a duplicate of this bug. ***

Comment 31

[Scott MacGregor]

I spun out Bug #230230 to help try to improve the mailing list case a little bit.

Comment 32

[Mike Cowperthwaite]

*** Bug 122719 has been marked as a duplicate of this bug. ***

Comment 33

[Constantine A. Murenin]

(In reply to comment #13)
> The original recipient address can be assumed to be (in order) in the To:
> header, and if it is not, then in the top (most recent) Received: header. 
> Making this assumption is just adding an educated guess to the system, but still
> a good guess.  If it gets the least bit confused, just default to the current
> behaviour.
> 
> PS, even if you're a BCC:, your username should be in the Received: headers.

I do not see that this patch does anything with the "Received: " header? mscott,
is there any way you could add that fix to this patch as well? 

I must disagree that we necessarily need to look in the most recent "Received:"
header, as some users have Catch-All addresses on domains with their registrar.
The registrar's mail-server is diverting all mail into customer's mailbox
elsewhere, so that the last "Received: " header always has the same address in it. 

Comment 34

[Mike Cowperthwaite]

If I understand what the patch is supposed to do, it's no longer doing it.
Bug 254209 reports, and I have reproduced, that reply to a message sent to 
identity A when the message is sitting in a folder under an account defaulting 
to identity B initiates the reply From identity B.

Comment 35

[Mike Cowperthwaite]

*** Bug 177515 has been marked as a duplicate of this bug. ***