110605 - Viewer has buggy support around showing recipient "groups" (RFC822)

Status: NEW

(MailNews Core :: MIME, defect)

Description

[Kari Hurtta]

Mail source is

From:   Sonera Plaza Oy <[e-mail]>
To:     Sonera Internet-asiakkaat:;

Mozilla shows this as

    To:  Sonera, "Internet-asiakkaat:;"

obviously Mozilla is misparsing To: header.
grammar from rfc 2822 is following:

group           =       display-name ":" [mailbox-list / CFWS] ";"
                        [CFWS]
display-name    =       phrase

Comment 1

[Kari Hurtta]

Attachment: Mail source

Whole mail source is attached so you can try it yourself.

Comment 2

[Kari Hurtta]

Attachment: Picture of mail window

From this picture you see how To: field is shown.

Comment 3

[Ashley Bischoff (blog at handcoding.com)]

Reporter: Your build is old. Please try a more recent build:
http://ftp.mozilla.org/pub/mozilla/nightly/latest/mozilla-i686-pc-linux-gnu-sea.tar.gz
(as always, be sure to delete your old Mozilla directory before installing the
new one)

Comment 4

[Kari Hurtta]

Bug occurs also with
http://ftp.mozilla.org/pub/mozilla/nightly/latest/mozilla-i686-pc-linux-gnu-sea.tar.gz


(And with BUILD 200111708 I do not have able to attach picture ....)

Comment 5

[Kari Hurtta]

Attachment: Mail window from build 200111708

( I backed to mazilla 0.9.5 so I can attach that picture.
  I reported attachment problem as bug 110615)

Mail window on build 200111708 looks same.
In other words To: is still misparsed.

Comment 6

[Ashley Bischoff (blog at handcoding.com)]

Old summary:
"mozilla 0.9.5 shows incorrectly phrase from group syntax"

New summary:
"Bad parsing between "To" separators"

No dupes found, marking NEW.

Comment 7

[Kari Hurtta]

Attachment: Mail window from Build ID 2001122108

Happens also with mozilla 0.9.7

Comment 8

[scottputterman]

reassigning to bienvenu

Comment 9

[David :Bienvenu]

libmime does the header parsing. 4.x displays this header as is, i.e., Sonera
Internet-asiakkaat:;

Comment 10

[John G. Myers]

Old summary: Bad parsing between "To" separators
New summary: Bad parsing of header group syntax

Comment 11

[Kari Hurtta]

Attachment: Mail window from build 2002031312

Happens also with mozilla 0.9.9 (build id 2002031312)

Comment 12

[Vadim Berezniker]

*** Bug 103817 has been marked as a duplicate of this bug. ***

Comment 13

[Vadim Berezniker]

this bug is already filed as bug 83516
the other bug is clearer and has quote form the rfc.

i'm afraid to mark this is a duplicate since this is already ASSIGNED
should this be marked as a dupe or should it be vice-versa?

Comment 14

[Boris 'pi' Piwinger]

Adding some information from bug 83516, which I will dupe.

pi

Comment 15

[Boris 'pi' Piwinger]

*** Bug 83516 has been marked as a duplicate of this bug. ***

Comment 16

[Mike Cowperthwaite]

*** Bug 156588 has been marked as a duplicate of this bug. ***

Comment 17

[Mike Cowperthwaite]

*** Bug 169493 has been marked as a duplicate of this bug. ***

Comment 18

[Mike Cowperthwaite]

Symptom copied from the report at the new duplicate:

Today I received a message with the following recipent list (2 Contacts):
"To: Kuno Meyer <[e-mail]>;[e-mail]"
Here is, what Mozilla recognizes and displays in the message pane of the
mailnews window (1 Contact):
"Kuno Meyer ;[e-mail] <[e-mail]>"

I was able to easily replicate this symptom, using 
  Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516

The info from recent dupe, bug 156588, is also worth investigating when working 
on this bug.

Comment 19

[Kari Hurtta]

Comment to #18:

If mail header really was
   To: Kuno Meyer <[e-mail]>;[e-mail]

then that is not valid syntax. So is Bug 169493 really then duplicate ?
On given header there is not group phrase. It should be

   To: something:Kuno Meyer <[e-mail]>;[e-mail]

to make it legal. At least starting ':' is missing (if phrase of
group is allowed to be emtpy.)

Comment 20

[Kuno Meyer]

I'm the reporter of Bug 169493 mentioned above.

The issue reported by me has nothing to do with groups treated here, but has to
do something with parsing a list of mail adresses.

So a recipient list of "To: me <me@domain>;another@domain" (please note the
missing space after the semicolon, as I assume) is treated as one single contact
with

name:           "me ;another@domain"
mail-address:   "me@domain"

Comment 21

[Kari Hurtta]

(In reply to comment #19  --- correcting a my own comment :-))
> Comment to #18:
> 
> If mail header really was
>    To: Kuno Meyer <[e-mail]>;[e-mail]
> 
> then that is not valid syntax. So is Bug 169493 really then duplicate ?
> On given header there is not group phrase. It should be
> 
>    To: something:Kuno Meyer <[e-mail]>;[e-mail]
> 
> to make it legal. At least starting ':' is missing (if phrase of
> group is allowed to be emtpy.)

Actual on #18  there also ',' missing.  Grammar is:



address         =       mailbox / group

mailbox         =       name-addr / addr-spec

name-addr       =       [display-name] angle-addr

angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr

group           =       display-name ":" [mailbox-list / CFWS] ";"
                        [CFWS]

display-name    =       phrase

mailbox-list    =       (mailbox *("," mailbox)) / obs-mbox-list

address-list    =       (address *("," address)) / obs-addr-list


So to make 
     To: Kuno Meyer <[e-mail]>;[e-mail]

correct also ',' need to be added. That is:

To: Something: Kuno Meyer <[e-mail]>;,[e-mail]


';' is not address list separator. It is group terminator.
Space are not significant on syntax.

Comment 22

[Boris 'pi' Piwinger]

>So to make 
>     To: Kuno Meyer <[e-mail]>;[e-mail]
>correct also ',' need to be added. That is:
>To: Something: Kuno Meyer <[e-mail]>;,[e-mail]
>';' is not address list separator. It is group terminator.
>Space are not significant on syntax.

How can you have ; without :?

pi

Comment 23

[Kari Hurtta]

(In reply to comment #22)
> >So to make 
> >     To: Kuno Meyer <[e-mail]>;[e-mail]
> >correct also ',' need to be added. That is:

> >To: Something: Kuno Meyer <[e-mail]>;,[e-mail]
                ^ (1)                           ^ (2)
> >';' is not address list separator. It is group terminator.
> >Space are not significant on syntax.
> 
> How can you have ; without :?

Note (1)   -- I put here that ':'

On original problem 169493(which I suspect that it is NOT duplicate of this
110605), there was ':' also missing also -- it ';' was used as address 
separator instead of ','. 

Comment 25

[Wayne Mery (:wsmwk)]

TCW does this reproduce for you?

Comment 26

[Arthur K. (he/him)]

(In reply to Wayne Mery (:wsmwk) from comment #25)

TCW does this reproduce for you?

Honestly, no. I've never gotten an email formatted as "To: <;>" or something similar and I searched through 20 years of email I still have. I checked against what's also mentioned in bug 607947. Some of what's mentioned in bug 83516 is valid and maybe some email processing systems (sendmail, etc.,) were out of spec (or someone kludged the spec) back 15+ years ago and allowed this to fly but today, not possible. We use a couple single-email-to-multiple-recipients here at my org and it always comes through as "To: <single-email-to-multiple-recipients>".

If there's a newer STR, I'd like to try it. But reading through some of the other steps and info from older comments above and the other associated bugs, this just doesn't seem valid anymore.

Comment 27

[Kari Hurtta]

I've never gotten an email formatted as "To: <;>"

Well, that was not about that ';' is inside of '<' and '>' characters.

I quoted syntax on comment #21

Group syntax is

       group name :   list of addressess ;

Special case, where list of addresses is empty, is common.

I rarely use group syntax where list of addresses is non-empty.

Many systems seems misparse them.

Another thing is that someones try use ; as address separator.
That ; is not address separator.

Some systems seems remove group syntax, so that IMAP client does not see it either.

/ Kari Hurtta

Comment 28

[Arthur K. (he/him)]

(In reply to Kari Hurtta from comment #27)

I've never gotten an email formatted as "To: <;>"

Well, that was not about that ';' is inside of '<' and '>' characters.

I quoted syntax on comment #21

Group syntax is

       group name :   list of addressess ;

Ok, I must have misunderstood.

Comment 29

[James Ralston]

I just now tested Thunderbird 60.8.0 ESR, and it appears to handle RFC5322 group constructs correctly.

For this test message:

From: James Ralston <[e-mail]>
To: group-test: <[e-mail]>;
Subject: test of groups syntax
MIME-Version: 1.0
Content-Type: text/plain

This is a test message; please ignore.

…Thunderbird shows a To: header of Me <[e-mail]>. That is, it realized that there was a single address in the To: header, and realized that it was my own address.

For this message:

From: undisclosed-recipients:;
To: group-test:;
Subject: group test
MIME-Version: 1.0
Content-Type: text/plain

This is a test message; please ignore.

…if I execute the reply function, in the composition window, the To: header is empty. And the quoted message is:

On 2019-07-22 11:17, wrote:
> This is a test message; please ignore.

Meaning, Thunderbird treated the To: header as effectively being empty.

Perhaps if I performed exhaustive tests of every possible RFC5322 group constructs variant, I could find a corner case where Thunderbird misbehaves. But from this brief testing, Thunderbird appears to handle RFC5322 group constructs correctly.

Comment 30

[Arthur K. (he/him)]

(In reply to James Ralston from comment #29)

I just now tested Thunderbird 60.8.0 ESR, and it appears to handle RFC5322 group constructs correctly.

For this test message:

From: James Ralston <[e-mail]>
To: group-test: <[e-mail]>;
Subject: test of groups syntax
MIME-Version: 1.0
Content-Type: text/plain

This is a test message; please ignore.

…Thunderbird shows a To: header of Me <[e-mail]>. That is, it realized that there was a single address in the To: header, and realized that it was my own address.

For this message:

From: undisclosed-recipients:;
To: group-test:;
Subject: group test
MIME-Version: 1.0
Content-Type: text/plain

This is a test message; please ignore.

…if I execute the reply function, in the composition window, the To: header is empty. And the quoted message is:

On 2019-07-22 11:17, wrote:
> This is a test message; please ignore.

Meaning, Thunderbird treated the To: header as effectively being empty.

Perhaps if I performed exhaustive tests of every possible RFC5322 group constructs variant, I could find a corner case where Thunderbird misbehaves. But from this brief testing, Thunderbird appears to handle RFC5322 group constructs correctly.

James,

Thanks for testing and validating the behavior. I agree that with enough testing, there's likely to be one or two obscure outliers. I would surmise though that given this bug's age, much of what's need to be in compliance with the spec has happened over that period of years. Wayne or others might chime with some other ideas as to what might trip an issue with the spec.

Comment 31

[James Ralston]

(In reply to Arthur K. from comment #30)

I agree that with enough testing, there's likely to be one or two obscure outliers.

For all practical purposes, the only time when RFC5322 group constructs are used is when they are empty; that is, the undisclosed-recipients:; case. If Thunderbird handles that case correctly, then even if there are one or two obscure outliers, I doubt anyone will ever find them in normal use.

The only potential bug I can find is that Thunderbird does not permit composing messages directly to RFC5322 group constructs.

Specifically, in the composition window, if I attempt to use undisclosed-recipients:; as a recipient address, Thunderbird gives me this error message when I attempt to send the message:

undisclosed-recipients:; <> is not a valid e-mail address because it is not of the form user@host. You must correct it before sending the e-mail.

Attempting to send to group-test: <[e-mail]>; results in this error message:

An error occurred while sending mail. The mail server responded:
5.1.1 <"undisclosed-recipients: username"@example.org>... user or mailing list unknown.
Please check the message recipient ""undisclosed-recipients: username"@example.org" and try again.

But I would argue that it is not worth the effort to support RFC5322 group constructs in composition, because equivalent functionality is available:

1. For the empty group case, where you want the To: header to be present but effectively empty, just omit the To: header from your message. When Thunderbird sends your message, it will add a To: header to it, with a value of undisclosed-recipients:;.
2. For the non-empty group case, just enumerate the individual addresses.

So, in summary:

• Thunderbird correctly handles RFC5322 group constructs when parsing messages.
• While Thunderbird does not permit composing a message directly to an RFC5322 group construct, if the user sends a message that lacks a To: header, Thunderbird will add a To: undisclosed-recipients:; header to the message in order to avoid having a message without a To: header. (This is important, because many MTA will synthesize a To: header if one is not already present in the message.)

I don't use SeaMonkey, so I can't easily test it, but if it is using the same MailNews Core engine that Thunderbird is using, I would expect its results to be the same.

Comment 32

[Kari Hurtta]

Wayne or others might chime with some other ideas as to what might trip an issue with the spec.

Perhaps test case where there several addresses on group and also addresses outside of group and several groups. Partially from comment #21

To: First group: Kuno Meyer <[e-mail]>, Second User <[e-mail]>;, Full Name <[e-mail]>, Second group: Other User <[e-mail]>;

First group
Kuno Meyer
Second User
Full Name
Second group
Other User

need parse as phrase !

Groups are separated by , when that , also used for for separating of addresses

( Inside of < > that , and : are used on route addresses (deprecated) )

Note that on my example makes both ; and , when there is other addresses or groups after end of group

On your test you was using
group-test
when my original report consist case where group name is phrase with several words

From: Sonera Plaza Oy <[e-mail]>
To: Sonera Internet-asiakkaat:;

On comment #29 there is no case where group phrase parsing is tested.

/ Kari Hurtta

Comment 33

[Magnus Melin [:mkmelin]]

The groups are shown correctly nowadays. There are some minor bugs around the recipient related functionality (context menu, contact star) and such though.