137742 - Multiple To and CC headers are not sent when using Long email addresses

Status: VERIFIED FIXED

(MailNews Core :: MIME, defect)

Description

[Ludovic Dubost]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0rc1) Gecko/20020415
BuildID:    2002041513

When long email addresses are used (Ludovic Dubost <[e-mail]>)
instead of simple email addresses ([e-mail]) with multiple To and/or
CC recipient, then the second one of the To and the CC are not added to the
headers.. However the email is sent to all people (so people that are not in the
To or CC receive the email like in a BCC)..

Example emails will follow.. The bug apparently appeared in Mozilla 0.9.8 and is
reproducible in 1.0rc1 on Linux and Windows... The problem does not appear in
Netscape 6.2.1

The email in the sent folder is correctly formated...

Reproducible: Always
Steps to Reproduce:
1. Write an email
2. Add To and CC looking like this:

To: Ludovic Dubost <[e-mail]>
To: Ludovic Dubost <[e-mail]>
CC: Ludovic Dubost <[e-mail]>
CC: Ludovic Dubost <[e-mail]>

3. Send the email..

Actual Results:  Message received looks like this:

Return-Path: <[e-mail]>
Received: from mail-fr.netvalue.fr ([192.168.1.18]) by mail.netvalue.fr
          (Netscape Messaging Server 3.6)  with ESMTP id AAA4A33;
          Tue, 16 Apr 2002 14:04:02 +0200
Received: from netvalue.com ([192.168.1.102]) by
          mail-fr.netvalue.fr (Netscape Messaging Server 4.01) with ESMTP
          id GUNTIP00.OBV; Tue, 16 Apr 2002 14:04:01 +0200 
Message-ID: <[e-mail]>
Date: Tue, 16 Apr 2002 14:04:01 +0200
From: "Ludovic Dubost" <[e-mail]>
Organization: Netvalue ( http://www.netvalue.com )
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0rc1) Gecko/20020415
X-Accept-Language: fr, en
MIME-Version: 1.0
To: Ludovic Dubost <[e-mail]>
CC: Ludovic Dubost <[e-mail]>
Subject: Test email to show missing headers
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit


Test email to show missing header



Expected Results:  Something like:

Message-ID: <[e-mail]>
Date: Tue, 16 Apr 2002 14:04:01 +0200
From: Ludovic Dubost <[e-mail]>
Organization: Netvalue ( http://www.netvalue.com )
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0rc1) Gecko/20020415
X-Accept-Language: fr, en
MIME-Version: 1.0
To: Ludovic Dubost <[e-mail]>, Ludovic Dubost
 <[e-mail]>
CC: Ludovic Dubost <[e-mail]>, Ludovic Dubost
 <[e-mail]>
Subject: Test email to show missing headers
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit


Test email to show missing header




All 4 people have actually received the email..
Look at the To and CC headers in the results:

To: Ludovic Dubost <[e-mail]>
CC: Ludovic Dubost <[e-mail]>

It should be:

To: Ludovic Dubost <[e-mail]>, Ludovic Dubost
 <[e-mail]>
CC: Ludovic Dubost <[e-mail]>, Ludovic Dubost
 <[e-mail]>

This problem is a corporate stopper as it defeats all filtering and email
exchanged inside the company.. This is actually a problem for me and I will need
to downgrade my version of Mozilla if there is not a solution to this problem...

I consider this is loss of data as people don't receive the email that they
should...

Comment 1

[Ludovic Dubost]

Correction:

The bug does not appear in Mozilla 0.9.8 and neither in 0.9.6

Comment 2

[Ludovic Dubost]

I believe this bug should be fixed for 1.0.. So I'm adding the mozilla1.0 keyword..

Comment 3

[Vadim Berezniker]

rc1 is not out yet.
you're running builds from the rc1 branch.

I tried reproducing this by sending an email adressed to 4 email addressed: 2 TO
and 2 CC and it came ok.

Are you sure this is not a MTA problem?
Can you send it in other clients and receive it properly formatter?

Comment 4

[Ludovic Dubost]

It only happens with certain versions of Mozilla.. It does not happen with
Mozilla <= 0.9.8 with the same MTA.. The problem also happens using RC1..

It could be a combined pb.. Mozilla + certain MTA.. Our MTA is a Netscape
Messaging 3..

When you are trying to reproduce are you sure you are putting email addresses
and a full name (Ludovic Dubost <[e-mail]>) .. Because the problem
only happens in this case.. not if the email address only contains a standard
email ([e-mail])

Comment 5

[Thierry Carrez]

I confirm the problem.
Happens using Mozilla >= 0.9.9 with MTA = Netscape Messaging Server 3.66
Must be something related to To: and CC: fields composition which changed
between 0.9.8 and 0.9.9.

Seems only to happen with long names with spaces in them like
John Doe <[e-mail]>
but not with long names without spaces like
Doe <[e-mail]>

I'll check that it doesn't happen if you use another MTA and that reverting to
0.9.8 will solve the problem.

I suggest the bug is reopened so that I can be in the 1.0.0 scope...

Comment 6

[Vadim Berezniker]

confirming based on Thierry's comment.

please provide the information you mentioned when you can

Comment 7

[Thierry Carrez]

Confirming this to be a regression : version 0.9.8 is working correctly.

A few of my test cases :

Sending with Mozilla >= 0.9.9 an email with :
To: John Doe <[e-mail]>
To: Jane Doe <[e-mail]>
-> message will be received by John and Jane
-> headers will just show John
-> every email client will just show John
-> some will show an incomplete To: line like "To: John Doe <[e-mail]>,"
-> INCORRECT BEHAVIOUR

Sending with Mozilla = 0.9.8 the same email :
-> message will be received by John and Jane
-> headers will show John and Jane
-> CORRECT BEHAVIOUR

Sending with Mozilla >= 0.9.9 an email with :
To: John Doe <[e-mail]>
To: Jane <[e-mail]>
-> message will be received by John and Jane
-> headers will show John and Jane (!)
-> CORRECT BEHAVIOUR

So the problem seems to be conditionned by :
- using multiple To: or Cc: addresses
- using long names with spaces in them
- using the Netscape Messaging Server MTA (I have to check that one)

I will check that behaviour is correct with Exim instead of NMS, but I think it
will be : must be the reason why you don't manage to reproduce when you're not
using NMS.

Something must have changed in the way To: or Cc: addresses are submitted
between 0.9.8 and 0.9.9 that confuses NMS (but sending is correct, it's just the
headers that are incorrect...)

Comment 8

[Thierry Carrez]

Informative network captures about differences between 0.9.8 and 1.0RC1 :

(1) Moz 0.9.8 writes the multiple To: header in the DATA like this :
To: John Doe <[e-mail]>,[0D][0A][09]Jane Doe <[e-mail]>[0D][0A]
and it works properly with NMS delivery

(2) Moz 1.0RC1 (and 0.9.9) writes it this way :
To: John Doe <[e-mail]>, Jane Doe[0D][0A] <[e-mail]>[0D][0A]
and it doesn't work properly.

(3) One-word names are written OK using 1.0RC1 :
To: John Doe <[e-mail]>, Jane <[e-mail]>[0D][0A]
and it works properly.

I think the 0.9.8 line is buggy but tolerated. It must have been corrected in
0.9.9 but there is still a problem with multiple-words in names (see trace (2)
above) which confuses NMS and might confuse others.

I think it should be corrected before the 1.0 final release since other mail
servers than NMS might be affected by the buggy To: line...

Comment 9

[Thierry Carrez]

I confirm this is not a NMS-specific bug.

The "To:" and "Cc:" (and probably others like "Bcc:") headers are written
incorrectly since Moz 0.9.9 :
- when you use multiple addresses
- when one of the names used (but not the first) is a multiple-word name like
"Jane Doe <[e-mail]>"

The part about NMS is that it rewrites the headers before delivering the
message, and the parsing of the wrongly-written headers fails, but this is
because there is a bug in Mozilla, not in NMS.

Reproduced on Linux and Win32.

Comment 10

[Vadim Berezniker]

I originally tried it with the conditions specified running qmail as the mta and
I did not see this problem. 
So as you said, some maybe more tolerant than others.

Comment 11

[Thierry Carrez]

More info :

The problem is not due to having multiple words in the display name but is due
to having a long name. More precisely, Mozilla 0.9.9 and over can cut the To:
header line between the display name and the address :

To: John Doe <[e-mail]>
To: JohnnieHasAVeryVeryLongName <[e-mail]>

will be cut in two lines between JohnnieHasAVeryVeryLongName and
<[e-mail]> (inserting space) :

To: John Doe <[e-mail]>, JohnnieHasAVeryVeryLongName
 <[e-mail]>

whereas with 0.9.8 and before, it wrote the display name and address on the same
line and will cut between addresses (inserting tab).

To: John Doe <[e-mail]>,
     JohnnieHasAVeryVeryLongName <[e-mail]>

The two behaviours should be accepted by the mail server, but NMS 3.X doesn't
like it and rewrites the headers (why?), dropping addresses from the incomplete
one to the end.

So this could be considered as an NMS bug, but there is no way to work around it
and Mozilla is the only client I know of with this behaviour.

Should this be corrected in Mozilla ? Are there any other mail servers having
problems with the Moz 1.0RC1-style recipient headers ?

Comment 12

[Thierry Carrez]

A last note on this issue :

I think we should change the title of this bug to :

"New recipient headers folding confuses some mail servers"

because the problem can be reduced to where exactly you should fold the
recipient header lines (like To: or Cc:) in case of long header fields.

From RFC #822 :
 Note:  While the standard  permits  folding  wherever  linear-
 white-space is permitted, it is recommended that struc-
 tured fields, such as those containing addresses, limit
 folding  to higher-level syntactic breaks.  For address
 fields, it  is  recommended  that  such  folding  occur
 between addresses, after the separating comma.

Old Netcape Messaging Server versions (like 3.X) expect the "recommended" way of
folding long header lines, and maybe other mail servers as well. Mozilla 0.9.8
was implementing the "recommended" way. Header folding in Mozilla 1.0RC1 is
standard-compliant strictly-speaking, but not the recommended way.

Comment 13

[Ludovic Dubost]

From our testing the problem also occurs using Netscape Messaging Server 4.x 
Does annybody know why the header folding has been changed ?

I think this should clearly be fixed because I'm not sure that even if patches
are created for the servers, corporation would not necessasrly deploy them easily..

This can dramatically reduce the adoption of Mozilla in corporations.. and this
is not what we all want...

Comment 14

[Takayuki Tei]

This is a regression by a fix for #73403.
Change platform and OS to ALL.

Comment 15

[nhottanscp]

Change component to MIME, cc ducarroz, putterman.

So the current known problem is that address may be removed by NMS 3.X?

Comment 16

[Ludovic Dubost]

Yes.. but only in the headers.. the email are actually sent to all people..

Comment 17

[nhottanscp]

>Yes.. but only in the headers.. the email are actually sent to all people..
so there may be a problem when reply all 

cc ducarroz, putterman

Comment 18

[Ludovic Dubost]

Yes.. the typical scenario of what happens is:

- somebody sends an email to multiple people in the company.. With the LDAP
server + rewritting of emails addresses by the server, the email comes in with
long names in the addresses

- reply to all using Mozilla 1.0 misses some addresses in the headers

- recepient receive the email with missing headers.. Filters might not put the
email in the right folder and reply to all will not include some users...

After a few weeks of usage this caused several communication problems for me,
and I had to revert to Mozilla 0.9.8 to avoid the problem...

Comment 19

[Takayuki Tei]

Just tried with NMS 4.15 MTA and works fine with following To: line:

To: Takayuki Tei <[e-mail]>, Johnnie Has A Very Very Long Name Johnnie
 Has A Very Very Long Name <[e-mail]>

Comment 20

[nhottanscp]

I think this has to be evaluated by mail group, adding nsbeta1.

Comment 21

[Thierry Carrez]

Remark on comment #19 :

Since the problem depends on where exactly the line folding occurs, it might
work with short names, not work with long names, work with very long names, etc...

But it is also possible that header parsing has been rewritten in NMS 4.15.

To correct this one, I think there is two possibilities : you can either
implement the RFC822 "recommended" way of folding or revert to the pre-0.9.9
recipient header encoding... Both should be compliant with older versions of NMS.

Like Ludovic, I had to delay Mozilla deployment in my company due to header
information lossage.

Comment 22

[scottputterman]

reassigning to ducarroz.  Jean-Francois, can you look at this?  Marking adt1
until more is known about this.

Comment 23

[nhottanscp]

>Since the problem depends on where exactly the line folding occurs, it might
>work with short names, not work with long names, work with very long names, etc...

Taka, is that also be applied to MIME encoded case which tends to be a long name?
Is that possible to enable the old folding behavior by option?

Comment 24

[Takayuki Tei]

> But it is also possible that header parsing has been rewritten in NMS 4.15.

Which patch version are you using?  There have been bunch of header rewrtting
problem in earlier version.  You should upgrade to latest patch.


> Both should be compliant with older versions of NMS.

We have released patch to collect these problem on server.

Do you know any other server implementation which doesn't work correctly
with folded structure headers?


> Like Ludovic, I had to delay Mozilla deployment in my company due to header
> information lossage.

We recommend you to upgrade to latest patch of NMS.

Comment 25

[Takayuki Tei]

> Taka, is that also be applied to MIME encoded case which tends to be a long name?
> Is that possible to enable the old folding behavior by option?

RFC 2047 "8 Examples" shows folded To: field body with multiple recipients.

RFC 822 "3.1.1.  LONG HEADER FIELDS" also shows folded To: field body with 
multiple recipients.  Since RFC 2047 is designed on top of RFC 822, it's 
impossible to go back to old broken MIME encoding style without seeing
bugs such as #73403.

Comment 26

[Takayuki Tei]

In comment #24:
> We have released patch to collect these problem on server.

I meant "correct" not "collect". 

Comment 27

[nhottanscp]

Attachment: patch from taka

explanation about the patch copied from e-mail:

Although the attached patch should fix the bug 137742, this bug is
fundamentally caused by a NMS3.x bug.

RFC 822 explicitly shows how long headers can be folded, and RFC 2047 is
designed on top of that folding logic.	This patch works only when given
structured field body is in ASCII, therefore, those MTA which cannot unfold
folded To:/Cc: field body can be screwed up when user reply to multiple
receipients with RFC 2047 encoded names.

Comment 28

[Thierry Carrez]

On comment #24 :

> Which patch version are you using?  There have been bunch of header rewrtting
> problem in earlier version.  You should upgrade to latest patch.

I'm using NMS version 3.62. If there is a patch which could correct the bug
appliable to this version, I would be interested to download and install it (I
cannot find anymore NMS 3.X patches on IPlanet site). However, if the only way
to get rid of this is to reinstall a complete NMS (or other mail server) I will
have to delay this (we have plans to change the mail server for Q4 2002) and
delay Mozilla deployment (planned for Q2 2002) accordingly.

> Do you know any other server implementation which doesn't work correctly
> with folded structure headers?

No. I tested with sendmail and exim without any problem.

On comment #27 :

> Although the attached patch should fix the bug 137742, this bug is
> fundamentally caused by a NMS3.x bug.

I agree completely. For Mozilla, this is more a compatibility feature than a
bug. But if NMS 3.X is the only mail server having problems, Mozilla is the only
mail client I know of that triggers the problem... 

I'm available and ready to test any win32 build to validate the problem
is gone. Just tell me in which nightly build it will appear. 

And thanks to you all for supporting all these very-specific demands and coping
with my bad English !

Comment 29

[Yasuo Ohgaki]

I have very simlar problem when I sent message to both NNTP and SMTP. It
sometimes fails. I cannot even see sent messages in Sent folder.
(I use 1.0rc1/Linux and didn't have problem with 0.9.9)

SMTP server is smtp.mail.yahoo.co.jp
NNTP server is news.php.net

I don't send much mails, so I cannot tell if I have the same problem or not. It
seems it's closely related.

Comment 30

[Takayuki Tei]

There is a known bug 137742 which also causes message delivery problem.
Can you give us a couple of examples you couldn't send message as expected?

Comment 31

[R.K.Aa.]

*** Bug 141703 has been marked as a duplicate of this bug. ***

Comment 32

[Jean-Francois Ducarroz]

sofar, no luck, I am not able to reproduce this problem using various version of
Mozilla/Netscape, even most recent builds (branch and trunk)

Comment 33

[Jean-Francois Ducarroz]

The only problem I see here is that when we reformat the headers to comply with
RFC2047, we can potentially fold an address in the middle between the long name
and the email address. Then some MTA might tried to reformat the headers and cut
some of the recipients by accident (bug?). I beleive this problem has been
introduced on February 20, 2002.

Taka proposed patch will fix that problem. Reassign to taka,

Comment 34

[Jean-Francois Ducarroz]

Comment on attachment 81004 [details] [diff] [review]
patch from taka

R=ducarroz

Comment 35

[Scott MacGregor]

Comment on attachment 81004 [details] [diff] [review]
patch from taka

Please make len a PRInt32 instead of a raw int. We should be using the NSPR
types for integers unless the code is platform specific.

Comment 36

[Jean-Francois Ducarroz]

reassign to myself as I have some free cycle...

Comment 37

[Jean-Francois Ducarroz]

Attachment: Proposed fix, v2

Replaced int by PRInt32.

Comment 38

[Jean-Francois Ducarroz]

accepting

Comment 39

[David :Bienvenu]

Comment on attachment 87115 [details] [diff] [review]
Proposed fix, v2

sr=bienvenu

Comment 40

[Jean-Francois Ducarroz]

Comment on attachment 87115 [details] [diff] [review]
Proposed fix, v2

R=ducarroz

Comment 41

[Jean-Francois Ducarroz]

Fix checked in the trunk

Comment 42

[scottputterman]

Esther, can you verify this on the trunk?

Comment 43

[Thierry Carrez]

Correction appears in 2002061208/Win32.
Thanks to everyone...

Comment 44

[esther]

I'll give it a try, but I need to first reproduce it to be able to verify it. It
will take time.

Comment 45

[esther]

I spent a gread deal of time trying to reproduce this and couldn't. I took:
linux mozilla build 0.9.9 
linux mozilla build rv:1.Orc1 with Gecko string 20020417 (couldn't find one with
20020415)
linux commercial branch 20020415 
win32 commercial branch 20020610 
All test conducted using Netscape Messaging Server 4.15 and testing addresses
with really long pretty names with spaces (50+characters),  multiple recipients,
mixing addressing using To: and CC:, I could not reproduce this.
Since reporter has tested this and commments that "correction appears in
2002061208/Win32", I will take a trunk build with the fix and do a spot check to
see if basic sending to mutiple recipients is still working. 

Comment 46

[Takayuki Tei]

You need to test against early version of NMS3.x since this symptom appeared due
to a bug in NMS3.x, not Mozilla.

Comment 47

[Ludovic Dubost]

I've checked it with build 2002061221 and the problem does not occur on my
computer anymore with a NMS 3.x

Great news.. that will probably allow deployment of the next release of mozilla
in our company..

Comment 48

[scottputterman]

Lowering impact since it only happens on one server.

Comment 49

[Jaime Rodriguez, Jr.]

esther - can you pls verify this fix on the branch? thanks!

Comment 50

[Gayatri Rimola]

Esther will not verify that this works with Netscape Messaging Server 3.66 since
we do not actively support that server. However, Esther will verify that this
does not break anything with the servers we do test with, i.e. NMS 6.0, NMS 6.1
and NMS 4.15.

Comment 51

[Jaime Rodriguez, Jr.]

Has Esther verified that this fix does not cause any regressions, with servers
we currently support?

Comment 52

[esther]

I have not done formal testing ye.   I'm testing other regressions that have
come up in the last 2 days that are now fixed on the trunk and waiting for
verification.  But I have been using the trunk builds for these regressions for
daily sending to email address of the same type mentioned in the original
scenario, there doesn't seem to be any problems.  Since the reporter is saying
it's fixed with the trunk build, if this hasn't gone into the branch yet I think
that it could.  I am confused by Jaime's question in comment #49 about verifying
it on the branch?  Is the fix in the branch or trunk or both now?

Comment 53

[esther]

In the past week 6-24 through today 7-1, I have used the trunk builds (this bug
was fixed in trunk on 6-12) and branch builds (I don't see that it was ever
checked into the branch yet) with our NS server 4.15 & 6.0.  I have seen no new
problems with multiple addressees in messages with long names or short names. 
Since we never saw the problem on these servers all I can say is the fix doesn't
appear to have broken anything in the addressing field.  If this was checked
into the branch after 6-27-2002 then I will say this is verified for both trunk
and branch.  If it hasn't been checked into branch, then I will say verified on
trunk. 

Comment 54

[Jean-Francois Ducarroz]

This fix has not been checked in the branch has it has never been approved by ADT.

Comment 55

[scottputterman]

adding adt1.0.1-.  Let's get this in the next release.

Comment 56

[esther]

I tested this scenario again with latest trunk builds using the 4.15 & 6.0
messaging servers, this works fine.  The fix that was checked in for a 4.33
messaging server bug, so I just verified the 4.15 and 6.0 servers still work. 
Since we will be basing future releases off the current trunk build I will
verify this as fixed.  

Comment 57

[Eli Rodriguez]

Did this make it into Netscape 7.02?  I'm seeing this problem with Netscape 7.02
and Netscape Messenger 3.6X.