54540 - Mail generates illegal news message-IDs

Status: VERIFIED DUPLICATE of bug 2935

(MailNews Core :: Backend, defect, P3)

Description

[Simon Fraser [no longer active]]

Mozilla mail seems to use the domain part of the users's email address to form 
the message-ID, but fails to verify that this is indeed a good domain. Here's an 
example of a bad one:

Path: secnews.netscape.com!not-for-mail
From: nospam@nospam
Newsgroups: netscape.public.mozilla.xpcom
Subject: XPCOM as de facto standard cross platoform component model?
Date: Thu, 28 Sep 2000 12:29:01 -0700
Organization: Another Netscape Collabra Server User
Lines: 12
Message-ID: <39D39BFD.8080608@nospam>
NNTP-Posting-Host: 12.39.176.22
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; m18) Gecko/20000926
X-Accept-Language: en

This is contrary to the recommendations in RFC 1036, which state that (section 
2.1.5):


    In order to conform to RFC-822, the Message-ID must have the format:

                          <unique@full_domain_name>

Comment 1

[Simon Fraser [no longer active]]

RFC Son-of-1036 states (section 5.3)

          The domain in the message ID SHOULD  be  the  full  Internet
          domain name of the posting agent's host.  Use of the ".uucp"
          pseudo-domain (for hosts registered in the UUCP maps) or the
          ".bitnet"  pseudo-domain  (for Bitnet hosts) is permissible,
          but SHOULD be avoided.

See also <http://www.jwz.org/doc/mid.html>

Comment 2

[(not reading, please use [e-mail] instead)]

taking from mscott.

Comment 3

[(not reading, please use [e-mail] instead)]

here's a patch to fix part of the problem.  if the user cancels out of the
account wizard, we set their email to nospam@nospam.  (that's how nospam ends up
as the FQDN)

this patch uses the domain attribute of nsIUserInfo when contructing a fake
email address.

for example, on my linux box it generates [e-mail]

on some platforms (OS/2 & Win), nsUserInfo::GetDomain() is not implemented, and
will throw an exception.  until we fix those, we'll use [e-mail]

adding doug and mike to the cc list.

Comment 4

[(not reading, please use [e-mail] instead)]

Attachment: patch (veto'd)

Comment 5

[Doug Turner (:dougt)]

fix looks okay. 

Comment 6

[Simon Fraser [no longer active]]

> here's a patch to fix part of the problem.  if the user cancels out of the
> account wizard, we set their email to nospam@nospam.  (that's how nospam ends
> up as the FQDN)

That's really evil. If they cancel out of the account setup wizard, you should 
never make the account.

If you do want to allow users a nospam email address, then it should use the TLD 
of "invalid", so it should be [e-mail]. ".invalid" is starting to be 
recognized as a non-real TLD, thus avoiding load on servers which 
unintentaionally match fake, nospam email addresses. [Imagine my surprise when I 
discovered that someone had registered bollocks.com, which I tried as a nospam 
address.]

Comment 7

[Simon Fraser [no longer active]]

Even with those comments, I think that you need to more careful in generating the 
message-ID. Message-IDs MUST BE guaranteed GLOBALLY UNIQUE for some reasonable 
length of time. Therefore you should uniquify each part of the message-ID, based 
on user data. If there are a ton of windows users out there using nospam.nospam 
domains, then you've immediately lost a lot of uniqueness.

I stress again that you should read <http://www.jwz.org/doc/mid.html>.

What I've done in another product is to generate the message-ID domain part based 
on a reverse lookup of the news server address (so that, for example, posting to 
news.mcom.com generates a message-ID like 
  <[e-mail]>
). If the reverse lookup fails, and you are left with an IP address, you have to 
format it with [], like
  <sfraser-E78A55.12274206112000@[192.168.1.100]>

Getting message-ID generate correct is an important think. I'd like to see this 
fixes the correct way.

A related issue is how we handle References header truncation. Guess I'll test 
that another day.

Comment 8

[timeless]

the patch looks good, although the indentation is off.

Comment 9

[timeless]

Sorry, ignore my comment.

Comment 10

[Simon Fraser [no longer active]]

I veto the patch  ;)

Talking to sspitzer, we agreed that this must be fixed the correct way.

Comment 11

[John Moreno]

I came across this bug while trying to make sure that I wouldn't duplicate 

a RFE bug, but here seems a good place to mention it:

<http://www.newsreaders.com/tech/draft-ietf-usefor-msg-id-00>

Has some guidelines for generating a message-id, including what I think 

is a very valuable feature -- id token's for the poster.

By including an identifying token in the message-id, followups to a 

particular poster can be more easily tracked (for either kill, tag, or score 

files).

Comment 12

[(not reading, please use [e-mail] instead)]

I've got a patch, I'll attach it for review.

this patch will also fix 2935.

thanks to sfraser for the continuing help to make sure I do this right.

Comment 13

[scottputterman]

does the patch fix this?

Comment 14

[(not reading, please use [e-mail] instead)]

the attached patch isn't good.

I've got a partial patch, but it isn't complete.

setting milestone to mozilla 1.0

Comment 15

[scottputterman]

changing milestone to unknown.  It will get changed back when we figure out what
milestone to put this bug in.

Comment 16

[scottputterman]

marking nsbeta1-

Comment 17

[basic]

Changing platform

Comment 18

[[e-mail] (gone - use [e-mail] instead)]

*** Bug 80819 has been marked as a duplicate of this bug. ***

Comment 19

[Frank Schönheit]

Hi,

what is the status of this? I found three bugs in this area:
bug 2935 "RFE to stop generating unsafe news Message-IDs"
bug 54540 "Mail generates illegal news message-IDs" (this one)
bug 80819 "Mozilla News Client does create Message-Ids"

It would be very good to have a fix for the problem in 1.0. However it is fixed,
if we chose the long or the short solution (generate valid ids or do not
generate ids at all): this fix is (I'd like to say urgently) needed.

I belong to the group mentioned in bug 2935: I have (amonst others) an account
@gmx.de. This means that every time a send a message I must reckon that it gets
dropped because the message id generated by Mozilla simply may be ambiguous.
Just as my very personal opinion, this is near to making Mozilla unusable for me.

I suggest adding the user preference suggested in bug 80819 which turns message
id generation off, and target bug 80819 for 1.0. After this, if we really think
that the _client_ needs to generate message ids (I'm not that deep in the RFC's
here, but I know that there are different opinions about it :)), we can fix 54540.

Frank

Comment 20

[Ben Bucksch (:BenB)]

Frank Schönheit, how often did this happen to you (msg dropped due to dup
msgid), both in absolute and relative terms?

Comment 21

[Frank Schönheit]

Ben,

I started using Mozilla two weeks ago (my previous "favourite mail/news client"
is EOL :( ). So I admit it did never happen to me. But I already got two mails
from people which I communicated with saying "Your mail client generates invalid
message ids - please stop this because ....".
I admit I really don't know if it is a real life problem, I just have been told
(and read somewhere in one of the three bugs) that it may be. And bug 80819
seems to propose a good first shot solution (disable message id generation).

Comment 22

[Holger Metzger]

Ben,
go read the German usenet hierarchy. News clients with invalid message-ids are 
*not* welcome. If you are lucky you get warned twice ("Your message-id is 
invalid, fix it or welcome to my score file."), if you have bad luck no one 
reads you from the start (*.gmx.de, *.web, *.yahoo.com, *.netscape,net make easy 
filtes). This is truly an important issue (and a very old/almost historical bug 
from NC 4.x days) for me as well, it is one of the things that should get fixed 
to get Mozilla taken seriously, which is not easy from the start -  in the 
de.comm.software.newsreaders FAQ Mozilla / Netscape 6.x isn't even listed!

The German usenet has strict rules, you might call it "typically german". But 
because German usenet adheres to the good old usenet rules and technical 
standards it's the most *legible* hierarchy in usenet.

- Holger

Comment 23

[Martin Sch]

See my comment in bug 2935. Why does mozilla generate the message id if the
source code says s.th. different?

Comment 24

[Frank Schönheit]

What about an (optional) FQDN in the preferences? We could add a (per-identity)
setting which contains such a FQDN, and is used if (and only if) not empty.
Would be at least a workaround for all those who are experienced enough. Of
course we would still need a "real" solution for the users who do not want to
bother with thinking about an own valid FQDN or modifying preference files.
Would be relatively easy to implement, as somebody in the german usenet found
out (yes, Holger, you're right, the german users are the most nitpicking ones -
for good reasons :)

Frank

Comment 25

[Franz G. Koehler]

the fqdn should be obtained from the system's general setup.

I think gethostbyname(3) is what we are looking for.

Comment 26

[Michael Lefevre]

that (comment 25) is a good idea only when the machine knows what its FQDN is.

many machines (particularly joe user's Windows desktop) don't know their own
names - Outlook Express, for example, uses the hostname, and so often produces
message IDs with unqualified host names - in many cases that name is the default
set by the OEM, so you have millions of machines worldwide all calling
themselves "oemcomputer".  taking the domain from the email address is a better
solution in many cases... maybe moz could get the hostname, do some kind of
bogusness test, and use it if it looks like a FQDN?

Comment 27

[Franz G. Koehler]

in my opinion, mozilla should use gethostbyname and provide an override value,
if desired by user.

Comment 28

[Frank Schönheit]

Franz,

"yes" about the override value, "not sure" about the gethostbyname. We would
replace one wrong solution with another wrong solution, wouldn't we?

Comment 29

[Sascha Claus]

Newer versions of Forte Agent use the domain 4ax.com for generating message-ids.
Mozilla might copy this and use e.g. mid.mozilla.org as FQDN if nothing else is
specified.

Comment 30

[Franz G. Koehler]

"We would
replace one wrong solution with another wrong solution, wouldn't we?"

No, we wouldn't, since gethostbanyme is the function designed for getting the
hostname.

btw: there is another bug in mozilla mail concerning the HELO command, mozilla
pretends to be whatever the user's domain part is, this is nearly alway wrong,
we should use the system's hostname here, too.


about forte agent:

using a hardcoded value as the domain part in my opinion is broken by design,
forte agend compensates this by calcculating the user's serial number into the
local part, but mozilla users (fortunately:-) do not have a serial number.

Comment 31

[Frank Schönheit]

But if the machine does not know it's name, then gethostbyname will fail. That's
the problem here.
If Mozilla is (also) targeting at all the millions of users who never heard
about hostnames and such "technical babble", and who especially work on windows
(and there are enough of them I think), then gethostbyname is a wrong solution,
too, because it will fail on all these machines.

Comment 32

[Karsten Düsterloh]

Windows - by default - doesn't know its correct hostname or has a default entry
like "oem computer", and there is *no* way of getting one. 
We have make a guess.
- read the hostname
- apply some (simple) validity checks to the hostname ( "valid_name.valid_tld"?
etc.) (Maybe we can check harder if we have an internet connection, but this
will not always be the case.)
- if hostname invalid:
  - we cannot take the user's mail domain, because it usually does *not* belong
to him (it may be his ISP or employer etc.). Fortunately, the *whole* mail
address should be unique - otherwise he wouldn't get mail anyway... ;-) So we
can try to build a more valid MID by adding the userpart to the MID:

user mail address: [e-mail]
Message-ID: <[e-mail]>

(There's still, of course, the problem of user addresses [e-mail]
and [e-mail], but because we can't (? AFAIR!) use more than one @
in the MID, this will as good as we can get in guessing...)

Comment 33

[Franz G. Koehler]

> But if the machine does not know it's name, then gethostbyname will fail. That's
> the problem here.

That is, what the ovveride value is for.

> If Mozilla is (also) targeting at all the millions of users who never heard
> about hostnames and such "technical babble", and who especially work on windows
> (and there are enough of them I think), then gethostbyname is a wrong solution,
> too, because it will fail on all these machines.

There must be a similar function on windows, because outlook express
does behave exactly this way.

Comment 34

[Michael Lefevre]

outlook express doesn't generate message-id for news messages at all (as I
understand bug 80819, mozilla doesn't either now unless you also send copies of
a news message by email, or set a pref).  on the email side, Outlook Express
does use that function - and as a result it suffers from the problem described.
from a quick check of our email log (covering thousands of emails), I can find
only 2 emails where OE has got it right, while there are messages from about 30
different addresses, all with a message-ID ending "@oemcomputer"

the problem is not that the function does not exist, the problem is that it
returns useless information in nearly all cases on Windows (a hostname but not a
FQDN), because Windows machines simply don't know their FQDN (with the exception
of some, but not all, Windows servers).  so it's not a good solution on Windows...

Comment 35

[Franz G. Koehler]

michael:
i was talking about oe's mail function.

every system is supposed to have an unique fqdn, however, we do know that in
practice this often is theory (expecially on windows boxes).

nevertheless, we should not discriminate people, who set up their boxes
correctly, therefore, gethostbyname is a good point to start with (at least as
good as mozilla's currently behaviour).

machines without a proper fqdn will have to set an override value, but this is
the user's fault anyway, we cannot provide a general solution to a problem, that
has been caused by the user's incompetence or lack of knowledge.

Comment 36

[Frank Schönheit]

submitted bug 178038 ("[RFE] allow for preference to override FQDN for
message-id generation"). This would not really fix this bug here as it would be
a solution for advanced users only, and requires manual add of a preference, but
I think it's related to this one here ....

Comment 37

[Ben Bucksch (:BenB)]

Seems to be a dup.

*** This bug has been marked as a duplicate of 2935 ***

Comment 38

[Ben Bucksch (:BenB)]

(ops, I missed that the bugs were already cross-referenced. Sorry, if the dup
was in error.)

Comment 39

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified dup