Open Bug 110389 Opened 23 years ago Updated 2 years ago

storage of "From" in messages may cause invalid S/MIME signatures

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Type:
defect

Tracking

(Not tracked)

People

(Reporter: julien.pierre, Unassigned)

References

Details

The client uses the string "From" internally to delimit messages within its mail
database. In many cases, a ">" character gets inserted into the file if a line
of message text begins with "From" to prevent the client from interpreting it as
the beginning of a new message when reading from disk.

In many cases - but not all - the > actually ends up being displayed to the user
in the UI. This means the message is not shown as it was sent, but tampered with.
This would be just a cosmetic (but annoying) problem, except for S/MIME signed
messages.

The tampering that occurs in the file when the message is stored on the disk can
cause the signature check to fail, even if it is valid. I don't have a test case
with Mozilla because I don't have a build with functional S/MIME currently,
however I have a test case with Communicator.

1. go to Outlook (this is necessary so that no message tampering occurs)
2. import your cert
3. send yourself a signed message. Include just one line, without a carriage
return, beginning with the word "From". My test message just contained the words
"From Julien".4. read the message in your Netscape client. In Communicator, an
"invalid signature" icon is displayed. If you do a view/source of the message,
you see that the first line says ">From Julien". This is what causes the S/MIME
signature check to fail.

The fix for this is for the client to use a better mail storage method that does
not tamper with the content of the messages.

I have not checked this with the Mozilla / Netscape 6 S/MIME builds because
again, I don't have one where the S/MIME UI is showing, but I know the mail
storage is the same in Mozilla as it was in Communicator, and I expect this
S/MIME bug to show up. Just today I got a non-S/MIME e-mail from bugzilla where
I had written a sentence beginning with the word "From", and it got mangled as
">From" when I read it in Netscape 6.2 ...
Julien,
Today's build work even though the menu do not show up. You can either:
act as if the menu string were present:
The option->[security] menu (string security missing)
shows a sub menu. The sub menu has two part separtaded by a horizontal sep.
the first part has three radio buttons. Make sure the second one is on for encrypt.
The part below the separator is a check mark. check means to sign.

The string may appear tomorrow.

I still don't have a functional S/MIME browser build. However, I'm now confident
that this problem will show up in it. Just using the smime perl script from the
toolkit to send a signed message containing the string "From Julien" in the body
causes Netscape 6.2 to prepend a > to the From word. This will cause the
signature check to fail.
Depends on: 110541
How can I tell that the signature failed?  I sent myself a signed message with
"From Scott" in the body.  I received the message and it shows up as ">From
Scott".  But I don't see any error message.  The message displays and it says
"signed" in the header area.
Status: NEW → ASSIGNED
Scott,

If the signature is not shown as invalid in my test case, it might be that the
client is not verifying S/MIME signatures at all, but rather just reporting that
the message has a "signed" attribute.  If that's the case it would be extremely
dangerous since it would show tampered messages as simply "signed" ...


Are we going to be verifying signatures?
Now Mozilla verifies the signature. But I don't know if the bug is still there.
Note if you send an signed e-mail directly from Mozilla (with the From in the
message on start of a line) it verifies fine, because Mozilla prepends a space
before the from just before it signs the message.

I don't have outlook so I can't send myself a signed message with From on
beginning of line. If someone can send it from outlook to my email
(tomas.mraz@pvt.cz) please do so.
Blocks: 74157
I have just verified that the bug is indeed still there.
I sent myself a signed e-mail from Outlook.
The message was raw text and simply said on the first line
"From Julien, with love"
(without the quotes).

I retrieved the message in Mozilla. It showed as
">From Julien, with love" . There was a broken pen icon, and when I clicked it, 
it said "Digital Signature Is Not Valid".

I also retrieved the message in Communicator 4.79, and it also displayed as 
">From Julien, with love" and got the "Invalid signature" icon in the upper 
right corner of the message window.

I think this is a showstopper for S/MIME in Mach V. We can not rely on S/MIME 
working if our database knowingly corrupts the message by adding this > 
character to all lines that begin with the word From. The message database 
format must be modified in order for S/MIME to work. Setting priority as P1. 
Please set the target as I'm not familiar with mozilla schedules.
Priority: -- → P1
Bear in mind this only happens on plain text messages, so by default, Mach V
won't send out messages that this happens to.
I am aware of this, but not everyone uses mozilla for mail !
A heck of a lot of people use Outlook. There are probably other S/MIME clients, 
and they may also send the signed messages plaintext. The S/MIME spec allows it, 
and Mozilla doesn't follow it, as it incorrectly considers some messages that 
have a perfectly valid signature to be tampered with.

I'm just trying to make sure everyone is aware of the facts. Also, we will not
be changing our mailbox format for Mach V. It's way too dangerous. The only way
I can imagine this getting fixed is to strip off the > before the From when
passing it to whatever code it is that's trying to calculate the signature -
that's the code that's going to need to change, IMO. I don't know how that code
works, however, so I don't know about the feasibility of that (and how we'll
handle messages that really have >From in them...)
David,

That still won't work. Take the case of the message below. Assume it gets sent 
from Outlook to a mozilla user.

>From what I read, X

From what I read, Y

The above message contains lines beginning with both >From and From .
When the Mozilla user receives it, it gets stored in mozilla as :

>From what I read, X

>From what I read, Y

If you then strip the > before the From it will become :

From what I read, X

From what I read, Y

When you pass that to the signature verification code, the signature check will 
fail. The signature check can only pass if the message has not been tampered 
with between the time the message was signed and the time it is verified.

Your suggestion to strip the > before From is an attempt to undo the tampering, 
unfortunately it doesn't work because at the time the message gets stored in the 
database, you don't know which lines got the > added before From and which ones 
already had it in the first place.
A solution would be perhaps to add a special header in the message in 
the database, call it "Hack-From:". That header would have a list of every lines 
that got a > prepended. Then you would be able to strip only those lines of the 
> before doing the signature verification, and it would work. You could also do 
that stripping before displaying/forward/doing all kinds of other operations on 
the message, so that they no longer appear mangled, even though they are within 
the message store.

Oh, by the way, the message above *must* be read over HTTP on bugzilla, 
otherwise it makes no sense. The e-mail notification message gets mangled in 
Mozilla or Communicator because of this very bug ...

Here is suggestion how to make it work for ALL messages:

Current behaviour:
   Sending plaintext message: Space is inserted before all lines which have
'From' at beginning NOT COUNTING SPACES. (I don't know if it's signed before or
after inserting the space.)
   Receiving plaintext msg: '>' is inserted before all lines which immediately
begin with From
   Verifying and displaying plaintext msg: One space is removed on all lines
which begin with n-spaces and From. (Again I don't know if it's verified before
or after removing the space but it's done so the signature is verified.) The
space is removed only in e-mails sent from Mozilla - it isn't removed on mails
from other clients but I don't know what's the trigger exactly.

Suggested behavior:
   No mangling of messages when sending them, receiving them and so on.
   Storing into local mailbox:
     Insert space before lines which begin with From NOT COUNTING SPACES.
   Retrieving from local mailbox:
     Remove one space from beginning of lines which begin with From NOT COUNTING
SPACES.
The signing is done before storing into a mailbox and the verifying is done
after retrieving it so it's done on unmangled messages and everything should
work fine.
It should be relatively easy to do, maybe the hardest work would be on the
removing of the current partially working kludge solution. :-)

The problem with that is interoperability between versions - we don't know which
version of Mozilla or Mozilla-derived products sent the mail. We could do a
similar fix with escaping of From lines from outlook, except use > instead of
space. So, if the line starts with 0 or more '>' followed by "From ", then add a
'>', and when calculating the signature, if the line starts with 1 or more '>'
followed by "From ", remove 1 '>'. This would only work for newly received mails
that dealt with lines starting with >From but how often do users go back to old
mail and verify signatures. Also, to be clear, this is only for POP3 messages
and not IMAP messages.
Yes. We can leave the old mechanism there as is and add the mechanism I
suggested but using different character - preferably >. This would allow 100%
compatibility with older versions.

The crucial question is if the current kludge mechanism adds the space before
From before it makes the signature or after. I think that the first case is much
more probable because it allows to verify the message fine in other S/MIME
clients. And it then removes it only for display. 
So the only compatibility problem with older Mozilla releases when this
mechanism would be removed would be that with a new Mozilla you would see the
space before From. And in the old Mozilla you wouldn't verify the signature. But
since Mozilla isn't end-user product and since there hasn't been yet released a
product based on Mozilla which would use the S/MIME mail maybe it doesn't matter.

How is the problem solved in NS4.x? I don't have it so maybe there could be
other issues when communicating with NS4.x mail.

Conclusion:

We don't have to remove the old mechanism but it would be surely nice clean up.
But what we need as soon as possible is to change the mechanism which adds the >
character to add it in >....>From cases too and add the code to remove it before
verifying and displaying the message.

Could this make it to mozilla1.0? Probably not :-(


This didn't work in 4.x either, according to jpierre. I don't know where the
"adding of the space" kludge came from, but I'm betting it happens before the
signature is calculated, so it should be working fine. No, I don't think this
will happen for Moz 1.0.
Yes, it was broken in 4.x, and that was an extremely embarassing S/MIME bug. 
Mozilla is the chance to finally fix it. Are we going to take it or do we just 
want to merely match what our previous 4-yr old release did ?

It's shame it didn't get attention a little bit sooner.
Nominating for Mozilla 1.0.1
Keywords: mozilla1.0.1
In this case we're going to merely match what our previous 4-yr old release did.
Target Milestone: --- → mozilla1.2alpha
*** Bug 167232 has been marked as a duplicate of this bug. ***
My signature was
"
----------------------------------------
From Mozilla and GNU/Linux
"
but I changed it to
"
----------------------------------------
Sent from Mozilla and GNU/Linux
"
because someone knew about this bug.
Yeah, I have a work around.

It seems that the escaping of lines starting with the word "From" is not
required by Internet mail standards (<a
href="http://www.faqs.org/rfcs/rfc822.html">RFC 822</a>).

Therefore, escaping is not necessary or standard (although it is used by some
programs, like currently Mozilla Mail for incoming messages and importing). For
example, Eudora uses standard text mailboxes, but does not escape "From" lines
in the message body, because clearly people don't like their "From" phrases
transformed into ">From".

(Personal note from a newcomer: In my opinion, Eudora is good in this point of
no escaping, but of course Mozilla Mail do already better than Eudora in many
other important points, like support for Asian languages, open source, constant
improvement, etc. Now, I'm moving soon from years of use of Eudora to Mozilla.)

A good and standard solution, with "From" start-of-message header lines but
without quoting "From" lines in the message text, is described in the following
link:

University of Washington:
IMAP Toolkit Frequently Asked Questions, 6.12
<a
href="http://www.washington.edu/imap/IMAP-FAQs/#6.12">http://www.washington.edu/imap/IMAP-FAQs/#6.12</a>

In the case that the solution of the above link could take some programming
time, a good temporary solution would be to return to the escaping with a space
used in Mozilla Mail previously (converting "From " into " From " at the start
of lines, in the message body).

This space escaping (" Form ") works perfectly to avoid any confusion with the
start-of-message "Form " lines, and therefore to avoid message fragmentation.
You can verify it editing a message file in Mozilla Mail with a text editor.

It seems that this was unfortunately changed to use ">From " about one year ago
(see http://bugzilla.mozilla.org/show_bug.cgi?id=119441 ). I think this happened
because import tools lacked any "From " escaping, so they added ">From " and
later changed " From " in other parts of the program into ">From ", thinking it
was more standard (it seems that it's not so: see RFCs 822, 976, 1521, 2822...).

Probably it would be very easy and fast to implement the temporary solution,
changing '>' into ' ' in the code for the "From " space escaping, until a
complete solution for Mozilla's standard mbox mailboxes (regularizing the
start-of-message "From " lines), such as the solution of the above link from the
 IMAP Information Center at the University of Washington, can be added later if
approved.
Product: MailNews → Core
Fixing bug 58308 could help this (maildir folders do not need >From escaping)
Assignee: mscott → nobody
Status: ASSIGNED → NEW
QA Contact: esther → backend
Product: Core → MailNews Core
Priority: P1 → --
Target Milestone: mozilla1.2alpha → ---

Does the bug require that another email application sent the message?

I cannot reproduce the bug with a message sent by Thunderbird.

I sent a plaintext message, signed with S/MIME.

We send the plaintext with
Content-Transfer-Encoding: quoted-printable

The F at the beginning of the line is sent as =46

Hmm, I had Enigmail installed. I wonder if the encoding was done by Enigmail.

I tried again without Enigmail.

Now, the sent message contains a single space at the beginning of the line, in front of the F. The F isn't encoded. The space isn't shown when viewing the message as plaintext. The signature is shown as valid.

See Also: → 1719121
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.