292763 - Need a tool that will produce file hashes

Status: RESOLVED WONTFIX

(NSS :: Tools, enhancement)

Description

[Doug Turner (:dougt)]

I need a tool that, given a file, will produce a hash.  I started on this tool
and it may be an interesting addition to the cmd tools/tests.

Comment 1

[Doug Turner (:dougt)]

Attachment: Source for a hashing tool

Comment 2

[Wan-Teh Chang]

Doug, you should accept string hash algorithm names
rather than their numeric values.

I'm wondering if an existing tool like signtool can
already produce the hash of a file.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

bltest will hash a file.  :)

My question is: who is going to use this tool?  
Is it intended to verify mozilla downloads?
Will the intended users already have the tool when they need it?
Is there a recursion problem in getting a known good version of the tool?

Should hashing files be a side effect of downloading them in the browser?
(That is, should the browser hash files as they are downloaded and display
the hashes thereafter?)

Inquiring minds ...  :)

Comment 4

[Doug Turner (:dougt)]

> My question is: who is going to use this tool? 

the mozilla foundation and others interested in generating a hash for a file :-)

> Is it intended to verify mozilla downloads?

the generated hash is intended to verify a downloaded file.

> Will the intended users already have the tool when they need it?

the tool is going to be used during the build process and is not intended client
use.

> Is there a recursion problem in getting a known good version of the tool?

I am not sure what you mean.

> Should hashing files be a side effect of downloading them in the browser?

Yes, and no.  


Basically what we are doing is generating a hash of a file to be download,
sticking the filename and the hash in a manifest, and serving this manifest over
https.  clients can download this manifest which they get over ssl, then
download the file over any channel -- say a mirror'ed website, then after the
file has been downloaded verifiy that the downloaded file has the same hash as
what the manifest said it would.  This probably could be discussed offline, if
you like.  

This tool I attached here would produce the hash at built time.  Mozilla already
has hashing APIs (basically we expose NSS to XPCOM).  I recently created a patch
that will allow these APIs to be scriptable.

I hope this helps -- sorry for the lack of background initally.

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

When users first download some mozilla software, if they care about authenticity
they will need a tool to verify the hash.  What tool will they use?  
How will they verify the authenticity of that tool when they download it?
(That's the recursion problem)

We already have full-blown support for signed zip/jar/xpi files.  In what way, 
if any, is this new mechanism -- which puts hashes (not signatures) into a
separate file that is downloaded from a separate source -- superior to the
existing jar file support?  The Jar/zip/xpi mechanism keeps the verifiable 
and unalterable signature together with (in) the download, requires only one
download, and verification is (or can be) combined with extraction.  Further
IINM, it's already built in to mozilla.  

I'd rather not see mozilla re-invent this wheel, again.

Comment 6

[Darin Fisher]

> When users first download some mozilla software, if they care about authenticity
> they will need a tool to verify the hash.  What tool will they use?  
> How will they verify the authenticity of that tool when they download it?
> (That's the recursion problem)

The initial firefox download uses NSS to compute hashes.  If that firefox
download is tampered with, then yes... we have a concern.  This is why the
windows firefox EXE downloads are signed.


> We already have full-blown support for signed zip/jar/xpi files.  In what way, 
> if any, is this new mechanism -- which puts hashes (not signatures) into a
> separate file that is downloaded from a separate source -- superior to the
> existing jar file support?  The Jar/zip/xpi mechanism keeps the verifiable 
> and unalterable signature together with (in) the download, requires only one
> download, and verification is (or can be) combined with extraction.  Further
> IINM, it's already built in to mozilla.  
> 
> I'd rather not see mozilla re-invent this wheel, again.

We discussed the option of signing our ZIP files, and that could still be done.
 However, what we need at minimum is a way to verify the integrity of what we
downloaded from the mirror network based on what our update servers tell us to
download.  At minimum, a crypto hash as dougt described is sufficient.

Let's turn the question around: what does signing give us that we need?

Comment 7

[Darin Fisher]

> The initial firefox download uses NSS to compute hashes.  If that firefox
> download is tampered with, then yes... we have a concern.  This is why the
> windows firefox EXE downloads are signed.

I read this again, and I think it's terribly unclear.  All I meant to say was
that users get the "hashing code" by virtue of downloading Firefox.  And, we
ensure that they get the "right code" by signing our Firefox installation
executable on Windows.  I think there is no signing done for other installation
packages, and that is of course a concern.

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

> what does signing give us that we need?

Everything that hashing gives you, plus the authenticity that you get with 
SSL, which authenticity remains provable after the connection/download is 
done (SSL's doesnt), without the server overhead of SSL.  Plus, as I said, 
it's already built into mozilla, no new file formats needed, no new protcols 
needed.  There are already tools built and supported by MULTIPLE vendors that 
are compatible with mozilla's built-in handling of signed jar/xpi/zip files.  

This wheel doesn't need reinvention.

Comment 9

[Daniel Veditz [:dveditz]]

If we put the updates in signed .zip files instead we can't just accept any old
correctly signed file. If we did then the security of the system is reduced to
that of the least careful developer with a signing certificate whose cert was
unknowning stolen, possibly with the key if they embedded it in a bat file
(stupid, but not unheard of).

So then instead of our secure server sending out hashes it'll have to send out
some unique identifier associated with the cert we expect to find on the
mirrored ftp site. What's safe to use for that? A hash of the cert I guess, so
we're still sending out hashes, and then have to deal with the extra steps of
unpacking a zip file before we can do any work.

Comment 10

[Darin Fisher]

> Everything that hashing gives you, plus the authenticity that you get with 
> SSL, which authenticity remains provable after the connection/download is 
> done (SSL's doesnt), without the server overhead of SSL.

You haven't told me why we need to prove authenticity after the
connection/download is done.  I contend that we are fine treating the user's
harddrive as a protected space.  Afterall, that is where Firefox is installed.

That said, I am leaning toward distributing the manifest file using a signed-JAR
instead of over HTTPS.  We have no need for encryption.  We only need to ensure
the integrity of the downloaded manifest file.  Once we have that manifest file,
we can safely download the update archive that it directs us to download, and we
can verify the integrity of that update archive based on the hash stored in the
update manifest.

Comment 11

[Daniel Veditz [:dveditz]]

(In reply to comment #10)
> That said, I am leaning toward distributing the manifest file using a signed-JAR
> instead of over HTTPS.  We have no need for encryption.  We only need to ensure
> the integrity of the downloaded manifest file.

Where will you get the list of certs acceptable to sign the the manifest file?
Will you be downloading that so we can deal with the fact that code-signing
certs expire regularly? If we lose that cert or have to revoke it, then what?

Serving the update manifest over https is unavoidable I think, whatever form it
takes.

> we can safely download the update archive that it directs us to download, and we
> can verify the integrity of that update archive based on the hash stored in the
> update manifest.

So it's just another way to send hashes :-)

Comment 12

[Doug Turner (:dougt)]

dan -- please take this offline.  this bug, although related, can be useful for
more than just the mozilla update system.

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

As I wrote in comment 4, the existing NSS QA test program bltest will read 
in a file and hash it with any of the hashes supported by NSS, and output
the computed hash value to stdout or to a named file, in raw binary or 
base64 encoded.  

bltest -H -m mode -i infilename [-o outfilename] [-a]

where mode is one of md5 sha1 sha256 sha384 sha512

Does this meet the need?  If so, this bug is WFM.

Comment 14

[Darin Fisher]

It looks like it does to me.  The only "challenge" is in figuring out how to
build the thing ;-)

Comment 15

[Doug Turner (:dougt)]

great. thanks nelson... 

darin, i think you can just do:

cd mozilla/security/nss
make nss_build_all 

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

Guys, I'm sorry.  This bltest program is seriously broken.  
If you use the -a command line option, you get essentially random results. :(
Further, I'm not sure the results are correct for any algorithm other than md5.

So, I am reopening this bug, and will also file a separate bug against
bltest, where I will document the numerous bugs I found in it today.
It will be P1 for NSS 3.11, I think.  But it won't help you out this week. :(

Comment 17

[Nelson Bolyard (seldom reads bugmail)]

Although many of the problems in bltest have been fixed (see bug 296487 for
details), at least one major one remains.  The input and output files are
always in the same file format, either both binary, or both base64 encoded
or both hex, etc.  Fixing that program to allow binary input and base64 
(or hex) output in the same run will require significant changes to the 
command line syntax.  

I should NOT have suggested that an NSS QA test program be used for 
production purposes on mozilla products.   Doug, I now recommend that you 
go ahead and complete the development of the hashing program you originally
proposed.  Sorry to have wasted time with that 

There are several issues with the program given as attachment 1 [details] [diff] [review] above. 
I will email review comments to Doug privately.  

Comment 18

[Darin Fisher]

We decided that it would be easier to use a perl module for this on the backend.
 Closing as WONTFIX.

Comment 19

[Tobias B. Besemer [:BesTo] (QA)]

Im interested in creating TnDCpp.
(http://sourceforge.net/projects/tndcpp)

I think a lot of other applications comeing up on gecko will use hasing in the
future. So it would be nice to have this in the source.

Plz reopen this bug.

Thx in advance.

Comment 20

[Doug Turner (:dougt)]

have you seen nsICryptoHash?

Comment 21

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to comment #20)
> have you seen nsICryptoHash?
No, Im not started yet with programming, but Im voted for that bug to monitor it.

Comment 22

[Christian :Biesinger (don't email me, ping me on IRC)]

I believe you are interested in the (fixed) bug 292368 instead of this bug,
which added the mentioned nsICryptoHash, which is what you want afaict.

this bug was about making a standalone executable.