421012 - update nightly AUS scripts to use sha1 or stronger encryption for hashes

Status: RESOLVED FIXED

(AUS Graveyard :: Systems, defect)

Description

[Michael Morgan [:morgamic]]

Currently update RDFs output hash sums created with md5 (see URL).

We should get this updated to use sha1 or better, especially for release updates.

Comment 1

[Michael Morgan [:morgamic]]

Added security flag, but it's not a secret that we use md5 for nightlies -- remove if you feel appropriate.

Comment 2

[Michael Morgan [:morgamic]]

Normal releases are indeed SHA1:
https://aus2.mozilla.org/update/2/Firefox/2.0.0.8/2007100816/Darwin_Universal-gcc3/en-US/release/Darwin%208.11.1/update.xml?force=1

Comment 3

[Reed Loden [:reed]]

SHA-1 has been compromised. Can we use SHA-256?

Comment 4

[Daniel Veditz [:dveditz]]

FF2 nightlies use sha1:
https://aus2.mozilla.org/update/2/Firefox/2.0.0.13pre/2008030203/WINNT_x86-msvc/en-US/nightly/Windows_NT%205.1/update.xml?force=1

Why would the trunk have gone "backwards" in hash quality?

What's used for trunk and FF2 release updates?

sha1 is OK for now. At some point the attacks will get better and we'll have to use sha-256. Cygwin contains the sha256sum utility so it should be easy to get on unix-based systems too.

Comment 5

[Robert Helmer [:rhelmer]]

Trunk betas are SHA1 as well:

https://aus2.mozilla.org/update/1/Firefox/3.0b1/2007110903/Linux_x86-gcc3/pl/beta/update.xml

Looks like MD5 is only in use for trunk nightlies, as far as I can tell.

Comment 6

[Reed Loden [:reed]]

Attachment: wip

wip patch on updating all the tools to use SHA-256 by default

Comment 7

[Reed Loden [:reed]]

Attachment: Make trunk use sha1 for now

Swap Firefox on Mac, Thunderbird on Mac, and Thunderbird on win32 to sha1 for now.

Comment 8

[Reed Loden [:reed]]

Comment on attachment 307408 [details] [diff] [review]
wip

Couple of questions I had when working on this.

>     def calc_file_sha_digest(self, filename):        
>         """ Returns sha digest of given filename"""
>         file_content = open(filename, 'r').read()
>-        return sha.new(file_content).digest()
>+        return hashlib.sha256(file_content).hexdigest()

Why is that using digest() currently instead of hexdigest()? I can use digest() here, but I'm just wondering why it was being used in the first place.

>             metadata.append({
...
>-             'to_sha1sum':to_shasum, 
>-             'from_sha1sum':from_shasum, 
>-             'partial_sha1sum':partial_shasum, 
>+             'to_sha256sum':to_shasum, 
>+             'from_sha256sum':from_shasum, 
>+             'partial_sha256sum':partial_shasum, 
...

Where is this used? I'll need to update whatever uses this metadata for the new names of things, or I could swap to something more generic...

Comment 9

[Michael Morgan [:morgamic]]

Dan -- since we're changing this anyway why not use sha512?

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Seems to be some conflation of nightly and release systems here. 

Nightly completes use whatever hash the tinderbox is set to, so attachment 307409 [details] [diff] [review] would be a step in the right direction. Partials inherit from the completes so we only need to set tinderbox to use sha1 for the quick fix. I think attachment 307409 [details] [diff] [review] is incomplete though, because we also need to fix:
  MOZILLA_1_8_0_BRANCH/firefox/macosx
  MOZILLA_1_8_0_BRANCH/thunderbird/win32
  MOZILLA_1_8_BRANCH/thunderbird/macosx
  MOZILLA_1_8_BRANCH_release/thunderbird/macosx
  release/firefox/macosx  (this doesn't mean release updates are using md5)

The proposed changes to the release scripts and tinderbox won't function as is, because the openssl available on various machines doesn't support sha-256 or higher. There should also be a change to mozilla/tools/tinderbox-defaults.pl. 

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [checked in] Global s/md5/sha1/ on tinder-config.pl, and update tinder-defaults.pl to reflect reality

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Is there anything else left to do here, or can we close this?

Comment 13

[Reed Loden [:reed]]

(In reply to comment #12)
> Is there anything else left to do here, or can we close this?

Are you all still interested in using something stronger than SHA-1 since SHA-1 has already been broken? If so, see the wip patch.

Comment 14

[Michael Morgan [:morgamic]]

Can we settle on SHA-1 until after fx3?

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #14)
> Can we settle on SHA-1 until after fx3?
+1

Based on comment#4, it seemed like sha-1 was good enough for now, and in this bug, we just wanted to bring trunk nightly updates up-to-par with the rest of the build systems. That work is landed, aiui.

If you want to, please file a separate bug on changing all build systems to use a different algorithm for update hashsums; however, I think thats beyond the scope of this bug, and not something we want to take on right now at this point in the FF3 release. 

Comment 16

[Daniel Veditz [:dveditz]]

Filed bug 472790 on finishing the job and moving to a SHA-2 hash.