684767 - Revisit creation of MD5SUMS/SHA1SUMS

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Currently we iterate over all the files to create MD5SUMS, then do the same again to create SHA1SUMS. This is slower than it could be because we're hashing gigabytes of data and can't expect the OS file cache to help the second time around.

We could do these instead:
* for each file calculate all the hashes together, so we only hit the disk for the first hash
* drop the existing files for a SHA512SUMS. IIRC md5 has known collisions, and SHA1 has theoretical collisions. Even if SHA1 was broken, it seems unlikely you could be able to modify a file and still make both hashes match the original, but we could just shift to something which is modern

Comment 1

[Matthew Elvey]

It would be good if the *SUMS files pointed users to URLs for downloading the files SECURELY. (See bug 687164.)

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Matthew Elvey (Working address - Bug 120030 fix works!!!) from comment #1)
> It would be good if the *SUMS files pointed users to URLs for downloading
> the files SECURELY. (See bug 687164.)

This bug is about an optimisation to the process that generates the SUMS files. This is not relevant, please continue that conversation in an appropriate place.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Rail, will this be fixed by bug 708656 ?

Comment 4

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #3)
> Rail, will this be fixed by bug 708656 ?

Yup. I'll track this bug as well.

Comment 5

[Rail Aliiev [:rail]]

This is fixed by bug 708656 and available in 11.0b1