710461 - Be smarter about downloading mar files in update verify

Status: RESOLVED DUPLICATE of bug 862910

(Release Engineering :: Release Automation, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Bug 710427 highlighted that we do a lot of repeated work when performing an update verify. Suppose, for the sake of argument, that you have a long lived branch "3.6" which has shipped N releases (N=31 currently). N-1 of those releases will have the same update.xml, as only the latest release is offered a real partial.

So, for every platform-locale pair you will end up downloading the complete.mar N times to verify the hash is correct, because all the old versions get the same complete. We can shortcut this by 
* building a cache of hashes & filesizes for urls we've retrieved, and comparing the cached values with the xml instead of downloading
* building a cache of verified xml content

With the current parallelization this may not result in a high hit rate, but we could break things up by locale to improve that. It should be a very nice speed improvement too.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

Comment 2

[[e-mail] (:bhearsum)]

Bulk move of bugs to Release Automation component.

Comment 3

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 4

[Pete Moore [:pmoore][:pete]]

Duplicate of bug 862910. I'm currently working on this.