1004570 - Jacuzzi for Update Verify, so we can use a persistent cache across jobs

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect, P3)

Description

[Pete Moore [:pmoore][:pete]]

Update verify currently downloads masses of data, consuming a lot of time. Would be great to move this into a jacuzzi, and keep a persistent cache of downloaded releases and mar files, to save needing to download all these artifacts, every time. Putting in a jacuzzi means we have more disk space available, to allow for keeping more binaries between jobs, and also we could potentially shrink the chunk sizes (since the chunk portions will shift over time, so we may cache on one chunk jacuzzi, yet a few iterations later it is needed by a different chunk). What I mean by this: a chunk is say a 6th of the list, and over time what is in a given chunk will shift (e.g. one day could be firefox 15 -> firefox 19, but a few months later could be firefox 20 - firefox 25). Maybe if we can even have a single chunk (i.e. no chunking) then we would benefit from all of the cached objects.

Partials would not be cached, not complete mars. Full releases would all be cached.

Please see bug 862910 to see how downloads are now cached in update verify. Currently this cache is dropped between jobs - but we could retain this cache if we implement this.

Comment 1

[Pete Moore [:pmoore][:pete]]

Apologies for my terrible English. Some corrections:

"and also we could potentially shrink the chunk sizes" -> "and also we could potentially increase the size of the chunks (reduce number of chunks)"

"Partials would not be cached, not complete mars." -> "Partial mars would not be cached, complete mars would be cached."

Comment 2

[[e-mail] (:bhearsum)]

I think we talked about this this morning and decided that it wouldn't help much. I think Nick said the same thing awhile ago too...

Comment 3

[Pete Moore [:pmoore][:pete]]

Hi Ben,

I think in its current form it doesn't help, but if we persist the cache between iterations, it will. What do you think?

Pete

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Pete Moore [:pete][:pmoore] from comment #3)
> Hi Ben,
> 
> I think in its current form it doesn't help, but if we persist the cache
> between iterations, it will. What do you think?
> 
> Pete

The files that you need may shift around to different chunks with every release, because you're adding new release/files with each one. The needed files for a chunk will be mostly static, but over time they will shift back to earlier chunks as each chunk grows bigger. This also means that we need a jacuzzi per chunk -- otherwise you're unlikely to get on a slave that has most of what you need.

So, it may or may not be worthwhile. I thought that we had decided on IRC that it probably wasn't, since bug 862910 is going to help out a lot. I don't feel terribly strongly about this though.

Comment 5

[Chris AtLee [:catlee]]

WONTFIX because we're killing off jacuzzis in bug 1231367.