1460943 - Allow expiration of artifacts to be updated

Status: RESOLVED WORKSFORME

(Taskcluster :: Services, enhancement)

Description

[Gregory Szorc [:gps]]

Artifacts are commonly used as a cache to avoid repeating previous work. For example, in bug 1460777, I'm implementing "fetch" tasks that download externally-hosted content and re-expose it as task artifacts so we eliminate a run-time dependency on 3rd party services. For scenarios like these, we would very much like some artifacts to be persisted for as long as they are used.

The way it works today, artifacts have an expiration time. When an artifact being used as a cached object expires, it must be rebuilt.

Without a store that is guaranteed to hold on to data for all of time, there's no guarantee that an artifact can be rematerialized in the future after it has expired. In other words, even if Firefox builds and CI are deterministic, we may not be able to reconstruct all the inputs required to rebuild a specific Firefox revision N years from now.

One solution to this is to provide a content store that never expires its data. Tooltool is arguably one such store.

Another potential solution to this problem is allowing the expiration of artifacts to be extended. We could, for example, perpetually renew the expiration of certain artifacts - say all artifacts that represent content retrieved from 3rd party services we can't rely on being available for eternity.

Another benefit of extending artifact expiration is it allows us to be much more aggressive about limiting the default artifact lifespan. *If* tasks are deterministic and reproducible over time, task artifacts are conceptually a cache and can be repopulated whenever. This means we can make the default artifact retention much lower - say 30 days - and have tasks that use old artifacts continuously bump their expiration so they never disappear. Frequently used artifacts will linger around. Unused artifacts will expire and not consume space.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I think that the approximation to this is to set an artifact expiration impossibly futuristic, e.g., 01-01-3000.  As long as we don't produce a massive quantity of such perma-artifacts, that's a great solution.  If that's the set of "fetched" artifacts then it is certainly not going to grow too quickly.

Comment 2

[Gregory Szorc [:gps]]

An impossibly futuristic expiration sounds like a reasonable solution to this problem!

Do we need any scopes to set such an expiration? Should we have scopes to control this to mitigate potential for abuse?

Please feel free to close this out WORKSFORME or some such.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

No scopes required :)

Comment 4

[Gregory Szorc [:gps]]

Chaining this up so the discussion can be more discoverable.