1720563 - Ensure tasks that use CoT don't depend on tasks that expire before they do

Status: NEW

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

Chain of trust needs to trace back the full task dependency chain. But if a task depends on a task that expires before it does, CoT verification will fail. For instance, say we have the following graph:

A -> B -> C

And let's say both A and B normally get optimized by replacement. If A expires before B does, then the graph is perfectly valid as far as taskcluster is concerned. But for CoT this will raise an exception.

Therefore, we shouldn't optimize by replacement when:
A) A task uses CoT
B) The dependency will expire before it does.

The solution here should be very similar to bug 1690947 except:

1. We want to compare expiry to expiry (not deadline to expiry)
2. It should only apply to tasks that are part of CoT (we might want to have an attribute to denote these)

Comment 1

[Aki Sasaki (not active)]

(In reply to Andrew Halberstadt [:ahal] from comment #0)

The solution here should be very similar to bug 1690947 except:

1. We want to compare expiry to expiry (not deadline to expiry)

+1
I wonder if this means we end up running fetches and docker-images every push, though:

• cached docker-image expires in 1y from day 1.
• build on day 2 depends on docker-image task, expires in 1y from day 2.
• we run a new docker-image task to get a 1y expiry from day 2
• repeat

If this ends up being the case, perhaps we extend the appropriate cached_task expiries to 2y, which would effectively make them ~1y.

2. It should only apply to tasks that are part of CoT (we might want to have an attribute to denote these)

An attribute could save us some compute time, but I worry that:

a) these may be overly broad, e.g. all docker-image, fetch, and build tasks rather than those used by CoT-verified tasks
b) these may not be broad enough, expecially if we later add new kinds that need to be CoT-verified, and don't add the attribute.

I was thinking about how to programmatically find these tasks:

• add an attribute to scriptworker tasks
• look at scriptworker tasks' upstreamArtifacts taskIds. Add these to the list of tasks to check for early expiry.
• if any task in the list has an in-tree docker image, add that task to the list as well
• check each of the tasks' expiries, compare against the scriptworker task's

... That may be sufficient? Even though this is potentially many network calls, which I don't love. We also care about decision tasks during CoT verification, but I think the likelihood of a given task's decision task expiring significantly earlier than the task itself is low or none.

Comment 2

[Aki Sasaki (not active)]

Hm, wouldn't this also be problematic for release promotion?

Build happens on week 1
Promotion happens on week 2
Push/ship happens week 3
Off-cycle partner repack happens week 4

Aren't we rebuilding the world for each of those?

We may want to rescope the bug, to ensure that CoT is verifiable for, say, 6months.

For release purposes, we really only need CoT verifiable during the deadline of the tasks, maybe a few weeks afterwards -- we haven't done a lot of auditing, and we intentionally break those audits when we, say, rotate CoT keys and deprecate the old ones. We're really addressing the CI issue for things like adhoc-signing, which may or may not be the thing we want to optimize for. We may want to brainstorm if this is the best solution.

Comment 3

[[e-mail] (:bhearsum)]

It's possible I'm missing something here, but when we're talking about how to identify tasks that must be verifiable by chain of trust, is there any way to do that doesn't ultimately involve:
a) maintaining a list of tasks (or maybe only leaf tasks?) that we need to verify
or
b) adding an attribute to every task that needs to be verifiable

If the above is right, I think it's going to be difficult to maintain going forward -- and unless we have some way to make sure one of the above solutions is correct eg: when a new task is added, we may end up with bustage a year down the road.

All of this makes me wonder if we should simply rebuild tasks that have expired rather than depend on them (in all cases).

Comment 4

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #2)

We're really addressing the CI issue for things like adhoc-signing, which may or may not be the thing we want to optimize for. We may want to brainstorm if this is the best solution.

So I think we're miscategorizing the bustage in adhoc-signing.

We weren't trying to audit CoT on day 364 of the year, and lamenting the fact that the chain wasn't verifiable at that point due to expiration of upstream tasks.

We were trying to run new signing tasks, and the problem was that the docker-image tasks of the fetch tasks we were verifying expired before the deadline of the signing tasks.

We may be able to resolve that specific problem by extending the solution in bug 1690947 to include the docker-image tasks of the upstream tasks (potentially only if CoT is involved, otherwise everywhere).

For release purposes, we really only need CoT verifiable during the deadline of the tasks, maybe a few weeks afterwards -- we haven't done a lot of auditing, and we intentionally break those audits when we, say, rotate CoT keys and deprecate the old ones.
We may want to rescope the bug, to ensure that CoT is verifiable for, say, 6months.

I'm actually leaning towards ~1 month now.

Comment 5

[Aki Sasaki (not active)]

(In reply to [e-mail] (:bhearsum) from comment #3)

If the above is right, I think it's going to be difficult to maintain going forward -- and unless we have some way to make sure one of the above solutions is correct eg: when a new task is added, we may end up with bustage a year down the road.

Agreed on difficulty of maintenance.

All of this makes me wonder if we should simply rebuild tasks that have expired rather than depend on them (in all cases).

I think we do, as of bug 1690947. If we include the docker image tasks in the dependents list in https://phabricator.services.mozilla.com/D115726, and rebuild those as well, then I think we're in good shape for adhoc-signing.

Comment 6

[Andrew Halberstadt [:ahal]]

Throwing an overly broad net over CoT verifiable tasks isn't the end of the world. The only repercussion (at least from this bug.. there may be others if we start using this attribute for other purposes), is that we might fail to optimize by replacement when we technically could have in some cases.

We could have a transform that runs late that checks for tasks that have the CoT=True attribute set, and then automatically go through and add it to dependencies. This way we would only need to maintain the list of leaf tasks as Ben suggests (it could just be manually added in the ci config files).

Maybe it's still worth computing the CoT attribute exactly, but if this is the only use case, meh.

All of this makes me wonder if we should simply rebuild tasks that have expired rather than depend on them (in all cases).

I think we do, as of bug 1690947. If we include the docker image tasks in the dependents list in https://phabricator.services.mozilla.com/D115726, and rebuild those as well, then I think we're in good shape for adhoc-signing.

I think this is right. Though the problem happened because it is a dependency of a dependency.. so to solve this generally we'd have to check the entire dependency chain all the way up to the Decision task (if CoT enabled). If not CoT enabled, we'd only have to care about the first layer of dependencies.