Currently we have an in-tree graph with the simple names to beetmove. We also have a bunch of yaml manifests in beetmoverscript describing the simple name -> pretty name conversion. When we add new artifacts to the list, we need to be able to do so in a way that doesn't require simultaneous updates to both sides. Also, if we have the manifests out-of-tree, at some point the trains' artifact lists will diverge, and beetmoverscript will have to handle all variants. (esr vs aurora/beta/release vs nightly) Possible solutions: Callek was suggesting we make one optional, then we can update the list on one side and then update the other side later without bustage. For instance: - beetmoverscript has a superset of all artifacts defined, and doesn't error out if we're missing some. Developers will likely update the in-tree stuff first, though, leading to bustage if they add an unknown artifact. - if we can't find the artifact in the manifest, beetmove it without renaming. This might help reduce bustage when we add a new artifact to the list. When we notice that we have a simple name in our beetmoved bucket, then we go back and update the manifests, and rerun the graph. This may lead to an un-pretty-name file during a beta1 or some other emergency, if we don't catch it in time. I was thinking about moving the yaml files in-tree: - manifests in-tree with both the simple + pretty names. Use these to generate the graph. The build uploads the in-tree manifest(s) as artifacts and we download them in the beetmover scriptworker, then use those manifests to know how to rename them. The advantage of this is, we can update both sides (simple/pretty) at the same time in the same place, and they can ride the trains. The disadvantage is this probably has more places we need to patch than the above solutions.
Summary: allow for artifact list changes without bustage → beetmoverscript: allow for artifact list changes without bustage
The in-beetmover manifests are a superset of the in-tree manifests, so we already have one solution in place for this. We could resolve this bug, or we could move the templates in-tree to minimize how many repos we have to touch to add new artifacts.
2 years ago
Would love to look at this bug when it gets prioritized.
If/when we land the beetmover templates in-tree, let's add some taskgraph tests to make sure all beetmover upstreamArtifacts match the appropriate template(s). This will allow people to add new artifacts to the graph and test locally, with a high likelihood of a green task.
Component: General Automation → General
Component: General → Release Automation: Uploading
QA Contact: catlee → mtabara
Status: NEW → RESOLVED
Closed: 11 months ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1466714
You need to log in before you can comment on or make changes to this bug.