declarative artifacts validation for Fennec

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
Last year
5 months ago

People

(Reporter: mtabara, Assigned: mtabara)

Tracking

({leave-open})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [releng:q42018])

Attachments

(2 attachments)

Facts:
1. Package filenames are all computed in package-name.mk[1] for the BUILD phase. 
2. Aforementioned files only depend on the target OS, CPU and application version. We could easily compute them all up-front for a given build if we wanted to since we have all that information in-tree. 
2. For the RELEASE workflow, the BUILD artifacts are being pretty named and transfered to S3 by beetmover (based on static templatized manifests in beetmover[2]) 
3. The artifacts name in the Taskcluster world are simple names (e.g. `target`) because they are predictable.
4. Currently, we need to guess the simple names when we're dealing with (e.g.) multiple platforms: is the installer target.exe or target.zip?  Or is it a different platform, target.tar.bz2 or target.dmg? Etc
===

Motivation:
1. Each time a new artifact is needed, there's at least three places that need to be updated: 
a) add the simple named artifact in the build system
b) add the simple named artifact in the task graph
c) add the simple name -> pretty name rule in the beetmover manifests

2. Each time a new artifact is needed, we need to touch at least two repos:
a) beetmoverscript repository[2] and roll-out a new python package
b) gecko tree

3. In order to manage N apk files (or any other similar requirement), not knowing the final state of artifacts name upfront makes it difficult to update various other tools

4. Use different filenames for each variant Android APK is slightly more difficult to do if we need to update multiple moving parts in the automation, rather than a single centralized source-of-truth file in-tree (that can ride the trains)

5. Some final multiplatform tasks would still need some sort of regex or map to figure out which files to look at per platform (e.g., is the shipping file for platform X a tarball, or exe, or apk, or ?)  A pre-declared manifest template would still help streamline those cases.
===

Solution:
tl;dr: Create a manifest with artifact names, that the build system then uses to name its output files. 

* we're thinking a declarative manifest may be the solution, mapping the role of an artifact with its name.  We can use that manifest to generate the taskcluster graph, and the build system could use it to know what to name its artifacts.  If the names are pretty names, beetmover doesn't have to perform any name transforms at all, and can just push pretty-named artifacts to the proper s3 bucket.
* templatized version could allow for easier generation outside of configure (e.g. win32 installer -> "Firefox {version}.exe" or "%(appname)s %(version)s.exe")

"See also" section encompasses few scenarios in which this problem came up. 

[1]: https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk
[2]: https://github.com/mozilla-releng/beetmoverscript/tree/master/beetmoverscript/templates
I would add to the set of motivations:

6. Downstream tasks implicitly depend on artifacts produced by their parent(s). They break at runtime if the expected artifacts are not produced. By declaring which artifacts are going to be output by the build tasks, we can add artifact dependency checks at taskgraph generation time.
Depends on: 1462393
See Also: 1462393
Whiteboard: [releng:q32018]
Blocks: 1442684
Assignee: nobody → mtabara
Depends on: 1490959
Depends on: 1490960
Depends on: 1490961
Been focusing on other OKRs and I missed documenting the first steps that need to be done here. 
Status update:

=================           ==============================          =================
In-tree manifest| <------>  |In-tree beetmover transforms| -------> |beetmoverscript|
=================           ==============================          =================

(1)                                  (2)                                (3)

(1) - manifest - we need to land this in-tree to be able to have the transform consume it.
(2) - enrich existing beetmover transforms to consume information from manifest, process it and sum-it up nicely in the beetmover tasks payload
(3) - beetmoverscript needs to handle the newly added payload (more specific, the artifactMap dict)


Since the manifest will ride the trains - once landed on central - we'll have to support both the old way (with templates in beetmoverscript) but also the new way (declarative artifacts with enriches payload with artifactMap). We can toggle between them depending on beetmover jobs having the artifactMap dict within payload or not.

I reckon it'd be easier if we started with development in the same manner as we did for tcmigration, that is:
* Start with Fennec nightlies
* manifest should encompass soley those artifacts
* scripts to generate the manifest and location & more information on that are in bug 1490959
* tweak the transforms (namely the `beetmover` ones) to handle the new behavior. that's tracked in bug 1490961
* polish the beetmoverscript changes and land those to a puppet environment 
* test end-to-end either on maple or on birch or w\e, wherever we land the manifest initially

If things work smoothly, proceed with Firefox nightilies, the Fennec releases, Firefox releases, etc.
In order to make some tangiable progress, let's start with a more granular scope.
Let's focus on getting Fennec nightlies done first.
That means:
a) we land the manifest for Fennec only, instead of the large one
b) we tweak the transforms to enrich with artifactMap
c) we slightly polish existing beetmoverscript PR (lisa's work) and bring it up-to-date
d) test end-to-end.

If things go well, we gradually add Firefox nightlies too. And so on.
Advantage of this incremental development is that it allows us to find all those corner cases gradually, instead of dealing with them from the very beginning.

I'll adjust the depending bugs to be specific to Fennec instead.
Duplicate of this bug: 1331141
Depends on: 1492485
Depends on: 1492494
Depends on: 1493588
Summary: [tracking] declarative artifacts → [tracking] declarative artifacts validation for Fennec
Whiteboard: [releng:q32018] → [releng:q42018]
Summary: [tracking] declarative artifacts validation for Fennec → declarative artifacts validation for Fennec
Keywords: leave-open
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e375e10f670a
declarative artifacts in-tree work for Fennec. r=mtabara
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8e91d8ea0141
follow-up flake8 fix. r=bustage CLOSED TREE
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/fe1fe0e21182
Port bug 1466714 [declarative artifacts in-tree work for Fennec]. rs=bustage-fix
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/41b13e00133e
Port Bug 1466714: follow-up, remove excess empty lines. rs=white-space-only
Green graphs for Fennec nightlies: https://tools.taskcluster.net/groups/U3OgpDLNRvaEooi-I3yj9Q

We're going to leave this bake until after the All-hands, following which we'll turn it on for Fennec beta and release.
In the meantime, in Q1 most likely we'll switch the other products as well for all channels.
We have some corrupt entries for Fennec nightlies, see here[1] for more. The locale prefix is being prepended here[2] but has a missing slash. Adding it in the manifest is not ideal but it's the best for now I think.

What do you think?
P.S. I tested this change against all graphs, works smoothly.

[1]: http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-aarch64/ 
[2]: https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/scriptworker.py#l574
Attachment #9029857 - Flags: review?(sfraser)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)
> Created attachment 9029857 [details] [diff] [review]
> Bustage fix for locale_prefix Fennec entries
> 
> We have some corrupt entries for Fennec nightlies, see here[1] for more. The
> locale prefix is being prepended here[2] but has a missing slash. Adding it
> in the manifest is not ideal but it's the best for now I think.

I think it should be in the manifest - otherwise we're trying to special-case add it to the formatting. I thought it already was there!
Attachment #9029857 - Flags: review?(sfraser) → review+
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b1e3a057032e
fix locale_prefix in declarative artifacts manifests. r=sfraser a=release
(In reply to Pulsebot from comment #13)
> Pushed by mtabara@mozilla.com:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/b1e3a057032e
> fix locale_prefix in declarative artifacts manifests. r=sfraser a=release

This is naturally going to get to beta by Monday given the continuos central -> beta migrations that happen before the nightly bump.
This has been working smoothly on central, proving this model works nicely for us. Still blocked on beta until after the holidays, pending Q1 prioritization and model validation conversations.
Goal for this bug was to validate Fennec on nightly which was accomplished. Simon++ for all the great help provided here.
We'll file another bug should we decide to prioritize this in Q1 and switch all products across all channels.
Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Pushed by sfraser@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a5c5e322deb7
Declarative artifacts location fix r=mtabara

I think this fixed the issue. (e.g. https://archive.mozilla.org/pub/mobile/nightly/2019/01/2019-01-01-09-47-42-mozilla-central-android-api-16/en-US/ seems correct)

Please re-open if anything else goes wrong.

Status: REOPENED → RESOLVED
Closed: 7 months ago7 months ago
Resolution: --- → FIXED
See Also: → 1520244
You need to log in before you can comment on or make changes to this bug.