1466714 - declarative artifacts validation for Fennec

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

Facts:
1. Package filenames are all computed in package-name.mk[1] for the BUILD phase. 
2. Aforementioned files only depend on the target OS, CPU and application version. We could easily compute them all up-front for a given build if we wanted to since we have all that information in-tree. 
2. For the RELEASE workflow, the BUILD artifacts are being pretty named and transfered to S3 by beetmover (based on static templatized manifests in beetmover[2]) 
3. The artifacts name in the Taskcluster world are simple names (e.g. `target`) because they are predictable.
4. Currently, we need to guess the simple names when we're dealing with (e.g.) multiple platforms: is the installer target.exe or target.zip?  Or is it a different platform, target.tar.bz2 or target.dmg? Etc
===

Motivation:
1. Each time a new artifact is needed, there's at least three places that need to be updated: 
a) add the simple named artifact in the build system
b) add the simple named artifact in the task graph
c) add the simple name -> pretty name rule in the beetmover manifests

2. Each time a new artifact is needed, we need to touch at least two repos:
a) beetmoverscript repository[2] and roll-out a new python package
b) gecko tree

3. In order to manage N apk files (or any other similar requirement), not knowing the final state of artifacts name upfront makes it difficult to update various other tools

4. Use different filenames for each variant Android APK is slightly more difficult to do if we need to update multiple moving parts in the automation, rather than a single centralized source-of-truth file in-tree (that can ride the trains)

5. Some final multiplatform tasks would still need some sort of regex or map to figure out which files to look at per platform (e.g., is the shipping file for platform X a tarball, or exe, or apk, or ?)  A pre-declared manifest template would still help streamline those cases.
===

Solution:
tl;dr: Create a manifest with artifact names, that the build system then uses to name its output files. 

* we're thinking a declarative manifest may be the solution, mapping the role of an artifact with its name.  We can use that manifest to generate the taskcluster graph, and the build system could use it to know what to name its artifacts.  If the names are pretty names, beetmover doesn't have to perform any name transforms at all, and can just push pretty-named artifacts to the proper s3 bucket.
* templatized version could allow for easier generation outside of configure (e.g. win32 installer -> "Firefox {version}.exe" or "%(appname)s %(version)s.exe")

"See also" section encompasses few scenarios in which this problem came up. 

[1]: https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk
[2]: https://github.com/mozilla-releng/beetmoverscript/tree/master/beetmoverscript/templates

Comment 1

[Chris AtLee [:catlee]]

I would add to the set of motivations:

6. Downstream tasks implicitly depend on artifacts produced by their parent(s). They break at runtime if the expected artifacts are not produced. By declaring which artifacts are going to be output by the build tasks, we can add artifact dependency checks at taskgraph generation time.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Been focusing on other OKRs and I missed documenting the first steps that need to be done here. 
Status update:

=================           ==============================          =================
In-tree manifest| <------>  |In-tree beetmover transforms| -------> |beetmoverscript|
=================           ==============================          =================

(1)                                  (2)                                (3)

(1) - manifest - we need to land this in-tree to be able to have the transform consume it.
(2) - enrich existing beetmover transforms to consume information from manifest, process it and sum-it up nicely in the beetmover tasks payload
(3) - beetmoverscript needs to handle the newly added payload (more specific, the artifactMap dict)


Since the manifest will ride the trains - once landed on central - we'll have to support both the old way (with templates in beetmoverscript) but also the new way (declarative artifacts with enriches payload with artifactMap). We can toggle between them depending on beetmover jobs having the artifactMap dict within payload or not.

I reckon it'd be easier if we started with development in the same manner as we did for tcmigration, that is:
* Start with Fennec nightlies
* manifest should encompass soley those artifacts
* scripts to generate the manifest and location & more information on that are in bug 1490959
* tweak the transforms (namely the `beetmover` ones) to handle the new behavior. that's tracked in bug 1490961
* polish the beetmoverscript changes and land those to a puppet environment 
* test end-to-end either on maple or on birch or w\e, wherever we land the manifest initially

If things work smoothly, proceed with Firefox nightilies, the Fennec releases, Firefox releases, etc.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

In order to make some tangiable progress, let's start with a more granular scope.
Let's focus on getting Fennec nightlies done first.
That means:
a) we land the manifest for Fennec only, instead of the large one
b) we tweak the transforms to enrich with artifactMap
c) we slightly polish existing beetmoverscript PR (lisa's work) and bring it up-to-date
d) test end-to-end.

If things go well, we gradually add Firefox nightlies too. And so on.
Advantage of this incremental development is that it allows us to find all those corner cases gradually, instead of dealing with them from the very beginning.

I'll adjust the depending bugs to be specific to Fennec instead.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e375e10f670a
declarative artifacts in-tree work for Fennec. r=mtabara

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8e91d8ea0141
follow-up flake8 fix. r=bustage CLOSED TREE

Comment 7

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/e375e10f670a
https://hg.mozilla.org/mozilla-central/rev/8e91d8ea0141

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/fe1fe0e21182
Port bug 1466714 [declarative artifacts in-tree work for Fennec]. rs=bustage-fix

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/41b13e00133e
Port Bug 1466714: follow-up, remove excess empty lines. rs=white-space-only

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Green graphs for Fennec nightlies: https://tools.taskcluster.net/groups/U3OgpDLNRvaEooi-I3yj9Q

We're going to leave this bake until after the All-hands, following which we'll turn it on for Fennec beta and release.
In the meantime, in Q1 most likely we'll switch the other products as well for all channels.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bustage fix for locale_prefix Fennec entries

We have some corrupt entries for Fennec nightlies, see here[1] for more. The locale prefix is being prepended here[2] but has a missing slash. Adding it in the manifest is not ideal but it's the best for now I think.

What do you think?
P.S. I tested this change against all graphs, works smoothly.

[1]: http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-aarch64/ 
[2]: https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/scriptworker.py#l574

Comment 12

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)
> Created attachment 9029857 [details] [diff] [review]
> Bustage fix for locale_prefix Fennec entries
> 
> We have some corrupt entries for Fennec nightlies, see here[1] for more. The
> locale prefix is being prepended here[2] but has a missing slash. Adding it
> in the manifest is not ideal but it's the best for now I think.

I think it should be in the manifest - otherwise we're trying to special-case add it to the formatting. I thought it already was there!

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b1e3a057032e
fix locale_prefix in declarative artifacts manifests. r=sfraser a=release

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Pulsebot from comment #13)
> Pushed by [e-mail]:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/b1e3a057032e
> fix locale_prefix in declarative artifacts manifests. r=sfraser a=release

This is naturally going to get to beta by Monday given the continuos central -> beta migrations that happen before the nightly bump.

Comment 15

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/b1e3a057032e

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

This has been working smoothly on central, proving this model works nicely for us. Still blocked on beta until after the holidays, pending Q1 prioritization and model validation conversations.

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Goal for this bug was to validate Fennec on nightly which was accomplished. Simon++ for all the great help provided here.
We'll file another bug should we decide to prioritize this in Q1 and switch all products across all channels.

Comment 18

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1466714 Declarative artifacts location fix r=mtabara

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a5c5e322deb7
Declarative artifacts location fix r=mtabara

Comment 20

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/a5c5e322deb7

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

I think this fixed the issue. (e.g. https://archive.mozilla.org/pub/mobile/nightly/2019/01/2019-01-01-09-47-42-mozilla-central-android-api-16/en-US/ seems correct)

Please re-open if anything else goes wrong.