1462393 - [tracking] beetmover's pretty naming templates should live in-tree

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Uploading, enhancement)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

This bug is to track the moving of templates from beetmoverscript to in-tree. 

a) the manifests should live in-tree with both the simple + pretty names. 
b) we should use these to generate the graph.
c) The build uploads the in-tree manifest(s) as artifacts and we download them in the beetmover scriptworker, then use those manifests to know how to rename them.

The advantage of this is, we can update both sides (simple/pretty) at the same time in the same place, and they can ride the trains.  The disadvantage is this probably has more places we need to patch than we anticipate.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

First things first, let's get rid of BalrogProps in beetmoverscript, so that we can ease the transition of the beetmover templates. That's tracked in bug 1449150.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

First we need to find a good place for templates[1] in-tree. Maybe `taskcluster/manifests/{firefox_candidates.yml, etc...}` under https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ ?

Then we need to figure it out how we transfer that manifest to beetmover. I see two options but could be more:

1. Via task payload
* decide in-tree which one is suited for that particular beetmover job and send the manifest via task.payload
* we don't need much information to decide on them other than[2] some product, platform and action - all of which lie in tree

Question: a
* is it even possible to do this? Considering we need to keep the manifests as yml files in-tree and then send them in payload with variables templatized via dict / JSON?
 
Advantages:
* if this is not feasible like this, we could even go further and move all that annoying logic from beetmover manifest-generation[3] and template-args-reckon[4] within in-tree computation so that we send the proper json file via task.payload
* we'd have the payload contained within each task so it's even easier to debug
* we'd clean parts of beetmover which are now hackish still

Disadvantages:
* not sure if this is doable actually. The yml -> json -> payload -> file templatized value in json seems difficult, unless I'm missing something obvious.


2. Via upstreamArtifacts
* we define the manifests in-tree in a good location as YAML files
* we load them in Build task to have them as artifacts
* upstreamArtifacts for all beetmover jobs should download the full set of mapping manifests and then decide based on existing logic which one is to go up next 

Advantages:
* possibly more easy to implement since it requires minimal changes in beetmoverscript once the yaml files are moved out
* 

Disadvantages:
* we'd download the full set of manifests with each beetmover job. It can add up to network and time. It's slim but it adds up in time
* we'd still need to debug in the logs if ever the manifest

@aki: what are your thoughts on this


[1]: https://github.com/mozilla-releng/beetmoverscript/tree/master/beetmoverscript/templates
[2]: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/utils.py#L157
[3]: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/utils.py#L179
[4]: https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/utils.py#L110

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Lisa is going to tackle this soon. Prepping some groundwork for her here beforehand.

Comment 4

[Aki Sasaki (not active)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)
> First we need to find a good place for templates[1] in-tree. Maybe
> `taskcluster/manifests/{firefox_candidates.yml, etc...}` under
> https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/ ?

Sure. I'm not sure if we want a more beetmover-specific location. I don't have
strong opinions here.

> Then we need to figure it out how we transfer that manifest to beetmover. I
> see two options but could be more:
> 
> 1. Via task payload
> * decide in-tree which one is suited for that particular beetmover job and
> send the manifest via task.payload
> * we don't need much information to decide on them other than[2] some
> product, platform and action - all of which lie in tree
> 
> Question: a
> * is it even possible to do this? Considering we need to keep the manifests
> as yml files in-tree and then send them in payload with variables
> templatized via dict / JSON?

I think so. What challenges do you see?
yml -> python dict -> json should work, I think. I may be missing something.

> Advantages:
> * if this is not feasible like this, we could even go further and move all
> that annoying logic from beetmover manifest-generation[3] and
> template-args-reckon[4] within in-tree computation so that we send the
> proper json file via task.payload
> * we'd have the payload contained within each task so it's even easier to
> debug
> * we'd clean parts of beetmover which are now hackish still
> 
> Disadvantages:
> * not sure if this is doable actually. The yml -> json -> payload -> file
> templatized value in json seems difficult, unless I'm missing something
> obvious.

It may be possible. We may be able to check the templates in as-is, as well.

> 2. Via upstreamArtifacts
> * we define the manifests in-tree in a good location as YAML files
> * we load them in Build task to have them as artifacts
> * upstreamArtifacts for all beetmover jobs should download the full set of
> mapping manifests and then decide based on existing logic which one is to go
> up next 
> 
> Advantages:
> * possibly more easy to implement since it requires minimal changes in
> beetmoverscript once the yaml files are moved out
> * 
> 
> Disadvantages:
> * we'd download the full set of manifests with each beetmover job. It can
> add up to network and time. It's slim but it adds up in time
> * we'd still need to debug in the logs if ever the manifest

Hm, also possible to just link to the manifest or specify the revision/repo in
the payload, and download them or sparse clone them outside of CoT.

I see this as a faster solution to in-tree templates than adding the contents
of the manifests to the task payload.

When given an open-ended problem for one part of a larger project, I tend to
keep a number of potential solutions open, and then either wait until decisions
about other parts of the project point to one of those solutions, or
choose the easiest to implement. In this case, discussions about what
declarative artifacts will look like longer-term will let us know if one of
these solutions gives us more progress towards the long-term solution. If both
solutions are temporary, the simplest or easiest-to-implement approach may be
best.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Okay, so to sum this up.

Context: given that the "declarative artifacts"

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Some IRC history for posterity.

~mtabara> aki: hey! am trying to set some specifics for moving beetmover mapping manifests in-tree (for now, separately from the 17:17:43 <~mtabara> re: "yml -> python dict -> json" - a) can we do the yml loading -> dict without interpolating the values b) suppose that's possible, in the beetmoverscript side, how do we interpolate all those dict variables all at once? 
17:20:40 <@aki> can jinja2 take a dict instead of a file as a template?
17:21:12 <@aki> no, it's not json
17:21:36 <@aki> could change it to json-e or something
17:21:58 <@aki> that's a harder task, but if we want json-e templates for declarative artifacts, it's part of the long term solution
17:22:24 <@aki> if we don't see this as part of the long-term solution, the easier solution is probably best
17:23:21 <@aki> https://hg.mozilla.org/mozilla-central/raw-file/default/.taskcluster.yml is an example of yaml that gets turned into templatized json
17:27:09 <~mtabara> seems like task.payload.revision + download proper yml file in beetmoverscript is most likely the easisest to go forward. especially in the context of upcoming declarative artifacts conversation
17:27:36 <~mtabara> by "link to the manifest" you meant passing the full url? 
17:29:33 <@aki> sure, or manifest dir
17:32:18 <@aki> the main tricky part is you probably have to land the manifests on all the trains, or you have to support both behaviors
17:32:46 <@aki> supporting both behaviors for a cycle may be best, since we may release a chemspill off a relbranch
17:32:56 <@aki> we could land the manifests on the relbranch if we don't support both
17:33:33 <@aki> then download if the manifest url is in the payload, and otherwise fall back to the beetmover-repo manifests
17:34:43 <~mtabara> so I guess that means
17:34:43 <~mtabara> a) moving manifests in-tree under a certain location
17:34:43 <~mtabara> b) pass in payload the url dir
17:34:43 <~mtabara> c) add some logic in beetmoverscript to async-download the proper template
17:34:43 <~mtabara> d) support both the behaviors for at least one cycle

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Okay, so to sup this up.
Context: given that the "declarative artifacts" conversation is due for next Friday, we should be leaning on the easiest solution. Ideally, one that takes us closer to our goal, but in the same time doesn't waste lots of time, given that in the upcoming weeks, we may choose a different solution.

For that particular reason, I think the best proposal to move forward with this is the next one:

a) [in-tree] we need to copy the manifests[1] from beetmoverscript to the tree in a given location. So far I couldn't come up with a better idea than `taskcluster/manifests/{firefox_candidates.yml, etc...}` under here[2]. Maybe we could run this by dustin for a second opinion.

b) [in-tree] once manifests are in-tree in a certain location, let's build the full-url to the directory and pass it along in all beetmover jobs that perform promotion phases, within the task.payload

c) [beetmoverscript] we need to support both the existing behavior (with mapping manifests in beetmoverscript) but also the new one (with manifests coming from tree)  for at least one cycle. Supporting both behaviors for a cycle may be best, since we may release a chemspill off a relbranch. To toggle between them, we can use the full_url present in payload or not, from b)

d) [beetmoverscript] we implement some small async-downloading function in beetmoverscript that downloads the manifest and loads its content

For working with the graph, a good read is Dustin's blogpost.

[1]: https://github.com/mozilla-releng/beetmoverscript/tree/master/beetmoverscript/templates
[2]: https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/
[3]: http://code.v.igoro.us/posts/2018/05/design-of-task-graph-generation.html

Comment 8

[Lisa Guo [:lguo] EST]

Here is a summary of the solution you guys came up with at the All-Hands as I understand it; please let me know if I've gotten something wrong! The solution is two-fold; the beetmoverscript task definition and the manifest structure will be modified.

a) The mozilla-central transforms will be modified in order to add an artifactMap to the payload. This maps each upstreamArtifact to its s3 pretty-name and pretty-path (currently called the "s3_key" and "destinations" in the beetmover template files), so that beetmovers can move each artifact to the appropriate location. If I'm not mistaken, each upstreamArtifact should have a mapping in artifactMap; no more and no less. Having the manifest in the payload gets rid of the need for templates in the beetmoverscript source code; we are effectively moving the pretty-naming in-tree.

[task] 

{
  "payload": {
    "upstreamArtifacts": [
      {
          "paths": [
            "public/build/target.tar.bz2"
          ],
          "taskId": "eSzfNqMZT_mSiQQXu8hyqg",
          "locale": "en-US",
          "taskType": "build"
        },
    ]
    "artifactMap": {
      "public/build/target.tar.bz2": {
        "pretty-name": "firefox-{{ version }}.{{ locale }}.{{ platform }}.tar.bz2",   
        "pretty-path": ["21-06-2018-beta-l10n/firefox-61.0b15.en-US.linux-i686.tar.bz2", "latest-beta-l10n/firefox-61.0b15.en-US.linuz-i686.tar.bz2"]
      }
    }
  }
}





b) The payload's artifactMap, as shown above, is a subset of the master manifest, which will be a single-source-of-truth YAML template that will be formatted like the sample below. The actual manifest will contain data for every single artifact; I've provided just two examples below for brevity. As you can see, we will maintain the logic for locale and platform. We plan on aggregating the names and paths for nightly and candidate branches, to have it all in one place. In addition, we may add other fields like "simple" and "developer" names, which may be useful for operations down the pipeline. 



[Sample manifest]

metadata:
    name: "Beet Mover Manifest"
    description: "Maps {{ product }} Nightly artifacts to pretty names for {{ locales }} L10n repack"
    owner: "[e-mail]"

mapping:
{% for locale in locales %}
  {{ locale }}:
    public/build/target.langpack.xpi:
      simple: target.langpack.xpi
      dev: langpack.xpi
      nightly: 
        pretty-name: firefox-{{ version }}.{{ locale }}.langpack.xpi
        pretty-path: 
          - pub/firefox/nightly/{{ upload_date }}-{{ branch }}-l10n/{{ platform }}/xpi/firefox-{{ version }}.{{ locale }}.langpack.xpi
          - pub/firefox/nightly/latest-{{ branch }}-l10n/{{ platform }}/xpi/firefox-{{ version }}.{{ locale }}.langpack.xpi
      candidate: 
        pretty-name: {{ platform }}/xpi/{{ locale }}.xpi
        pretty-path: 
          - pub/firefox/candidates/{{ version }}-candidates/build{{ build_number }}/{{ platform }}/xpi/{{ locale }}.xpi

  {% if platform in ["linux-i686", "linux-x86_64"] %}
    public/build/target.tar.bz2:
      simple: target.tar.bz2
      dev: target.tar.bz2
      nightly: 
        pretty-name: firefox-{{ version }}.{{ locale }}.{{ platform }}.tar.bz2   
        pretty-path: 
          - pub/firefox/nightly/{{ upload_date }}-{{ branch }}/firefox-{{ version }}.{{ locale }}.{{ platform }}.tar.bz2
          - pub/firefox/nightly/latest-{{ branch }}/firefox-{{ version }}.{{ locale }}.{{ platform }}.tar.bz2
          - pub/firefox/nightly/latest-{{ branch }}-l10n/firefox-{{ version }}.{{ locale }}.{{ platform }}.tar.bz2
      candidate:
        pretty-name: {{ platform }}/{{ locale }}/firefox-{{ version }}.tar.bz2   
        pretty-path: 
          - pub/firefox/nightly/{{ platform }}/{{ locale }}/firefox-{{ version }}.tar.bz2
        }
      }
    }
 



Comments, feedback, nitpicks? Please let me know if you anything cleared up!

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

We talked about this over our Friday 1x1. Please flag me again with NI if I need to go with a more nit-picking comment ;-)

Comment 10

[Lisa Guo [:lguo] EST]

Some initial feedback from the Releng team resulted in the following 2 amendments to design:



***
(1) Ammendmend to artifactMap design
***

We should nest the artifacts of artifactMap inside a taskId-key dict. This is because we may have duplicate paths across different taskIds in the upstreamArtifacts.

[ design sketch ]

artifactMap: { taskId: { path1: ...}}




[ example ] 

    "artifactMap": {
      "eSzfNqMZT_mSiQQXu8hyqg" : {
        "public/build/target.tar.bz2": {
          "pretty-name": "firefox-{{ version }}.{{ locale }}.{{ platform }}.tar.bz2",   
          "pretty-path": ["21-06-2018-beta-l10n/firefox-61.0b15.en-US.linux-i686.tar.bz2", "latest-beta-l10n/firefox-61.0b15.en-US.linuz-i686.tar.bz2"]
      }
    }






***
(2) Amendment to the manifest template design
***

The manifest shouldn't have logic; we're going for templatized strings. There's bits and pieces of logic here and there that will have to be moved into beetmoverscript later on. (this is a TODO for me later)


[ design sketch ] 
I tried to illustrate the structure below. The real thing is formatted as a YAML file.



            PLATFORM                  ARTIFACT                 TYPE/PRODUCT 


mapping ---- win32 ----  "public/build/target.complete.mar" ----- simple ----- "target.complete.mar"
             win64                                              default ---- "target.complete.mar"
           linux-i686                                           devedition  |
           linux-x86_6                                            fennec    |----- | nightly   |----------| pretty name
              mac                                                 firefox   |      | candidate |          | pretty path  
                                                               thunderbird  |

Comment 11

[Aki Sasaki (not active)]

(In reply to Lisa Guo [:lguo] EST from comment #8)
> Here is a summary of the solution you guys came up with at the All-Hands as
> I understand it; please let me know if I've gotten something wrong! The
> solution is two-fold; the beetmoverscript task definition and the manifest
> structure will be modified.
> 
> a) The mozilla-central transforms will be modified in order to add an
> artifactMap to the payload. This maps each upstreamArtifact to its s3
> pretty-name and pretty-path (currently called the "s3_key" and
> "destinations" in the beetmover template files), so that beetmovers can move
> each artifact to the appropriate location. If I'm not mistaken, each
> upstreamArtifact should have a mapping in artifactMap; no more and no less.
> Having the manifest in the payload gets rid of the need for templates in the
> beetmoverscript source code; we are effectively moving the pretty-naming
> in-tree.
> 
> [task] 
> 
> {
>   "payload": {
>     "upstreamArtifacts": [
>       {
>           "paths": [
>             "public/build/target.tar.bz2"
>           ],
>           "taskId": "eSzfNqMZT_mSiQQXu8hyqg",
>           "locale": "en-US",
>           "taskType": "build"
>         },
>     ]
>     "artifactMap": {
>       "public/build/target.tar.bz2": {

This path may need to include the taskId, since it's possible we could get two duplicate paths across two different taskIds. We already download artifacts into `work_dir`/cot/`task_id`/`path`, so prepending the task_id to the path should map well.

>         "pretty-name": "firefox-{{ version }}.{{ locale }}.{{ platform
> }}.tar.bz2",   

I'm not sure if we need to pass this in, if it duplicates info in pretty-path?

>         "pretty-path":
> ["21-06-2018-beta-l10n/firefox-61.0b15.en-US.linux-i686.tar.bz2",
> "latest-beta-l10n/firefox-61.0b15.en-US.linuz-i686.tar.bz2"]

Most likely `paths` rather than `path` if it's a list. I'm guessing we want to start with `candidates/` for candidates, and `nightly` for nightlies.

We may need to pass in some more metadata for beetmoverscript to verify these, e.g. version, locale, platform. These could go in a release config dict or similar.

>       }
>     }
>   }
> }
> 
> 
> 
> 
> 
> b) The payload's artifactMap, as shown above, is a subset of the master
> manifest, which will be a single-source-of-truth YAML template that will be
> formatted like the sample below. The actual manifest will contain data for
> every single artifact; I've provided just two examples below for brevity. As
> you can see, we will maintain the logic for locale and platform. We plan on
> aggregating the names and paths for nightly and candidate branches, to have
> it all in one place. In addition, we may add other fields like "simple" and
> "developer" names, which may be useful for operations down the pipeline. 
> 
> 
> 
> [Sample manifest]

This manifest looks like the jinja template in beetmoverscript.
Pretty sure we'll want something more yaml and a simple dict with no logic.

nickname (e.g. windows-stub-installer):
    description: the windows stub installer
    simple-name: target-stub.exe
    nightly-paths:
        - nightly/{nightly_dated_dir}/{dash_build_id}-{branch}/{product}-{version}.en-US.{os}.installer-stub.exe
    candidates-paths:
        - candidates/{version}-candidates/build{build_number}/{os}/en-US/{Product}%20Setup%20{version}.exe
windows-stub-installer-l10n:
    ...

Part of the trick of creating these templates is naming everything. For instance, we have `firefox` and `Firefox`; I tentatively named these `product` and `Product`. The `os` string is sometimes `win` and sometimes `win32` and possibly sometimes `Windows`; we may need separate names for each of these for the template string. We'll need to document what each of them is used for for posterity.

Comment 12

[Aki Sasaki (not active)]

(In reply to Lisa Guo [:lguo] EST from comment #10)
> Some initial feedback from the Releng team resulted in the following 2
> amendments to design:
> 
> 
> 
> ***
> (1) Ammendmend to artifactMap design
> ***
> 
> We should nest the artifacts of artifactMap inside a taskId-key dict. This
> is because we may have duplicate paths across different taskIds in the
> upstreamArtifacts.
> 
> [ design sketch ]
> 
> artifactMap: { taskId: { path1: ...}}
> 
> 
> 
> 
> [ example ] 
> 
>     "artifactMap": {
>       "eSzfNqMZT_mSiQQXu8hyqg" : {
>         "public/build/target.tar.bz2": {
>           "pretty-name": "firefox-{{ version }}.{{ locale }}.{{ platform
> }}.tar.bz2",   
>           "pretty-path":
> ["21-06-2018-beta-l10n/firefox-61.0b15.en-US.linux-i686.tar.bz2",
> "latest-beta-l10n/firefox-61.0b15.en-US.linuz-i686.tar.bz2"]
>       }
> 

This could work too; the `pretty-name` could help verify the `pretty-path`, as long as we have some path sanity checks as well.

> ***
> (2) Amendment to the manifest template design
> ***
> 
> The manifest shouldn't have logic; we're going for templatized strings.
> There's bits and pieces of logic here and there that will have to be moved
> into beetmoverscript later on. (this is a TODO for me later)
> 
> 
> [ design sketch ] 
> I tried to illustrate the structure below. The real thing is formatted as a
> YAML file.
> 
> 
> 
>             PLATFORM                  ARTIFACT                 TYPE/PRODUCT 
> 
> 
> mapping ---- win32 ----  "public/build/target.complete.mar" ----- simple
> ----- "target.complete.mar"
>              win64                                              default ----
> "target.complete.mar"
>            linux-i686                                           devedition  |
>            linux-x86_6                                            fennec   
> |----- | nightly   |----------| pretty name
>               mac                                                 firefox  
> |      | candidate |          | pretty path  
>                                                                thunderbird  |

I'm not entirely sure what this mapping refers to... if this and my description above don't match up, we may want to revisit before you sink a lot of time into this.

Comment 13

[Lisa Guo [:lguo] EST]

I was trying to be fancy with the mapping but guess that wasn't a good idea... essentially, the manifest will model the following structure:

os_platform: {
  artifact: {
    type: {
      artifact-string OR 
      {
        candidate: {pretty-name: ..., pretty-path: ...}, 
        nightly: {pretty-name: ..., pretty-path: ...}
      }
    }
  }
}

Hopefully that's more clear.

Comment 14

[Aki Sasaki (not active)]

(In reply to Lisa Guo [:lguo] EST from comment #13)
> I was trying to be fancy with the mapping but guess that wasn't a good
> idea... essentially, the manifest will model the following structure:
> 
> os_platform: {
>   artifact: {
>     type: {
>       artifact-string OR 
>       {
>         candidate: {pretty-name: ..., pretty-path: ...}, 
>         nightly: {pretty-name: ..., pretty-path: ...}
>       }
>     }
>   }
> }
> 
> Hopefully that's more clear.

+1, but I don't know if we have to split by os_platform? We'll also need a simple name. We might need to allow for multiple paths in candidates and nightly, or maybe that's unneeded.
The yaml format at the bottom of comment #11 may work here.

Comment 15

[Lisa Guo [:lguo] EST]

(In reply to Aki Sasaki [:aki] from comment #14)
> simple name. We might need to allow for multiple paths in candidates and
> nightly, or maybe that's unneeded.
> The yaml format at the bottom of comment #11 may work here.

+1 Yes, we'll have to allow for multiple paths, exactly

Comment 16

[(not currently active) Ted Mielczarek]

Sorry for not looking at this sooner, the timing with my vacation wasn't great! One thing I don't quite get here is how this will map to individual tasks. Currently most tasks have no idea what artifacts they will upload, they just provide an entire directory to be published as `public/build`:
https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/taskcluster/taskgraph/transforms/job/common.py#41

Somehow we need a mapping that connects things like "this task is a 64-bit Windows build, it will produce a public/build/target.exe", right?

I guess what I'm missing is--what populates the `artifactMap`? You mention putting it in the transform, but I can't connect that to the example you've provided in https://github.com/mozilla-releng/beetmoverscript/pull/167 .

Happy to chat more about this on slack or vidyo or whatever!

Comment 17

[(not currently active) Ted Mielczarek]

We chatted a bit about this on Vidyo and I feel good about it. I think the missing piece is that we'll want to have the build transforms put artifact names into the build tasks as well, so that we can plumb that down into the build system for individual builds so they know to generate artifacts with the expected names. That sounds like it will be straightforward to implement given this manifest.

One other item that I thought of last minute is that we'd like to have tags or aliases or something like that for artifacts so that various parts of the task graph can get artifacts from build tasks without having to resort to regexes or hardcoding platform-specific filenames. Ideally the build task would have enough information attached that when generating downstream tasks like test runs we could have code to say "give me the artifact filename for the build package" or "give me the artifact name for the symbols package".

Comment 18

[Lisa Guo [:lguo] EST]

Mihai, Ted and I just had a chat. Here are the conclusions drawn from that conversation. The tagging solution in [2] should solve Ted's last item in the comment above.

[1] 
We should reorganize the manifest by artifact first. 
Currently, we have it organized like: platform-artifact-product
Instead, it should be: artifact-platform-product
Rationale: The motivation behind this manifest is primarily to make it easy for developers to add new artifact.

[2]
We should add tagging for each artifact so that any task can search the manifest to know which artifacts it needs.
Currently, this information is scattered around various make files, I believe.
Instead, having it all in the manifest makes it easier to maintain.

For example, we could tag "target.crashreporter-symbols.zip" with 
    tags: ["symbols"]
... or any other tags like "build", "tests", "dev", etc. that other teams may need.


[3]
Proposition: let's enforce an API for manifest data in task payloads. All tasks should store manifest data in an object called  "artifactMap" in the task payload.
Rationale: having a standard naming convention is clean. 

For example, this is what artifactMap looks like for beetmover:

   "artifactMap": {
      "<task_id>" : {
        "public/build/target.tar.bz2": {
          "destinations":
              - pub/{{product}}/nightly/{{dateddir}}/{{platform}}/{{locale}}/{{product}}-{{version}}-{{locale}}.tar.bz2
              - pub/{{product}}/nightly/latest-{{project}}/...
              - pub/{{ product }}/candidates/{{version}}/{{build number}}/...
         }
      }
   }

The beetmover transforms use the manifest to construct artifactMap in the task payload. Other tasks may choose to structure their artifactMap differently.

Comment 19

[Mihai Tabara [:mtabara]⌚️GMT]

To follow-up, we're currently working on polishing a beetmoverscript implementation to validate our artifactMap + manifest approach. Left aside the transforms for now, yet with them in mind since lots of the changes that were previously in beetmoverscript are going to be in transforms. We're using the manifest[1] PR to track all notes for that, to ease our implementation on transforms soon.

[1]: https://github.com/mozilla-releng/beetmoverscript/pull/167

Comment 20

[Lisa Guo [:lguo] EST]

Attachment: beetmover implementation: consumes artifactMap data structure in task payload

Comment 21

[Mihai Tabara [:mtabara]⌚️GMT]

More work of this validted for Fennec, nightly under bug 1466714.