1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Jordan Lund (:jlund)]

+++ This bug was initially created as a clone of Bug #1282186 +++

where balrogscript is the likely result of 'the generic method' used by scriptworker in: bug 1244181

Comment 1

[Jordan Lund (:jlund)]

like 1282188 (beetmover) and 1282186 (signing), we will want to support publishing on a developer channel. e.g. 'nightly-test' depending on scopes in the graph and, eventually, whether graph is part of 'chain of trust'

Comment 3

[Justin Wood (:Callek)]

...I'm going to patch (in this bug) desktop and fennec

Comment 4

[Justin Wood (:Callek)]

Attachment: Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

Review commit: https://reviewboard.mozilla.org/r/90212/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90212/

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

This is currently a near-done WIP.

I expect to need to change some minor things with the payload format here, but in the meantime this is indeed the bulk of it

Comment 6

[Kim Moir [:kmoir] ET]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

https://reviewboard.mozilla.org/r/90212/#review90102

Looks good.  I ran a try run and the resulting task graph looks good.  The only thing is that I would advise naming beetmover_task as balrog_task in  taskcluster/taskgraph/task/balrog.py for clarity.

Comment 7

[Jordan Lund (:jlund)]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

https://reviewboard.mozilla.org/r/90212/#review91194

::: taskcluster/taskgraph/transforms/task.py:493
(Diff revision 1)
>  
> +@payload_builder('balrog')
> +def build_balrog_payload(config, task, task_def):
> +    worker = task['worker']
> +
> +    task_def['payload'] = {

patch looks great. I have some more updated info with regards to the payload.

beetmoverscript: https://github.com/mozilla-releng/beetmoverscript/pull/9

and balrogworker: https://github.com/mozilla-releng/funsize-balrogworker/pull/2

are both nearing the stage where they both can handle creating a manifest and submitting to balrog.

the payload body from the graph should be as follows:

```
task_def['payload'] = {
    "parent_task_artifacts_url": "https://queue.taskcluster.net/v1/task/<taskID>/artifacts/public",
    "signing_cert": "dep"
    }
```

where <taskID> should be interpolated to the taskid of the beetmoverscript task that beet moves the signingscript task's artifacts. iow - the second beetmoverscript task's taskid that handles the signed artfacts

signing_cert's value doesn't really matter. It probably should be 'dep' for projects/date based nightlies, and "nightly" for central/aurora. However, in our world it's ignored but needed satisfy balrogworker's current expected payload. If you are curious, it is used if balrogworker is also uploading to s3 prior to balrog submission. in our world, we use beetmover for that so we skip s3 upload: https://github.com/mozilla-releng/funsize-balrogworker/blob/master/bin/balrogworker.py#L70
    
once that is on date and beetmoverscript+balrogworker PRs get deployed, we should be looking in good shape for balrog submission

Comment 8

[Jordan Lund (:jlund)]

I can look at this tomorrow if you are working on something else. otherwise, see comment 7 for latest state

Comment 9

[Justin Wood (:Callek)]

(In reply to Jordan Lund (:jlund) from comment #8)
> I can look at this tomorrow if you are working on something else. otherwise,
> see comment 7 for latest state

I'll continue on this, but we have some small pieces to discuss today as well (probably at, or just after, the tc-nightly mtg)

Comment 10

[Jordan Lund (:jlund)]

(In reply to Justin Wood (:Callek) from comment #9)
> (In reply to Jordan Lund (:jlund) from comment #8)
> > I can look at this tomorrow if you are working on something else. otherwise,
> > see comment 7 for latest state
> 
> I'll continue on this, but we have some small pieces to discuss today as
> well (probably at, or just after, the tc-nightly mtg)

via irc:

<Callek>  sooo, we create the balrog manifest from "signed" -- I recognize we need balrog to depend on the signing of the apk for fennec, but for desktop we'll be signing some needed stuff in two passes, .mar and .tar.bz2 -- which does balrogworker need to depend on (or does it need both)?

<jlund>: looks like for desktop we only care about the .mar in terms of balrog submission 
11:09:25 yeah. we don't care about hte tar.bz2

Comment 11

[Justin Wood (:Callek)]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/90212/diff/1-2/

Comment 12

[Jordan Lund (:jlund)]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

https://reviewboard.mozilla.org/r/90212/#review92136

r+ with two comments that are not date blocking but should be addressed or commented on.

::: taskcluster/taskgraph/task/balrog.py:19
(Diff revision 2)
> +    """
> +
> +    @classmethod
> +    def get_inputs(cls, kind, path, config, params, loaded_tasks):
> +        if config.get('kind-dependencies', []) != ["beetmover"]:
> +            raise Exception("Balrog kinds must depend on builds or signing builds")

copy/paste from beetmoverscript taskgraph? shouldn't this be we depend on beetmover tasks?

::: taskcluster/taskgraph/transforms/task.py:318
(Diff revision 2)
>          Required('update_manifest'): bool,
> +    }, {
> +        Required('implementation'): 'balrog',
> +
> +        # taskid of the signed beetmoved task
> +        Required('taskid', default="invalid"): taskref_or_string,

should we include the payload requirements in this schema like beetmover tasks?

Comment 13

[Justin Wood (:Callek)]

(In reply to Jordan Lund (:jlund) from comment #12)
> Comment on attachment 8806972 [details]
> Bug 1282187 - create task that publishes nightlies on update server via
> scriptworker -> balrogscript.
> 
> https://reviewboard.mozilla.org/r/90212/#review92136
> 
> r+ with two comments that are not date blocking but should be addressed or
> commented on.
> 
> ::: taskcluster/taskgraph/transforms/task.py:318
> (Diff revision 2)
> >          Required('update_manifest'): bool,
> > +    }, {
> > +        Required('implementation'): 'balrog',
> > +
> > +        # taskid of the signed beetmoved task
> > +        Required('taskid', default="invalid"): taskref_or_string,
> 
> should we include the payload requirements in this schema like beetmover
> tasks?

*Maybe*. I wasn't really a fan of potential duplication, nor am I a big fan of a direct payload passthrough outside of `implementation` of the worker (below).

Either way, this isn't hard to change.

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/90212/diff/2-3/

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

https://reviewboard.mozilla.org/r/90212/#review92298

::: taskcluster/taskgraph/transforms/balrog.py:84
(Diff revision 3)
> +            'description': "{} Balrog".format(
> +                dep_job.task["metadata"]["description"]),
> +            # do we have to define worker type somewhere?
> +            'worker-type': 'scriptworker-prov-v1/balrogworker-v1',
> +            'worker': {'implementation': 'balrog',
> +                       'taskid': {"task-reference":

There's something wrong here - it doesn't properly subtitute later on. 

  "payload": {
    "signing_cert": "dep",
    "parent_task_artifacts_url": "https://queue.taskcluster.net/v1/task/{u'task-reference': u'<beetmover>'}/artifacts/public"
  },

Comment 16

[Justin Wood (:Callek)]

Landed a bustage fix...

https://hg.mozilla.org/projects/date/rev/b5ec815f13d099e6a35dab57ba6eb49f51b8ec8a

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Do you mind landing another to change `_` to `-` in that description attribute?  The `_` in the payload are fine, but everything *before* the task definition should be `-`-separated.

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8806972 [details]
Bug 1282187 - create task that publishes nightlies on update server via scriptworker -> balrogscript.

https://reviewboard.mozilla.org/r/90212/#review92298

> There's something wrong here - it doesn't properly subtitute later on. 
> 
>   "payload": {
>     "signing_cert": "dep",
>     "parent_task_artifacts_url": "https://queue.taskcluster.net/v1/task/{u'task-reference': u'<beetmover>'}/artifacts/public"
>   },

Thanks Callek for pushing fix for this - https://hg.mozilla.org/projects/date/rev/b5ec815f13d099e6a35dab57ba6eb49f51b8ec8a