1413219 - add firefox source task for in-tree release promotion

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Kim Moir [:kmoir] ET]

aki mentioned that he has heard from gerv that we don't actually need to provide a tarball of source, it would be sufficient to provide link to the revision used and instructions on how to build.

That being said, he mentioned that an easy path forward would be to copy the existing fennec source job into the firefox graph

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1413219.patch

This adds the following tasks to the taskgraph.   I thought there was only one generated source but looking at the fennec graph this includes the ones for each platform as well.

upload-generated-sources-linux-devedition-nightly/opt
upload-generated-sources-linux-nightly/opt
upload-generated-sources-linux64-devedition-nightly/opt
upload-generated-sources-linux64-nightly/opt
upload-generated-sources-macosx64-devedition-nightly/opt
upload-generated-sources-macosx64-nightly/opt
upload-generated-sources-win32-devedition-nightly/opt
upload-generated-sources-win32-nightly/opt
upload-generated-sources-win64-devedition-nightly/opt
upload-generated-sources-win64-nightly/opt

Comment 2

[Rail Aliiev [:rail]]

hmm, I think it's not the same source task: https://dxr.mozilla.org/mozilla-central/source/taskcluster/docs/kinds.rst#73

it was implemeted in https://hg.mozilla.org/mozilla-central/rev/4a6df9405fb5

Comment 3

[Kim Moir [:kmoir] ET]

Rail, I'm not sure I understand.  Could you elaboarate? The new tasks include a call to the scripts your reference in the commits in comment 2 

  "metadata": {
        "description": "Upload generated source files from build ([Treeherder push](https://treeherder.mozilla.org/#/jobs?repo=maple&revision=559e44e920bfabfdb4ee39749fb3769a2034407f))",
        "name": "upload-generated-sources-linux-devedition-nightly/opt",
        "owner": "[e-mail]",
        "source": "https://hg.mozilla.org/projects/maple/file/559e44e920bfabfdb4ee39749fb3769a2034407f/taskcluster/ci/upload-generated-sources"
      },   
      "payload": {
        "cache": {
          "level-3-checkouts-36e3b4faee27c3795552": "/builds/worker/checkouts",
          "level-3-maple-dotcache-36e3b4faee27c3795552": "/builds/worker/.cache"
        },   
        "command": [
          "/builds/worker/bin/run-task",
          "--vcs-checkout=/builds/worker/checkouts/gecko",
          "--fetch-hgfingerprint",
          "--",
          "bash",
          "-cx",
          "cd /builds/worker/checkouts/gecko && ./mach python build/upload_generated_sources.py ${ARTIFACT_URL}\n"
        ],   
        "env": {
          "ARTIFACT_URL": {
            "task-reference": "https://queue.taskcluster.net/v1/task/<build>/artifacts/public/build/target.generated-files.tar.gz"

Comment 4

[Rail Aliiev [:rail]]

The name of the task is a bit misleading. From what I see (https://tools.taskcluster.net/groups/b2GhZ0woTOGyJQawUEmzAw/tasks/DJh2XrdGTjelKkvov4kpgA/details) the generated "source" tarballs are partial to a particular task. This is the list of things in one of the tarballs:

tree -L 2                                                                                                                                                                                                                              ✔  15:56:01 
.
├── accessible
│   └── xpcom
├── build
│   └── unix
├── buildid.h
├── dist
│   └── include
├── dom
│   ├── base
│   └── bindings
├── gfx
│   └── cairo
├── ipc
│   └── ipdl
├── js
│   └── src
├── layout
│   └── style
├── mozilla-config.h
├── netwerk
│   └── necko-config.h
├── security
│   └── nss
├── source-repo.h
├── toolkit
│   ├── components
│   └── library
└── xpcom
    ├── base
    └── xpcom-config.h

25 directories, 5 files

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: wip patch

work in progress, still quite a few things to fix

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1413219.patch

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: graph.json

generated part of graph

Comment 8

[Kim Moir [:kmoir] ET]

Comment on attachment 8926978 [details] [diff] [review]
bug1413219.patch

Not sure if this is on the right path or I should take a different approach.  I still have to what tasks it depends on and tasks that depend on it

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8926978 [details] [diff] [review]
bug1413219.patch

LGTM!

I think you don't need to worry of the previous tasks dependency, because you can build the source tarball any time after the push. Still need to figure out how to beetmove this task though.

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1413219-4.patch

rebased patch

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: bug1413219-5.patch

I'll wait for Mihai's beetmover patches to land and then I'll test to ensure the artifacts are beetmoved

Comment 12

[Rail Aliiev [:rail]]

Comment on attachment 8927859 [details] [diff] [review]
bug1413219-5.patch

Review of attachment 8927859 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for the delay. SHIP IT!

Comment 13

[Kim Moir [:kmoir] ET]

Comment on attachment 8927859 [details] [diff] [review]
bug1413219-5.patch

checked in on maple

Comment 14

[Kim Moir [:kmoir] ET]

From conversation with Mihai today

<mtabara> Mihai Tabara kmoir: https://github.com/mozilla-releng/beetmoverscript/pull/92/files
1:14 PM kmoir: beetmover-dev1.srv.releng.use1.mozilla.com
1:20 PM kmoir: https://archive.mozilla.org/pub/firefox/candidates/57.0b9-candidates/build1/beetmover-checksums/
1:22 PM kmoir: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/generate-checksums.py

-Currently we have beetmover-repackage kind which computes checksums
-the checksum signing task signs the checksums
-beetmover-checksums consumes signed artifacts

-need to check if source needs to be signed
-so we just need to add a beetmover-source kind which depends on existing release-source, update dependencies
-beetmover script should just work

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug1413219-bm.patch

So these are my patches which are not working.  Right now, the release-source is generated but not signed.  These patches attempt to sign it and but the release- source-signing task is not included.  I'm not sure that as suggested in irc, the build-signing kind should depend on the release-source kind.  The source isn't generated on push, it is only part of release promotion.  Anyways, I'm stuck and would appreciate pointers in the right direction.

Comment 16

[Aki Sasaki (not active)]

>diff --git a/taskcluster/ci/beetmover-repackage/kind.yml b/taskcluster/ci/beetmover-repackage/kind.yml
>--- a/taskcluster/ci/beetmover-repackage/kind.yml
>+++ b/taskcluster/ci/beetmover-repackage/kind.yml
>@@ -8,16 +8,17 @@ transforms:
>    - taskgraph.transforms.name_sanity:transforms
>    - taskgraph.transforms.beetmover_repackage_l10n:transforms
>    - taskgraph.transforms.beetmover_repackage:transforms
>    - taskgraph.transforms.task:transforms
>
> kind-dependencies:
>   - repackage-signing
>   - partials-signing
>+  - release-source

This should probably be `release-source-signing`, since we want to beetmove the signed bits.
I think we also need to add shipping-phase and shipping-product to release-source.

I think the main problem is that transforms/beetmover_repackage.py is hardcoding kind names:

https://hg.mozilla.org/projects/maple/file/bdb3a072306c/taskcluster/taskgraph/transforms/beetmover_repackage.py#l200
https://hg.mozilla.org/projects/maple/file/bdb3a072306c/taskcluster/taskgraph/transforms/beetmover_repackage.py#l208
https://hg.mozilla.org/projects/maple/file/bdb3a072306c/taskcluster/taskgraph/transforms/beetmover_repackage.py#l224

I think this needs to be rewritten to be more data driven and less logic-driven-with-hardcodes. We may be able to solve this current issue without rewriting.

One solution may be to go back to having the source be a build kind, but a different shipping-phase; then we'd match beetmover-repackage's kind expectations. Another may be to add more hardcodes to beetmover_repackage.py.

Comment 17

[Kim Moir [:kmoir] ET]

Attachment: bug1413219-bm2.patch

Here are updated patches.  The release-source-signing task is included in the graph now.  However, the beetmover parts are not working. As was noted before, there are values hardcoded in beetmover_repackage.py for signing_name, build_name and repackage_name and these look at the tasks dependencies.  There aren't dependencies for release-source and release-source-signing in the graph so I'm wondering if the best approach is create a new transforms for beetmover-source but that seems like a lot of duplicated code.

Comment 18

[Aki Sasaki (not active)]

12:23 <•aki> kmoir: hm, i'm thinking you should be using beetmover since this isn't a repackage job. am i wrong?
12:23 <•aki> beetmover moves unsigned/signed bits (build+signing); beetmover-repackage moves repackaged bits. the source task doesn't have a repackage task
12:24 <kmoir> okay I will try that,  I was talking to m.ihai about beetmover repackage but perhaps that was in reference to the checksums
12:24 <kmoir> thanks

Comment 19

[Kim Moir [:kmoir] ET]

Attachment: bug1413219-bm3.patch

These patches fix the earlier issues. They include beetmoving the checksum for the source build but this is not yet implemented.  I could comment out temporarily for testing on maple.

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8932623 [details] [diff] [review]
bug1413219-bm3.patch

Review of attachment 8932623 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

We already have target.tar.bz2 + *asc files present in the PR[1], hence the existing beetmoverscript package that the dep beetmoverworker is using so we should be all good to go here!

As long as the beetmover-source tasks are being correctly generated and the artifacts are good, it should just work! \o/

[1]: https://github.com/mozilla-releng/beetmoverscript/pull/92/files#diff-ca319768c840fd3d444edc95cd0fd017R103

Comment 21

[Kim Moir [:kmoir] ET]

Comment on attachment 8932623 [details] [diff] [review]
bug1413219-bm3.patch

Thanks! Now asking for review :-) so I can land on maple

Comment 22

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8932623 [details] [diff] [review]
bug1413219-bm3.patch

Ship-it! \o/

Comment 23

[Kim Moir [:kmoir] ET]

Attachment: bug1413219-debug.patch

rebased patch after aki's patches landed last night.  The issue I'm having is that my new tasks of release-source-signing and beetmover-source are no longer included in the promote graph

Comment 24

[Aki Sasaki (not active)]

Still digging to see why the release-source-signing and beetmover tasks aren't showing up in the *full* graph.

https://public-artifacts.taskcluster.net/M9o0FgB4TYepcqg9blT4Tg/0/public/logs/live_backing.log is the latest source run - we seem to be missing tooltool_script.

[task 2017-11-29T20:23:11.876Z] 20:23:11    FATAL - The key 'tooltool_script' could not be determined and is needed for the action 'build'. Please add this to your config or run without that action (ie: --no-{action})

Comment 25

[Aki Sasaki (not active)]

Attachment: this seemed to fix it

Comment 26

[Kim Moir [:kmoir] ET]

Attachment: bug1413219latest.patch

Comment 27

[Kim Moir [:kmoir] ET]

Comment on attachment 8933063 [details] [diff] [review]
bug1413219latest.patch

landed on maple

Comment 28

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Tweak beetmover-source task to grab balrog-props too

So there's a chicken-egg sort of problem with this one. 

Facts:
* `Beetmover-source depends` on `release-signing-source` which in turns depends on `release-source`
* there's no nightly builds in the chain of dependencies so there's pretty much no way I can add it nicely
* if I dare to add it as a dependency, the beetmover transform will fail because we can't have more than two dependencies in signing[1]
* we need to somehow slip en-US build in the dependencies along with tweaking the upstreamArtifacts list to get balrog_props file
* this is supposed to be a temporary fix until we manage to kill balrog_props in beetmoverscript

Solutions at hand:
1) Hack via a transform the task definition, after beetmover transforms finishes with it.
2) define a separate beetmover-source which would pretty much copy-paste entire beetmover transform but would amend the needs for source file. This would kill the idea of single-sourcing and would require us to make changes in two places every time we need to amend something in beetmover transform
3) make if/else changes in beetmover transform - I guess this can blow off other tasks as well so it's very risky given our tight timeline

I went on with 1) becausee:
i) it's temporary until we get rid of balrog_props
ii) it's not a singular case - we're hacking stuff alike in [2][3] as well in the graph
iii) it works :) 

Please let me know what you think of this.
Thanks!

[1]: https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/transforms/beetmover.py#l363
[2]: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/l10n.py#227
[3]: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/use_toolchains.py#38

Comment 29

[Aki Sasaki (not active)]

Comment on attachment 8935059 [details] [diff] [review]
Tweak beetmover-source task to grab balrog-props too

This works, and I'm glad. I really don't like this hardcode:

> +        job['dependencies']['build'] = u'build-linux64-nightly/opt'

I wonder if we can pass that down.

Let's land this and see if it fixes things, and deal with followups.
Thank you!

Comment 30

[Aki Sasaki (not active)]

We'll probably have to deal with that hardcode before we can create source tasks for fennec or devedition.

Comment 31

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8935059 [details] [diff] [review]
Tweak beetmover-source task to grab balrog-props too

https://hg.mozilla.org/projects/maple/rev/a7e9356f160f78b918eb509492933779a0c198fb

Addressed aki's comments too to condition that hardcode to firefox only for now. Better to deal hard failure than bad behavior.

Comment 32

[Mihai Tabara [:mtabara]⌚️GMT]

This worked like a charm!
http://ftp.stage.mozaws.net/pub/firefox/candidates/58.0b12-candidates/build11/

Will track devedition separately I suppose.