1178286 - switch release automation source builder to taskcluster

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

This includes task definition and any necessary changes to the script that does this job. If the we have an initial task graph by the time this gets looked at, it should integrate with that, too.

Rail, you said you wanted to use this builder as a test case for trialing out a few things. Assign to you

Comment 2

[Rail Aliiev [:rail]]

Attachment: [WIP] source-mozharness.diff

Switching to mozharness at the same time. Looks like we already have support to package sources there, but it needs some tweaks.

Comment 3

[Rail Aliiev [:rail]]

https://tools.taskcluster.net/task-inspector/#TTm2blD7RL2LIADoB_HSoA/ FTW!

Comment 4

[Rail Aliiev [:rail]]

Attachment: source-railz-mozharness-3.diff

This patch:
 * updates the existing config with missing variables
 * refactors the package_source to use a new wrapper called check_run_command_m() - not sure if it belongs to BuildScript, though.
 * uses `mach configure' instead of `make -f client.mk configure'
 * no need to upload, TC handles that

Example job: https://tools.taskcluster.net/task-inspector/#fn98oVkaQcafZSh89OTVuw/0

A diff between `find | sort > list.txt` of existing 39.0 tarball (on FTP) and the generated by the task above: https://gist.github.com/rail/fff90a1920c2bf444f21

Comment 5

[Rail Aliiev [:rail]]

Attachment: task.json

FTR, this is the task definition I used

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8628458 [details] [diff] [review]
source-railz-mozharness-3.diff

Review of attachment 8628458 [details] [diff] [review]:
-----------------------------------------------------------------

looks great! I can't attest to the targets you are calling and needed cmd args but the code looks good to me. Small questions/comments below.

::: configs/builds/releng_sub_linux_configs/64_source.py
@@ +10,5 @@
>      'purge_minsize': 3,
> +    'buildbot_json_path': 'buildprops.json',
> +    'app_ini_path': 'FAKE',  # Not used, but required by the script
> +    'objdir': 'obj-firefox',
> +    'env': {

sanity check as your env looks minimal: this won't append to any mh base config that you are using in the script call, it will overwrite. Ignore me if you only want 3 vars + os.environ

::: mozharness/mozilla/building/buildbase.py
@@ +1695,5 @@
>                                    halt_on_failure=True)
>          if self.config.get('enable_ccache'):
>              self._ccache_s()
>  
> +    def check_run_command_m(self, *args, **kwargs):

neat! I wonder if this wrapper could go into mock: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/mock.py

@@ +1712,5 @@
> +
> +    def _create_empty_mozconfig(self):
> +        """Create an empty mozconfig file"""
> +        self.log("creating an empty mozconfig...")
> +        with open(os.path.join(self.query_abs_dirs()['abs_src_dir'],

mh has a _touch_file[1] that comes with some logging and exception catching for free. might be applicable here.

@@ +1730,5 @@
>          )
> +        self.check_run_command_m(
> +            command=[
> +                'make', 'source-package', 'hg-bundle',
> +                'HG_BUNDLE_REVISION=%s' % self.query_revision(),

does the env var have to go into the command? I know some make targets required that on certain platforms but if not, extending `env` seems more natural.

Comment 7

[Rail Aliiev [:rail]]

Attachment: source-railz-mozharness-4.diff

interdiff: https://gist.github.com/rail/b7197e5d4c584fd8ade1

Thanks for the comments, they made me simplify the patch.

(In reply to Jordan Lund (:jlund) from comment #6)
> sanity check as your env looks minimal: this won't append to any mh base
> config that you are using in the script call, it will overwrite. Ignore me
> if you only want 3 vars + os.environ

I tried to keep the env as much as possible less polluted. It perfectly works with those 3 variables set.
 
> ::: mozharness/mozilla/building/buildbase.py
> @@ +1695,5 @@
> >                                    halt_on_failure=True)
> >          if self.config.get('enable_ccache'):
> >              self._ccache_s()
> >  
> > +    def check_run_command_m(self, *args, **kwargs):
> 
> neat! I wonder if this wrapper could go into mock:
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/mock.py

Bah, it turns out that I reimplemented halt_on_failure! :) Dropping this from the patch. I don't need any fancy status logic here. If it fails, it's red.


> mh has a _touch_file[1] that comes with some logging and exception catching
> for free. might be applicable here.

Bah #2! I used self._touch_file() instead.
 
> @@ +1730,5 @@
> >          )
> > +        self.check_run_command_m(
> > +            command=[
> > +                'make', 'source-package', 'hg-bundle',
> > +                'HG_BUNDLE_REVISION=%s' % self.query_revision(),
> 
> does the env var have to go into the command? I know some make targets
> required that on certain platforms but if not, extending `env` seems more
> natural.

Even though it looks like an env variable, it's not. It's passed to make as an argument and make uses it to (re)define internal variables.

Comment 8

[Jordan Lund (:jlund)]

Comment on attachment 8628760 [details] [diff] [review]
source-railz-mozharness-4.diff

Review of attachment 8628760 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm! :D

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8628760 [details] [diff] [review]
source-railz-mozharness-4.diff

https://hg.mozilla.org/build/mozharness/rev/4d528b0dfc44

Comment 10

[Rail Aliiev [:rail]]

https://github.com/mozilla/releasetasks/pull/24 and https://github.com/mozilla/releasetasks/pull/26 implement the scheduling part.

Comment 11

[Rail Aliiev [:rail]]

Need to update invocation, probably just MOZHARNESS_SCRIPT env.

https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/Dg-5E5zSSd2vHWJ4LFGHjA/5/public/logs/live_backing.log

Comment 12

[Rail Aliiev [:rail]]

Attachment: source_update.diff

Can't use system wide GCC anymore :/
Have to switch to tooltool

Comment 13

[Rail Aliiev [:rail]]

https://tools.taskcluster.net/task-inspector/#rWoxOnIPRFyyZWkXOQ9C5g/ is my "try" push (to date)

Comment 14

[Rail Aliiev [:rail]]

Still need to fix the artifacts (they are gone now)

Comment 15

[Jordan Lund (:jlund)]

Comment on attachment 8677192 [details] [diff] [review]
source_update.diff

Review of attachment 8677192 [details] [diff] [review]:
-----------------------------------------------------------------

shweet

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8677192 [details] [diff] [review]
source_update.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/c8bca7decb0c

Comment 17

[Rail Aliiev [:rail]]

It turns out that the current script doesn't handle DIST_TARGET_UPLOADS in https://github.com/mozilla/releasetasks/blob/master/releasetasks/templates/source.yml.tmpl#L51

We need to call "make upload", which in TC case will set UPLOAD_HOST to "localhost" in https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/build_pool_specifics.py#46

In this case upload.py just copies files to UPLOAD_PATH, see https://dxr.mozilla.org/mozilla-central/source/build/upload.py#18

To make the solution solid, we could add a new mach command to handle source generation. I have a patch at https://gist.github.com/rail/401b7b9681fb6f44f5e1 which needs some love:

* It times out. The commands take more than default 180 secs. Probably need to tweak the timeout or make the commands print something (not sure if it works in hg-bundle case)
* No default value for --revision
* Maybe make hg-bundle optional

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/c8bca7decb0c

Comment 19

[Rail Aliiev [:rail]]

I pushed https://hg.mozilla.org/projects/date/rev/c6f40a198297 to date to disable tooltool all together, because the new mozconfig doesn't require it.

Comment 20

[Rail Aliiev [:rail]]

and https://hg.mozilla.org/projects/date/rev/c6c535a4c21e to fix a typo (missing coma).

Comment 21

[Rail Aliiev [:rail]]

and http://hg.mozilla.org/projects/date/rev/f2941b773c74 to upload the bundle

Comment 22

[Rail Aliiev [:rail]]

Attachment: source-gecko.diff

* no more tooltool with "source" mozconfig
* fix targets to upload files for reals
* generate signing manifest

Comment 23

[Rail Aliiev [:rail]]

Attachment: fix source tasks

Comment 24

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/94d790995f2c

Comment 25

[Rail Aliiev [:rail]]

Comment on attachment 8722113 [details] [review]
fix source tasks

merged

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 8722112 [details] [diff] [review]
source-gecko.diff

https://hg.mozilla.org/integration/mozilla-inbound/rev/94d790995f2c

Comment 27

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/94d790995f2c