Land more Nightly OSX support to central

RESOLVED FIXED in mozilla56

Status

task
RESOLVED FIXED
2 years ago
Last year

People

(Reporter: Callek, Assigned: Callek)

Tracking

unspecified
mozilla56

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(6 attachments, 1 obsolete attachment)

We should finish out OSX nightly support, based on what is on date right now.  (Some of the refactored changes here will be in support of Windows-on-tc, but won't be enabling for any windows stuff)
Comment on attachment 8878653 [details]
Bug 1373326 - Use macapp instead of dmg signing format.

https://reviewboard.mozilla.org/r/149944/#review154642
Attachment #8878653 - Flags: review?(aki) → review+
Comment on attachment 8878654 [details]
Bug 1373326 - Reformat repackage tasks taskgraph to support new input style.

https://reviewboard.mozilla.org/r/149946/#review154644

::: taskcluster/taskgraph/transforms/job/__init__.py:98
(Diff revision 1)
>  
>  @transforms.add
>  def validate(config, jobs):
>      for job in jobs:
>          yield validate_schema(job_description_schema, job,
> -                              "In job {!r}:".format(job['name']))
> +                              "In job {!r}:".format(job.get('name', job.get('label'))))

This is needed because job['name'] is optional, and we accept label instead, but then fail if we don't have job['name'] here.

::: taskcluster/taskgraph/transforms/job/common.py:20
(Diff revision 1)
> -def docker_worker_add_workspace_cache(config, job, taskdesc):
> +def docker_worker_add_workspace_cache(config, job, taskdesc, extra=None):
>      """Add the workspace cache based on the build platform/type and level,
> -    except on try where workspace caches are not used."""
> +        except on try where workspace caches are not used.
> +
> +    extra, is an optional kwarg passed in that supports extending the cache
> +           key name to avoid undesired conflicts with other caches."""

Need to fix indenting here...

::: taskcluster/taskgraph/transforms/repackage.py:111
(Diff revision 1)
> -        dependencies = {dep_job.attributes.get('kind'): dep_job.label}
> +        task_env = {}
> +        locale_output_path = ""
> +        if attributes['build_platform'].startswith('macosx'):
> -        if job.get('locale'):
> +            if job.get('locale'):
> -            input_string = 'https://queue.taskcluster.net/v1/task/<nightly-l10n-signing>/' + \
> -                'artifacts/public/build/{}/target.tar.gz'
> +                input_string = 'https://queue.taskcluster.net/v1/task/' + \
> +                    '{}/artifacts/public/build/{}/target.tar.gz'

Whops, should be <{}> I'll fix.
Comment on attachment 8878655 [details]
Bug 1373326 - Reformat repackage mozharness code to support new mach repackage stuff.

https://reviewboard.mozilla.org/r/149948/#review154646

::: testing/mozharness/configs/repackage/osx_signed.py:14
(Diff revision 1)
> +    },
> +
> +    "repackage_config": [[
> +        "dmg",
> +        "-i", "{abs_work_dir}/inputs/target.tar.gz",
> +        "-o", "{output_home}/target.dmg"

We'll probably need similar changes in win_signed.py, though that's probably another bug.

::: testing/mozharness/scripts/repackage.py:71
(Diff revision 1)
> -        outfile = os.path.join(dirs['abs_upload_dir'], config['output_filename'])
> -        command = [sys.executable, 'mach', '--log-no-times', 'repackage',
> -                   'dmg',
> -                   '--input', infile,
> -                   '--output', outfile]
> -        return self.run_command(
> +        # Make sure the upload dir is around.
> +        self.mkdir_p(dirs['output_home'])
> +
> +        for repack_config in config["repackage_config"]:
> +            command = [sys.executable, 'mach', '--log-no-times', 'repackage'] + \
> +                [arg.format(**dirs)

smart.
Attachment #8878655 - Flags: review?(aki) → review+
Comment on attachment 8878654 [details]
Bug 1373326 - Reformat repackage tasks taskgraph to support new input style.

https://reviewboard.mozilla.org/r/149946/#review154644

> Whops, should be <{}> I'll fix.

err actually this was already the case..
Comment on attachment 8878652 [details]
Bug 1373326 - Flesh out docs a bit.

https://reviewboard.mozilla.org/r/149942/#review155706

::: taskcluster/docs/kinds.rst:17
(Diff revision 1)
>  actions for static analysis or to produce instrumented artifacts.
>  
>  build-signing
> ---------------
> +-------------
> +
> +Many builds must be signed, the build-signing task takes the unsigned `build`

s/,/./
Attachment #8878652 - Flags: review?(dustin) → review+
Comment on attachment 8878654 [details]
Bug 1373326 - Reformat repackage tasks taskgraph to support new input style.

https://reviewboard.mozilla.org/r/149946/#review155714

I think the bit of overly complex code should be pretty simple to fix up, so r+ with that change.

::: taskcluster/taskgraph/transforms/repackage.py:75
(Diff revision 2)
>  
>          yield job
>  
>  
>  @transforms.add
>  def make_task_description(config, jobs):

`make_job_description`

::: taskcluster/taskgraph/transforms/repackage.py:94
(Diff revision 2)
>          treeherder.setdefault('kind', 'build')
> +        signing_task = None
> +        for dependency in dependencies.keys():
> +            if 'signing' in dependency:
> +                signing_task = dependency
> +        signing_task_ref = "<{}>".format(signing_task)

So on entry to this hunk, `dependencies` is a one-element dictionary mapping the dependent task's kind to its label. You're then iterating over that dependency (so, one loop) and only assigning `signing_task` if the kind name has the substring `signing`.  Otherwise `signing_task` is left as None.  I think that could all get simplified a bit, and the substring check could be avoided.
Attachment #8878654 - Flags: review?(dustin) → review+
Comment on attachment 8878656 [details]
Bug 1373326 - Add beetmover and balrog to taskgraph.

https://reviewboard.mozilla.org/r/149950/#review155746
Attachment #8878656 - Flags: review?(dustin) → review+
Comment on attachment 8878654 [details]
Bug 1373326 - Reformat repackage tasks taskgraph to support new input style.

https://reviewboard.mozilla.org/r/149946/#review155714

> So on entry to this hunk, `dependencies` is a one-element dictionary mapping the dependent task's kind to its label. You're then iterating over that dependency (so, one loop) and only assigning `signing_task` if the kind name has the substring `signing`.  Otherwise `signing_task` is left as None.  I think that could all get simplified a bit, and the substring check could be avoided.

We talked on IRC, dustin won't block as long as I open a new bug for this. The mindset here is that it avoids painful bitrot on merges in the short-term
Attachment #8878651 - Attachment is obsolete: true
Pushed by Callek@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/a5fcfe57e6d4
Flesh out docs a bit. r=dustin
https://hg.mozilla.org/integration/autoland/rev/97b785a31a3c
Use macapp instead of dmg signing format. r=aki
https://hg.mozilla.org/integration/autoland/rev/2fdc6c2ad384
Reformat repackage tasks taskgraph to support new input style. r=dustin
https://hg.mozilla.org/integration/autoland/rev/a678d4b0ee78
Reformat repackage mozharness code to support new mach repackage stuff. r=aki
https://hg.mozilla.org/integration/autoland/rev/7f0177a71077
Add beetmover and balrog to taskgraph. r=dustin
Backed out for breaking decision tasks.

https://hg.mozilla.org/integration/autoland/rev/10b8a212a812553c98a436043ac7c930c8bbf6cb
Flags: needinfo?(bugspam.Callek)
(In reply to Wes Kocher (:KWierso) from comment #24)
> Backed out for breaking decision tasks.
> 
> https://hg.mozilla.org/integration/autoland/rev/
> 10b8a212a812553c98a436043ac7c930c8bbf6cb

Yep, I never make mistakes like failing to run taskgraph full before landing a patchset, ... never .... :/

Anyway, thanks for the backout (at my request) and patch to fix is up for dustin's review.
Flags: needinfo?(bugspam.Callek)
Comment on attachment 8879764 [details]
Bug 1373326 - unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency.

https://reviewboard.mozilla.org/r/151128/#review156258

::: taskcluster/taskgraph/transforms/repackage.py:85
(Diff revision 1)
>              raise NotImplementedError(
>                  "Can't repackage a signing task with multiple dependencies")
> +        signing_dependencies = dep_job.dependencies
> +        # This is so we get the build task in our dependencies to
> +        # have better beetmover support.
> +        dependencies.update(signing_dependencies)

Beetmover's dependency-of-dependency thing is pretty annoying :(
Attachment #8879764 - Flags: review?(dustin) → review+
Pushed by Callek@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/59fd099d38fc
Flesh out docs a bit. r=dustin
https://hg.mozilla.org/integration/autoland/rev/b378d3b26860
Use macapp instead of dmg signing format. r=aki
https://hg.mozilla.org/integration/autoland/rev/c347e7729caf
Reformat repackage tasks taskgraph to support new input style. r=dustin
https://hg.mozilla.org/integration/autoland/rev/c473197afe6f
Reformat repackage mozharness code to support new mach repackage stuff. r=aki
https://hg.mozilla.org/integration/autoland/rev/9ee7ad587c73
Add beetmover and balrog to taskgraph. r=dustin
https://hg.mozilla.org/integration/autoland/rev/9a062dfcb670
unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency. r=dustin
Pushed by Callek@gmail.com:
https://hg.mozilla.org/mozilla-central/rev/54d716a149e6
Flesh out docs a bit. r=dustin
https://hg.mozilla.org/mozilla-central/rev/b66e292f02fa
Use macapp instead of dmg signing format. r=aki
https://hg.mozilla.org/mozilla-central/rev/2496da009b94
Reformat repackage tasks taskgraph to support new input style. r=dustin
https://hg.mozilla.org/mozilla-central/rev/98946e971d8c
Reformat repackage mozharness code to support new mach repackage stuff. r=aki
https://hg.mozilla.org/mozilla-central/rev/5798361cc1a2
Add beetmover and balrog to taskgraph. r=dustin
https://hg.mozilla.org/mozilla-central/rev/132f5498f915
unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency. r=dustin
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.