Closed Bug 1373326 Opened 8 years ago Closed 8 years ago

Land more Nightly OSX support to central

Categories

(Firefox Build System :: Task Configuration, task)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla56

People

(Reporter: Callek, Assigned: Callek)

Details

Attachments

(6 files, 1 obsolete file)

Bug 1373326 - Flesh out docs a bit.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1373326 - Use macapp instead of dmg signing format.
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1373326 - Reformat repackage tasks taskgraph to support new input style.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1373326 - Reformat repackage mozharness code to support new mach repackage stuff.
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1373326 - Add beetmover and balrog to taskgraph.
59 bytes, text/x-review-board-request
dustin
: review+
Details
Bug 1373326 - unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency.
59 bytes, text/x-review-board-request
dustin
: review+
Details
We should finish out OSX nightly support, based on what is on date right now. (Some of the refactored changes here will be in support of Windows-on-tc, but won't be enabling for any windows stuff)
Comment on attachment 8878653 [details] Bug 1373326 - Use macapp instead of dmg signing format. https://reviewboard.mozilla.org/r/149944/#review154642
Attachment #8878653 - Flags: review?(aki) → review+
Comment on attachment 8878654 [details] Bug 1373326 - Reformat repackage tasks taskgraph to support new input style. https://reviewboard.mozilla.org/r/149946/#review154644 ::: taskcluster/taskgraph/transforms/job/__init__.py:98 (Diff revision 1) > > @transforms.add > def validate(config, jobs): > for job in jobs: > yield validate_schema(job_description_schema, job, > - "In job {!r}:".format(job['name'])) > + "In job {!r}:".format(job.get('name', job.get('label')))) This is needed because job['name'] is optional, and we accept label instead, but then fail if we don't have job['name'] here. ::: taskcluster/taskgraph/transforms/job/common.py:20 (Diff revision 1) > -def docker_worker_add_workspace_cache(config, job, taskdesc): > +def docker_worker_add_workspace_cache(config, job, taskdesc, extra=None): > """Add the workspace cache based on the build platform/type and level, > - except on try where workspace caches are not used.""" > + except on try where workspace caches are not used. > + > + extra, is an optional kwarg passed in that supports extending the cache > + key name to avoid undesired conflicts with other caches.""" Need to fix indenting here... ::: taskcluster/taskgraph/transforms/repackage.py:111 (Diff revision 1) > - dependencies = {dep_job.attributes.get('kind'): dep_job.label} > + task_env = {} > + locale_output_path = "" > + if attributes['build_platform'].startswith('macosx'): > - if job.get('locale'): > + if job.get('locale'): > - input_string = 'https://queue.taskcluster.net/v1/task/<nightly-l10n-signing>/' + \ > - 'artifacts/public/build/{}/target.tar.gz' > + input_string = 'https://queue.taskcluster.net/v1/task/' + \ > + '{}/artifacts/public/build/{}/target.tar.gz' Whops, should be <{}> I'll fix.
Comment on attachment 8878655 [details] Bug 1373326 - Reformat repackage mozharness code to support new mach repackage stuff. https://reviewboard.mozilla.org/r/149948/#review154646 ::: testing/mozharness/configs/repackage/osx_signed.py:14 (Diff revision 1) > + }, > + > + "repackage_config": [[ > + "dmg", > + "-i", "{abs_work_dir}/inputs/target.tar.gz", > + "-o", "{output_home}/target.dmg" We'll probably need similar changes in win_signed.py, though that's probably another bug. ::: testing/mozharness/scripts/repackage.py:71 (Diff revision 1) > - outfile = os.path.join(dirs['abs_upload_dir'], config['output_filename']) > - command = [sys.executable, 'mach', '--log-no-times', 'repackage', > - 'dmg', > - '--input', infile, > - '--output', outfile] > - return self.run_command( > + # Make sure the upload dir is around. > + self.mkdir_p(dirs['output_home']) > + > + for repack_config in config["repackage_config"]: > + command = [sys.executable, 'mach', '--log-no-times', 'repackage'] + \ > + [arg.format(**dirs) smart.
Attachment #8878655 - Flags: review?(aki) → review+
Comment on attachment 8878654 [details] Bug 1373326 - Reformat repackage tasks taskgraph to support new input style. https://reviewboard.mozilla.org/r/149946/#review154644 > Whops, should be <{}> I'll fix. err actually this was already the case..
Comment on attachment 8878652 [details] Bug 1373326 - Flesh out docs a bit. https://reviewboard.mozilla.org/r/149942/#review155706 ::: taskcluster/docs/kinds.rst:17 (Diff revision 1) > actions for static analysis or to produce instrumented artifacts. > > build-signing > --------------- > +------------- > + > +Many builds must be signed, the build-signing task takes the unsigned `build` s/,/./
Attachment #8878652 - Flags: review?(dustin) → review+
Comment on attachment 8878654 [details] Bug 1373326 - Reformat repackage tasks taskgraph to support new input style. https://reviewboard.mozilla.org/r/149946/#review155714 I think the bit of overly complex code should be pretty simple to fix up, so r+ with that change. ::: taskcluster/taskgraph/transforms/repackage.py:75 (Diff revision 2) > > yield job > > > @transforms.add > def make_task_description(config, jobs): `make_job_description` ::: taskcluster/taskgraph/transforms/repackage.py:94 (Diff revision 2) > treeherder.setdefault('kind', 'build') > + signing_task = None > + for dependency in dependencies.keys(): > + if 'signing' in dependency: > + signing_task = dependency > + signing_task_ref = "<{}>".format(signing_task) So on entry to this hunk, `dependencies` is a one-element dictionary mapping the dependent task's kind to its label. You're then iterating over that dependency (so, one loop) and only assigning `signing_task` if the kind name has the substring `signing`. Otherwise `signing_task` is left as None. I think that could all get simplified a bit, and the substring check could be avoided.
Attachment #8878654 - Flags: review?(dustin) → review+
Comment on attachment 8878656 [details] Bug 1373326 - Add beetmover and balrog to taskgraph. https://reviewboard.mozilla.org/r/149950/#review155746
Attachment #8878656 - Flags: review?(dustin) → review+
Comment on attachment 8878654 [details] Bug 1373326 - Reformat repackage tasks taskgraph to support new input style. https://reviewboard.mozilla.org/r/149946/#review155714 > So on entry to this hunk, `dependencies` is a one-element dictionary mapping the dependent task's kind to its label. You're then iterating over that dependency (so, one loop) and only assigning `signing_task` if the kind name has the substring `signing`. Otherwise `signing_task` is left as None. I think that could all get simplified a bit, and the substring check could be avoided. We talked on IRC, dustin won't block as long as I open a new bug for this. The mindset here is that it avoids painful bitrot on merges in the short-term
Attachment #8878651 - Attachment is obsolete: true
Pushed by Callek@gmail.com: https://hg.mozilla.org/integration/autoland/rev/a5fcfe57e6d4 Flesh out docs a bit. r=dustin https://hg.mozilla.org/integration/autoland/rev/97b785a31a3c Use macapp instead of dmg signing format. r=aki https://hg.mozilla.org/integration/autoland/rev/2fdc6c2ad384 Reformat repackage tasks taskgraph to support new input style. r=dustin https://hg.mozilla.org/integration/autoland/rev/a678d4b0ee78 Reformat repackage mozharness code to support new mach repackage stuff. r=aki https://hg.mozilla.org/integration/autoland/rev/7f0177a71077 Add beetmover and balrog to taskgraph. r=dustin
(In reply to Wes Kocher (:KWierso) from comment #24) > Backed out for breaking decision tasks. > > https://hg.mozilla.org/integration/autoland/rev/ > 10b8a212a812553c98a436043ac7c930c8bbf6cb Yep, I never make mistakes like failing to run taskgraph full before landing a patchset, ... never .... :/ Anyway, thanks for the backout (at my request) and patch to fix is up for dustin's review.
Flags: needinfo?(bugspam.Callek)
Comment on attachment 8879764 [details] Bug 1373326 - unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency. https://reviewboard.mozilla.org/r/151128/#review156258 ::: taskcluster/taskgraph/transforms/repackage.py:85 (Diff revision 1) > raise NotImplementedError( > "Can't repackage a signing task with multiple dependencies") > + signing_dependencies = dep_job.dependencies > + # This is so we get the build task in our dependencies to > + # have better beetmover support. > + dependencies.update(signing_dependencies) Beetmover's dependency-of-dependency thing is pretty annoying :(
Attachment #8879764 - Flags: review?(dustin) → review+
Pushed by Callek@gmail.com: https://hg.mozilla.org/integration/autoland/rev/59fd099d38fc Flesh out docs a bit. r=dustin https://hg.mozilla.org/integration/autoland/rev/b378d3b26860 Use macapp instead of dmg signing format. r=aki https://hg.mozilla.org/integration/autoland/rev/c347e7729caf Reformat repackage tasks taskgraph to support new input style. r=dustin https://hg.mozilla.org/integration/autoland/rev/c473197afe6f Reformat repackage mozharness code to support new mach repackage stuff. r=aki https://hg.mozilla.org/integration/autoland/rev/9ee7ad587c73 Add beetmover and balrog to taskgraph. r=dustin https://hg.mozilla.org/integration/autoland/rev/9a062dfcb670 unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency. r=dustin
Pushed by Callek@gmail.com: https://hg.mozilla.org/mozilla-central/rev/54d716a149e6 Flesh out docs a bit. r=dustin https://hg.mozilla.org/mozilla-central/rev/b66e292f02fa Use macapp instead of dmg signing format. r=aki https://hg.mozilla.org/mozilla-central/rev/2496da009b94 Reformat repackage tasks taskgraph to support new input style. r=dustin https://hg.mozilla.org/mozilla-central/rev/98946e971d8c Reformat repackage mozharness code to support new mach repackage stuff. r=aki https://hg.mozilla.org/mozilla-central/rev/5798361cc1a2 Add beetmover and balrog to taskgraph. r=dustin https://hg.mozilla.org/mozilla-central/rev/132f5498f915 unbreak decision task from beetmover addition, make sure repackage jobs have a 'build' dependency. r=dustin
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: