[meta] Support artifact builds on try for macosx64 on TaskCluster

RESOLVED FIXED in Firefox 55

Status

defect
P2
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: maja_zf, Assigned: cmanchester)

Tracking

(Blocks 1 bug)

unspecified
mozilla55
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(5 attachments)

See https://treeherder.mozilla.org/#/jobs?repo=try&revision=805d3f2a4af1&selectedJob=30324225 (OS X 10.7 opt/debug: build-macosx64/opt, build-macosx64/debug)

Fails with:
>mozbuild.configure.options.InvalidOptionError: --with-ccache is not available in this configuration

This is a Tier-2 build, so it was beyond the scope of Bug 1299702.
I discussed this bug with Maja on IRC and we will make it a tracking bug. Individual work items should be filed as new bugs blocking this meta bug. With that we can ensure to be able to star failures on Treeherder.
Depends on: 1315960
Summary: Support artifact builds on try for macosx64 on TaskCluster → [meta] Support artifact builds on try for macosx64 on TaskCluster
No longer depends on: 1315960
Duplicate of this bug: 1344041
Assignee: nobody → cmanchester
Comment on attachment 8843108 [details]
Bug 1314678 - Unset TOOLCHAIN_PREFIX in mozconfig.artifact

https://reviewboard.mozilla.org/r/116870/#review118840
Attachment #8843108 - Flags: review?(mshal) → review+
Comment on attachment 8843109 [details]
Bug 1314678 - Do not include mozconfig.cache in cross-mozconfig.common.

https://reviewboard.mozilla.org/r/116872/#review118842
Attachment #8843109 - Flags: review?(mshal) → review+
Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

https://reviewboard.mozilla.org/r/116874/#review118846

::: build/autoconf/clang-plugin.m4:11
(Diff revision 1)
>  
>  MOZ_ARG_ENABLE_BOOL(clang-plugin,
>  [  --enable-clang-plugin   Enable building with the mozilla clang plugin ],
>     ENABLE_CLANG_PLUGIN=1,
>     ENABLE_CLANG_PLUGIN= )
> -if test -n "$ENABLE_CLANG_PLUGIN"; then
> +if test -n "$ENABLE_CLANG_PLUGIN" -a "$COMPILE_ENVIRONMENT"; then

This seems to be the wrong place to check for COMPILE_ENVIRONMENT. The AC_SUBST(ENABLE_CLANG_PLUGIN) is set outside of this if, which means if you --enable-clang-plugin without a compile environment, I think you'll end up with ENABLE_CLANG_PLUGIN set but you'll have skipped all the checks for it (like that you actually have clang installed).

I imagine it might be better to put the if around the call to MOZ_CONFIG_CLANG_PLUGIN in old-configure.in / js/src/old-configure.in, or maybe provide a stub AC_DEFUN([MOZ_CONFIG_CLANG_PLUGIN], ...) when COMPILE_ENVIRONMENT is not set.
Attachment #8843110 - Flags: review?(mshal)
Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

https://reviewboard.mozilla.org/r/116876/#review118848

::: python/mozbuild/mozbuild/artifacts.py:317
(Diff revision 1)
> +        oldcwd = os.getcwd()
>          try:
>              self.log(logging.INFO, 'artifact',
>                  {'tempdir': tempdir},
>                  'Unpacking DMG into {tempdir}')
> -            mozinstall.install(filename, tempdir) # Doesn't handle already mounted DMG files nicely:
> +            if self._substs['HOST_OS_ARCH'] == 'Linux':

Why not just add this to mozinstall? You could check the host type without using substs and do a hfs+dmg or hdiutil branch in there, right?
Attachment #8843111 - Flags: review?(mshal) → review-
Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

https://reviewboard.mozilla.org/r/116876/#review118848

> Why not just add this to mozinstall? You could check the host type without using substs and do a hfs+dmg or hdiutil branch in there, right?

Well, the paths to hfs and dmg tools are supplied by build config, so we'd have to pass those in, and the constants for platform detection in mozinstall.py aren't going to find the host type correctly. I don't really think it's worth it.
Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

https://reviewboard.mozilla.org/r/116874/#review118878
Attachment #8843110 - Flags: review?(mshal) → review+
Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

https://reviewboard.mozilla.org/r/116876/#review118884

::: python/mozbuild/mozbuild/artifacts.py:326
(Diff revision 2)
> -            #   File "/Users/nalexander/Mozilla/gecko/mobile/android/mach_commands.py", line 250, in artifact_install
> -            #     return artifacts.install_from(source, self.distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 457, in install_from
> -            #     return self.install_from_hg(source, distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 445, in install_from_hg
> -            #     return self.install_from_url(url, distdir)
> +                    subprocess.check_call([
> +                        self._substs['DMG_TOOL'],
> +                        'extract',
> +                        filename,
> +                        'extracted_img',
> +                    ], stdout=devnull, stderr=devnull)

As mentioned in IRC, please keep stderr.

::: python/mozbuild/mozbuild/artifacts.py:331
(Diff revision 2)
> -            #     return self.install_from_url(url, distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 418, in install_from_url
> -            #     return self.install_from_file(filename, distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 336, in install_from_file
> -            #     mozinstall.install(filename, tempdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/objdir-dce/_virtualenv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 117, in install
> +                    ], stdout=devnull, stderr=devnull)
> +                    subprocess.check_call([
> +                        self._substs['HFS_TOOL'],
> +                        'extracted_img',
> +                        'extractall'
> +                    ], stdout=devnull, stderr=devnull)

And stderr here as well.
Attachment #8843111 - Flags: review?(mshal) → review+
Attachment #8843107 - Flags: review?(nalexander)
Comment on attachment 8843107 [details]
Bug 1314678 - Add variants for OS X cross artifact builds in mozharness.

https://reviewboard.mozilla.org/r/116868/#review119608
Attachment #8843107 - Flags: review?(mjzffr) → review+
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 2497e1da722d -d 11f369a25fba: rebasing 379934:2497e1da722d "Bug 1314678 - Add variants for OS X cross artifact builds in mozharness. r=maja_zf"
merging testing/mozharness/mozharness/mozilla/building/buildbase.py
warning: conflicts while merging testing/mozharness/mozharness/mozilla/building/buildbase.py! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/76e82700067d
Add variants for OS X cross artifact builds in mozharness. r=maja_zf
https://hg.mozilla.org/integration/autoland/rev/9374cadfa451
Unset TOOLCHAIN_PREFIX in mozconfig.artifact r=mshal
https://hg.mozilla.org/integration/autoland/rev/3b808edbbf3e
Do not include mozconfig.cache in cross-mozconfig.common. r=mshal
https://hg.mozilla.org/integration/autoland/rev/40bf14322955
Don't perform clang plugin checks in artifact builds. r=mshal
https://hg.mozilla.org/integration/autoland/rev/ebe143b04b5b
Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds. r=mshal
You need to log in before you can comment on or make changes to this bug.