1314678 - [meta] Support artifact builds on try for macosx64 on TaskCluster

Status: RESOLVED FIXED

(Testing :: General, defect, P2)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

See https://treeherder.mozilla.org/#/jobs?repo=try&revision=805d3f2a4af1&selectedJob=30324225 (OS X 10.7 opt/debug: build-macosx64/opt, build-macosx64/debug)

Fails with:
>mozbuild.configure.options.InvalidOptionError: --with-ccache is not available in this configuration

This is a Tier-2 build, so it was beyond the scope of Bug 1299702.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

I discussed this bug with Maja on IRC and we will make it a tracking bug. Individual work items should be filed as new bugs blocking this meta bug. With that we can ensure to be able to star failures on Treeherder.

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1314678 - Add variants for OS X cross artifact builds in mozharness.

Review commit: https://reviewboard.mozilla.org/r/116868/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116868/

Comment 4

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1314678 - Unset TOOLCHAIN_PREFIX in mozconfig.artifact

Review commit: https://reviewboard.mozilla.org/r/116870/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116870/

Comment 5

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1314678 - Do not include mozconfig.cache in cross-mozconfig.common.

It's already included by the nightly and debug mozconfigs which end up including
cross-mozconfig.common, including it again appears ineffectual and causes
failures in artifact builds.

Review commit: https://reviewboard.mozilla.org/r/116872/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116872/

Comment 6

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1314678 - Don't perform clang plugin checks in artifact builds.

Until we have the ability to properly make these flags conditional in
moz.configure, do not perform these checks.

Review commit: https://reviewboard.mozilla.org/r/116874/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116874/

Comment 7

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

Mozinstall uses hdiutil, which doesn't exist on cross osx-linux builders.

Review commit: https://reviewboard.mozilla.org/r/116876/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/116876/

Comment 8

[Michael Shal [:mshal]]

Comment on attachment 8843108 [details]
Bug 1314678 - Unset TOOLCHAIN_PREFIX in mozconfig.artifact

https://reviewboard.mozilla.org/r/116870/#review118840

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8843109 [details]
Bug 1314678 - Do not include mozconfig.cache in cross-mozconfig.common.

https://reviewboard.mozilla.org/r/116872/#review118842

Comment 10

[Michael Shal [:mshal]]

Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

https://reviewboard.mozilla.org/r/116874/#review118846

::: build/autoconf/clang-plugin.m4:11
(Diff revision 1)
>  
>  MOZ_ARG_ENABLE_BOOL(clang-plugin,
>  [  --enable-clang-plugin   Enable building with the mozilla clang plugin ],
>     ENABLE_CLANG_PLUGIN=1,
>     ENABLE_CLANG_PLUGIN= )
> -if test -n "$ENABLE_CLANG_PLUGIN"; then
> +if test -n "$ENABLE_CLANG_PLUGIN" -a "$COMPILE_ENVIRONMENT"; then

This seems to be the wrong place to check for COMPILE_ENVIRONMENT. The AC_SUBST(ENABLE_CLANG_PLUGIN) is set outside of this if, which means if you --enable-clang-plugin without a compile environment, I think you'll end up with ENABLE_CLANG_PLUGIN set but you'll have skipped all the checks for it (like that you actually have clang installed).

I imagine it might be better to put the if around the call to MOZ_CONFIG_CLANG_PLUGIN in old-configure.in / js/src/old-configure.in, or maybe provide a stub AC_DEFUN([MOZ_CONFIG_CLANG_PLUGIN], ...) when COMPILE_ENVIRONMENT is not set.

Comment 11

[Michael Shal [:mshal]]

Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

https://reviewboard.mozilla.org/r/116876/#review118848

::: python/mozbuild/mozbuild/artifacts.py:317
(Diff revision 1)
> +        oldcwd = os.getcwd()
>          try:
>              self.log(logging.INFO, 'artifact',
>                  {'tempdir': tempdir},
>                  'Unpacking DMG into {tempdir}')
> -            mozinstall.install(filename, tempdir) # Doesn't handle already mounted DMG files nicely:
> +            if self._substs['HOST_OS_ARCH'] == 'Linux':

Why not just add this to mozinstall? You could check the host type without using substs and do a hfs+dmg or hdiutil branch in there, right?

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

https://reviewboard.mozilla.org/r/116876/#review118848

> Why not just add this to mozinstall? You could check the host type without using substs and do a hfs+dmg or hdiutil branch in there, right?

Well, the paths to hfs and dmg tools are supplied by build config, so we'd have to pass those in, and the constants for platform detection in mozinstall.py aren't going to find the host type correctly. I don't really think it's worth it.

Comment 13

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843107 [details]
Bug 1314678 - Add variants for OS X cross artifact builds in mozharness.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116868/diff/1-2/

Comment 14

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843108 [details]
Bug 1314678 - Unset TOOLCHAIN_PREFIX in mozconfig.artifact

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116870/diff/1-2/

Comment 15

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843109 [details]
Bug 1314678 - Do not include mozconfig.cache in cross-mozconfig.common.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116872/diff/1-2/

Comment 16

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116874/diff/1-2/

Comment 17

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116876/diff/1-2/

Comment 18

[Michael Shal [:mshal]]

Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

https://reviewboard.mozilla.org/r/116874/#review118878

Comment 19

[Michael Shal [:mshal]]

Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

https://reviewboard.mozilla.org/r/116876/#review118884

::: python/mozbuild/mozbuild/artifacts.py:326
(Diff revision 2)
> -            #   File "/Users/nalexander/Mozilla/gecko/mobile/android/mach_commands.py", line 250, in artifact_install
> -            #     return artifacts.install_from(source, self.distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 457, in install_from
> -            #     return self.install_from_hg(source, distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 445, in install_from_hg
> -            #     return self.install_from_url(url, distdir)
> +                    subprocess.check_call([
> +                        self._substs['DMG_TOOL'],
> +                        'extract',
> +                        filename,
> +                        'extracted_img',
> +                    ], stdout=devnull, stderr=devnull)

As mentioned in IRC, please keep stderr.

::: python/mozbuild/mozbuild/artifacts.py:331
(Diff revision 2)
> -            #     return self.install_from_url(url, distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 418, in install_from_url
> -            #     return self.install_from_file(filename, distdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/artifacts.py", line 336, in install_from_file
> -            #     mozinstall.install(filename, tempdir)
> -            #   File "/Users/nalexander/Mozilla/gecko/objdir-dce/_virtualenv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 117, in install
> +                    ], stdout=devnull, stderr=devnull)
> +                    subprocess.check_call([
> +                        self._substs['HFS_TOOL'],
> +                        'extracted_img',
> +                        'extractall'
> +                    ], stdout=devnull, stderr=devnull)

And stderr here as well.

Comment 20

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843107 [details]
Bug 1314678 - Add variants for OS X cross artifact builds in mozharness.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116868/diff/2-3/

Comment 21

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843108 [details]
Bug 1314678 - Unset TOOLCHAIN_PREFIX in mozconfig.artifact

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116870/diff/2-3/

Comment 22

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843109 [details]
Bug 1314678 - Do not include mozconfig.cache in cross-mozconfig.common.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116872/diff/2-3/

Comment 23

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116874/diff/2-3/

Comment 24

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116876/diff/2-3/

Comment 25

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8843107 [details]
Bug 1314678 - Add variants for OS X cross artifact builds in mozharness.

https://reviewboard.mozilla.org/r/116868/#review119608

Comment 26

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 2497e1da722d -d 11f369a25fba: rebasing 379934:2497e1da722d "Bug 1314678 - Add variants for OS X cross artifact builds in mozharness. r=maja_zf"
merging testing/mozharness/mozharness/mozilla/building/buildbase.py
warning: conflicts while merging testing/mozharness/mozharness/mozilla/building/buildbase.py! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)

Comment 27

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843107 [details]
Bug 1314678 - Add variants for OS X cross artifact builds in mozharness.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116868/diff/3-4/

Comment 28

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843108 [details]
Bug 1314678 - Unset TOOLCHAIN_PREFIX in mozconfig.artifact

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116870/diff/3-4/

Comment 29

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843109 [details]
Bug 1314678 - Do not include mozconfig.cache in cross-mozconfig.common.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116872/diff/3-4/

Comment 30

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843110 [details]
Bug 1314678 - Don't perform clang plugin checks in artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116874/diff/3-4/

Comment 31

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8843111 [details]
Bug 1314678 - Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/116876/diff/3-4/

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/76e82700067d
Add variants for OS X cross artifact builds in mozharness. r=maja_zf
https://hg.mozilla.org/integration/autoland/rev/9374cadfa451
Unset TOOLCHAIN_PREFIX in mozconfig.artifact r=mshal
https://hg.mozilla.org/integration/autoland/rev/3b808edbbf3e
Do not include mozconfig.cache in cross-mozconfig.common. r=mshal
https://hg.mozilla.org/integration/autoland/rev/40bf14322955
Don't perform clang plugin checks in artifact builds. r=mshal
https://hg.mozilla.org/integration/autoland/rev/ebe143b04b5b
Use hfs and dmg tools rather than mozinstall to unpack downloaded builds during cross artifact builds. r=mshal

Comment 33

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/76e82700067d
https://hg.mozilla.org/mozilla-central/rev/9374cadfa451
https://hg.mozilla.org/mozilla-central/rev/3b808edbbf3e
https://hg.mozilla.org/mozilla-central/rev/40bf14322955
https://hg.mozilla.org/mozilla-central/rev/ebe143b04b5b