1401166 - ./mach bootstrap fails to install cargo package on BSDs

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Jan Beich]

cargo is no longer a separate package but a part of "rust" on FreeBSD and OpenBSD.

$ ./mach bootstrap
[...]
Please choose the version of Firefox you want to build:
1. Firefox for Desktop Artifact Mode
2. Firefox for Desktop
3. Firefox for Android Artifact Mode
4. Firefox for Android
Your choice: 2
Executing as root: su root -c "pkg install autoconf213 cargo gmake gtar mercurial pkgconf rust watchman zip"
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'cargo' have been found in the repositories
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '[u'su', u'root', u'-c', u'pkg install autoconf213 cargo gmake gtar mercurial pkgconf rust watchman zip']' returned non-zero exit status 70

  File "python/mozboot/mozboot/mach_commands.py", line 32, in bootstrap
    bootstrapper.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 292, in bootstrap
    self.instance.install_system_packages()
  File "python/mozboot/mozboot/freebsd.py", line 51, in install_system_packages
    self.pkg_install(*self.packages)
  File "python/mozboot/mozboot/freebsd.py", line 48, in pkg_install
    self.run_as_root(command)
  File "python/mozboot/mozboot/base.py", line 300, in run_as_root
    subprocess.check_call(command, stdin=sys.stdin)
  File "/usr/local/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)

https://github.com/freebsd/freebsd-ports/commit/0b663e6dddf67c26d69ee9e838a493969939381c
https://github.com/openbsd/ports/commit/5057038dd9dea7c0db2c6727aed89f0e52da61cb
https://github.com/openbsd/ports/commit/da08f3aa7aab21e4c6d96e0679846be6426852aa

Comment 1

[Jan Beich]

Attachment: Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

Review commit: https://reviewboard.mozilla.org/r/181244/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181244/

Comment 2

[Jan Beich]

Attachment: Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

GCC isn't safe to use on architectures that switched to Clang because
libstdc++ and libc++ aren't very compatible. Newer LLVM and Clang are
often already installed as a dependency for Mesa packages. So, always
require llvm* package.

Review commit: https://reviewboard.mozilla.org/r/181246/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181246/

Comment 3

[Jan Beich]

Attachment: Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD. f=gaston

Review commit: https://reviewboard.mozilla.org/r/181248/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181248/

Comment 4

[Jan Beich]

Comment on attachment 8909748 [details]
Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD.  f=gaston

Does this unbreak "./mach bootstrap" on OpenBSD?

Comment 5

[Jan Beich]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

https://reviewboard.mozilla.org/r/181246/#review186468

::: python/mozboot/mozboot/freebsd.py:11
(Diff revision 1)
>  
>  class FreeBSDBootstrapper(BaseBootstrapper):
>      def __init__(self, version, flavor, **kwargs):
>          BaseBootstrapper.__init__(self, **kwargs)
>          self.version = int(version.split('.')[0])
>          self.flavor = flavor.lower()

Forgot to drop this line.

Comment 6

[Landry Breuil (:gaston)]

Comment on attachment 8909748 [details]
Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD.  f=gaston

I don't really use mach bootstrap, but probably.

Comment 7

[Jan Beich]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

https://reviewboard.mozilla.org/r/181246/#review186468

> Forgot to drop this line.

Hmm, removing self.flavor but leaving self.version wouldn't make sense as long as the file is also used on DragonFly.

Comment 8

[Jan Beich]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181244/diff/1-2/

Comment 9

[Jan Beich]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181246/diff/1-2/

Comment 10

[Jan Beich]

Comment on attachment 8909748 [details]
Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD.  f=gaston

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181248/diff/1-2/

Comment 11

[Jan Beich]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181244/diff/2-3/

Comment 12

[Jan Beich]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181246/diff/2-3/

Comment 13

[Jan Beich]

Comment on attachment 8909748 [details]
Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD.  f=gaston

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181248/diff/2-3/

Comment 14

[Ralph Giles (:rillian)]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

https://reviewboard.mozilla.org/r/181244/#review186568

Does this apply to released installs as well as new ones?

Comment 15

[Ralph Giles (:rillian)]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

https://reviewboard.mozilla.org/r/181246/#review186572

Note configure still prefers gcc to clang if it's in the PATH. I assume if llvm40 supplies libclang for stylo is also supplies the `clang` front-end, and developers won't normally have gcc on their system?

Comment 16

[Ralph Giles (:rillian)]

Comment on attachment 8909748 [details]
Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD.  f=gaston

https://reviewboard.mozilla.org/r/181248/#review186576

Comment 17

[Jan Beich]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

https://reviewboard.mozilla.org/r/181244/#review186568

Yes. FreeBSD only supports 2 sets of packages per-architecture for each major[1] release: /latest and /quarterly. The latter is rerolled from /latest on each quarter. 2017Q3 still has cargo package but will reach EOL on 2017-10-01. Firefox 56 is scheduled on 2017-09-28, so 2 day window of broken "./mach bootstrap" in some scenarios can be ignored.

[1] Packages are built on the oldest supported minor release e.g., FreeBSD 10.3 and 10.4 or 11.0 and 11.1 currently share packages. Once 11.0 reaches EOL on 2017-10-26, 11.1 packages would be built on 11.1.

Comment 18

[Jan Beich]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

https://reviewboard.mozilla.org/r/181246/#review186572

FreeBSD doesn't have GCC installed by default on i386 (Tier1), amd64 (Tier1), armv6 (Tier2), aarch64 (Tier2). If GCC is later installed from ports/packages as a dependency "gcc" and "g++" symlinks are often excluded. On powerpc* (Tier2), sparc64 (Tier2), mips* base GCC 4.2.1 is too old to be usable for building Firefox, not to mention, big-endian issues and lack of Rust bootstrap.

Building on FreeBSD 10.* requires explicitly setting CC/CXX/CPP via .mozconfig (or environment) as base Clang 3.4.1 is too old. Ideally, it should be advertised somewhere or auto-tried similar to llvm-config*.

Comment 19

[Jan Beich]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

Approval Request Comment
[Feature/Bug causing the regression]: Keeping up with downstream packaging
[User impact if declined]: ./mach bootstrap aborts installing dependencies required to build Firefox
[Is this code covered by automated tests?]: NPOTB
[Has the fix been verified in Nightly?]: Verified on FreeBSD via pristine jail.
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Can only break ./mach bootstrap due to a typo causing syntax error
[String changes made/needed]: None.

Comment 20

[Jan Beich]

Comment on attachment 8909748 [details]
Bug 1401166 - Skip Stylo dependency with ./mach bootstrap on OpenBSD.  f=gaston

Approval Request Comment
[Feature/Bug causing the regression]: bug 1310852 regression
[User impact if declined]: ./mach bootstrap always fails on OpenBSD with: NotImplementedError: mozboot.base does not yet implement ensure_stylo_packages()
[Is this code covered by automated tests?]: NPOTB
[Has the fix been verified in Nightly?]: No but tested via similar change done on FreeBSD.
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Can only break ./mach bootstrap due to a typo causing syntax error.
[String changes made/needed]: None

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4dbee711378f
Drop obsolete cargo dependency from ./mach bootstrap on BSDs. r=rillian
https://hg.mozilla.org/integration/autoland/rev/65a459d1739b
Simplify llvm40 dependency for ./mach bootstrap on FreeBSD. r=rillian
https://hg.mozilla.org/integration/autoland/rev/991e8f1b7a0a
Skip Stylo dependency with ./mach bootstrap on OpenBSD. r=rillian f=gaston

Comment 22

[Jan Beich]

Comment on attachment 8909747 [details]
Bug 1401166 - Simplify llvm40 dependency for ./mach bootstrap on FreeBSD.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1290647 regression
[User impact if declined]: Startup crashes if libxul.so indirectly pulls libstdc++.so after Firefox was built by GCC. Affects i386, amd64, armv6, aarch64.
[Is this code covered by automated tests?]: NPOTB
[Has the fix been verified in Nightly?]: Applied and verified in a pristine jail.
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Can only break ./mach bootstrap due to a typo causing syntax error.
[String changes made/needed]: None. FreeBSD uses generic Unix instructions for building via ./mach bootstrap + ./mach build.

Comment 23

[Ralph Giles (:rillian)]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

https://reviewboard.mozilla.org/r/181244/#review186658

Comment 24

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/4dbee711378f
https://hg.mozilla.org/mozilla-central/rev/65a459d1739b
https://hg.mozilla.org/mozilla-central/rev/991e8f1b7a0a

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

56 is on mozilla-release now.

Comment 26

[Liz Henry (:lizzard)]

Comment on attachment 8909746 [details]
Bug 1401166 - Drop obsolete cargo dependency from ./mach bootstrap on BSDs.

Fix for BSD linux builds only, should be ok for uplift to m-r.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/6c904344b049
https://hg.mozilla.org/releases/mozilla-release/rev/4b0a7dcbe024
https://hg.mozilla.org/releases/mozilla-release/rev/906d69eec53c

Comment 28

[Landry Breuil (:gaston)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #26)

> Fix for BSD linux builds only, should be ok for uplift to m-r.

sigh.. 'BSD linux' isnt a thing, please :)