1496708 - ./mach bootstrap aborts trying to install NodeJS on Tier3

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Jan Beich]

bug 1424921 + bug 1433036 added NodeJS support to all systems with bootstrap, including FreeBSD and OpenBSD but bug 1481693 appears to have factored NodeJS dependency out into its own package set built by Mozilla. Artifacts are not supported on Tier3 platforms and aren't desired due to low trust in pre-built binaries compared to system packages. Not to mention, Mozilla isn't even upstream for NodeJS to trust its binaries.

$ ./mach bootstrap
[...]
Your version of Mercurial (4.7.1) is sufficiently modern.
Your version of Python (2.7.15) is new enough.
Your version of Rust (1.29.1) is new enough.
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

NotImplementedError: mozboot.base does not yet implement ensure_node_packages()

  File "python/mozboot/mozboot/mach_commands.py", line 38, in bootstrap
    bootstrapper.bootstrap()
  File "python/mozboot/mozboot/bootstrap.py", line 440, in bootstrap
    checkout_root)
  File "python/mozboot/mozboot/bootstrap.py", line 337, in maybe_install_private_packages_or_exit
    self.instance.ensure_node_packages(state_dir, checkout_root)
  File "python/mozboot/mozboot/base.py", line 269, in ensure_node_packages
    % __name__)

Comment 1

[Jan Beich]

I'm not even sure why ./mach bootstrap tries to download toolchain artifacts when user explicitly opted out from Artifact Mode i.e.,

$ ./mach bootstrap
Please choose the version of Firefox you want to build:
1. Firefox for Desktop Artifact Mode
2. Firefox for Desktop
3. Firefox for Android Artifact Mode
4. Firefox for Android
Your choice: 2

Comment 2

[(not currently active) Ted Mielczarek]

We should probably just print a warning here that node needs to be manually installed for systems that don't have bootstrap support.

(In reply to Jan Beich from comment #1)
> I'm not even sure why ./mach bootstrap tries to download toolchain artifacts
> when user explicitly opted out from Artifact Mode i.e.,

node is used for front-end Firefox development, so it'll be required to build even in artifact mode.

Comment 3

[Jan Beich]

Attachment: [part 1] use system cbindgen

(Not on Phabricator due to MFA shenanigans i.e., BMO MFA is incompatible with GitHub login).

Comment 4

[Jan Beich]

Attachment: [part 2] use system node

Comment 5

[Jan Beich]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #2)
> We should probably just print a warning here that node needs to be manually
> installed for systems that don't have bootstrap support.

I don't follow. ./mach bootstrap is/was supported on OpenBSD, FreeBSD and (maybe) DragonFly.

Comment 6

[Jan Beich]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f5abb1864d5a6e83ecc5b9d0534a99aa3bb9cb1c

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

Thanks for the quick bug-filing and patches!  

(In reply to Jan Beich from comment #0)
>
> Mozilla isn't even upstream for NodeJS to trust its binaries.

Can you elaborate on what you mean by this, exactly?

Comment 8

[(not currently active) Ted Mielczarek]

(In reply to Jan Beich from comment #5)
> I don't follow. ./mach bootstrap is/was supported on OpenBSD, FreeBSD and
> (maybe) DragonFly.

Sorry, I meant "for systems that don't currently have support for installing node in bootstrap". Printing a warning that you'll need to manually install it seems nicer than erroring.

Comment 9

[Noemi Erli[:noemi_erli]]

Tried to land this but received: 
applying bug1496708.node.diff
patching file python/mozboot/mozboot/freebsd.py
Hunk #2 succeeded at 70 with fuzz 2 (offset 0 lines).
patching file python/mozboot/mozboot/openbsd.py
Hunk #2 FAILED at 48
1 out of 2 hunks FAILED -- saving rejects to file python/mozboot/mozboot/openbsd.py.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working directory
errors during apply, please fix and qrefresh bug1496708.node.diff

Comment 10

[Jan Beich]

(In reply to Noemi Erli[:noemi_erli] from comment #9)
I can't reproduce.

$ hg clone https://hg.mozilla.org/mozilla-unified
$ cd mozilla-unified; hg up -r inbound
$ hg tip
changeset:   491620:63a27bd95ecf
bookmark:    inbound
tag:         tip
parent:      491616:82b600b76a38
user:        aceman <[e-mail]>
date:        Fri Oct 05 14:56:00 2018 +0300
summary:     Bug 1496524 - add CSP to docshell/resorces/content/netError.xhtml. r=ckerschb

$ hg import 'https://bugzilla.mozilla.org/attachment.cgi?id=9014740'
applying https://bugzilla.mozilla.org/attachment.cgi?id=9014740
$ hg import 'https://bugzilla.mozilla.org/attachment.cgi?id=9014741'
applying https://bugzilla.mozilla.org/attachment.cgi?id=9014741

# another to be checked in bug touching mozboot/freebsd.py
$ hg import 'https://bugzilla.mozilla.org/attachment.cgi?id=9014796'
applying https://bugzilla.mozilla.org/attachment.cgi?id=9014796

Comment 11

[Jan Beich]

(In reply to Dan Mosedale (:dmose, :dmosedale) from comment #7)
> (In reply to Jan Beich from comment #0)
> > Mozilla isn't even upstream for NodeJS to trust its binaries.
> Can you elaborate on what you mean by this, exactly?

Sorry for sarcasm. ./mach bootstrap is recommended for simple way to build from source. I don't expect it to download binaries outside of package manager (which may point to a self-hosted repo also built from source). How long the specific version NodeJS version Firefox 64 uses is going to be available as toolchain artifact, anyway?

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/584018b0397d
Install cbindgen as system package. f=gaston r=Build r=froydnj
https://hg.mozilla.org/integration/mozilla-inbound/rev/7b660cc5c5b4
Install node as system package. f=gaston r=Build r=froydnj

Comment 13

[Jan Beich]

Comment on attachment 9014740 [details] [diff] [review]
[part 1] use system cbindgen

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: Bug 1481693

User impact if declined: Broken ./mach bootstrap on FreeBSD and OpenBSD. Bootstrap is recommended as part of the simple way to build from source.

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_build/Linux_and_MacOS_build_preparation

Is this code covered by automated tests?: Unknown

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): ./mach bootstrap isn't used by the automation. If I'm wrong then assume this can only break build in case of syntax errors (e.g., FreeBSDBootstrapper fails to import due to bogus whitespace).

String changes made/needed:

Comment 14

[Jan Beich]

Comment on attachment 9014741 [details] [diff] [review]
[part 2] use system node

See comment 13.

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: 

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: Yes

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): 

String changes made/needed:

Comment 15

[Jan Beich]

The approval request form filled the answers in comment 14 when nothing was selected/entered. Unfortunately, I wasn't offered a choice to edit or cancel.

Comment 16

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/584018b0397d
https://hg.mozilla.org/mozilla-central/rev/7b660cc5c5b4

Comment 17

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9014740 [details] [diff] [review]
[part 1] use system cbindgen

Sorry, this is too late in the Beta cycle (we build the last 63 beta on Thursday before entering RC) to uplift unplanned tier-3 patches as we need to minimize any potential regression for the final release.

Comment 18

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to Jan Beich from comment #11)
>
> How long the specific version NodeJS version Firefox 64 uses is going to be
> available as toolchain artifact, anyway?

The toolchain artifact is generated dynamically (if it's not already cached) by the CI system, by downloading the official binary from NodeJS.org, which has its archives available indefinitely (see http://nodejs.org/dist/).  As far as I know, this means that even if, at some point in the future, none of the CI builds are generating it by default, pushing a Firefox 64 build to try should cause it to be re-cached.  Needinfo to gps to verify that my understanding is correct.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9014740 [details] [diff] [review]
[part 1] use system cbindgen

Per IRC discussion, these patches are NPOTB and don't need approval for uplift. Sorry for the confusion :(

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/8a68e674c265
https://hg.mozilla.org/releases/mozilla-beta/rev/27fc86657940

Comment 21

[Landry Breuil (:gaston)]

Comment on attachment 9014741 [details] [diff] [review]
[part 2] use system node

probably works for me :)

Comment 22

[Gregory Szorc [:gps]]

"fetch" task artifacts expire in like 1000 years or something ridiculous. The current expiration for toolchain artifacts is the same as most other artifacts: 1 year. e.g. https://tools.taskcluster.net/groups/b9fwUkb9TS-EI9YdwLw6_w/tasks/bGAq2m5BR_6gNX1DDhdVDw/details.

The theory is tasks are deterministic over time. So if an artifact disappears, it can be reconstructed whenever. Obviously dependencies on 3rd party services undermine this by introducing unreliability. Hence the reason for "fetch" tasks and their non-expiration.