1577517 - Building Firefox seems to not actually complete properly but returns a success status

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Boris Zbarsky [:bzbarsky]]

This is a recent-ish regression on Mac; I haven't built on Mac in about a month or two until yesterday, so I think sometime in there.

The symptoms are that I run "mach build" and the build kinda finishes: it gets to linking XUL and processing jars, then throws an exception about terminal-notifier, which I don't recall seeing before, even though that code is fairly old. It does look like the code around that exception changed in bug 1563797 back in July, so maybe that's when this started?

I'll attach a log of what that looks like, but the upshot is that the build claims to complete, albeit with a bunch of exception stack spew, and returns a success status for shell purposes (so doing "mach build && echo OK" echoes "OK").

But now if I run the build again, without changing anything at all, instead of being a no-op it rebuilds various Rust stuff. If I do this a few times, eventually the build completes without building any Rust stuff anymore, and then I also don't get the terminal-notifier exception.

Comment 1

[Boris Zbarsky [:bzbarsky]]

Attachment: Log of build that throws the exception

Comment 2

[Boris Zbarsky [:bzbarsky]]

Attachment: Log of build (two more "mach build later") that does not throw the exception

Though maybe it does and this time we manage to JSON serialiaze it?

Though again, once we get to this stage no more Rust stuff gets built and I don't get those stack traces logged, until I change some code and rebuild... and then the whole thing starts happening again for a bit.

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Rust rebuilding unnecessarily may not be related and may be due to an older version of sccache interacting badly with a newer rustc. I'm landing a version check for this so people don't hit it anymore, but upgrading or disabling sccache for now may fix that.

Andrew, does that exception seem like it could be related to recent python 3 upgrade work?

Comment 4

[Boris Zbarsky [:bzbarsky]]

I'm not using sccache, fwiw. I do have a newer rustc (1.37).

Comment 5

[Andrew Halberstadt [:ahal]]

The build still runs with Python 2, so I wouldn't expect many differences there. I'd definitely be more suspicious of bug 1563797. Since this is a bug in an error case, there's also a good chance it was regressed long ago (or was broken to begin with) and no one noticed until now.

Boris, can you confirm whether terminal-notifier is on your $PATH? If not, I bet installing it will fix the issue. Either way we should fix the error handling here.

Comment 6

[Boris Zbarsky [:bzbarsky]]

Boris, can you confirm whether terminal-notifier is on your $PATH?

It's not. I added MOZ_NOSPAM to my environment to fix things for the moment.

Comment 7

[Andrew Halberstadt [:ahal]]

Oh, this is a regression from:
https://hg.mozilla.org/mozilla-central/rev/983a86c020c9

So it was the Python 3 work after all.

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1577517 - [mozbuild] Fix bug in error handling case when 'terminal-notifier' is not installed, r?chmanchester

Depends on D43820

Comment 9

[Andrew Halberstadt [:ahal]]

Kind of odd that the site of the log call in mozbuild.base didn't show up in the stack :/

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/19e1fe1adc3c
[mozbuild] Fix bug in error handling case when 'terminal-notifier' is not installed, r=firefox-build-system-reviewers,mshal

Comment 11

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/19e1fe1adc3c