1391106 - support `mach watch` for non-artifact builds

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Myk Melez [:myk] [@mykmelez]]

It'd be helpful to allow `mach watch` for non-artifact builds, so you can gain its benefits when working with such a build (for the subset of changes that `mach watch` supports, i.e. frontend changes that don't require recompilation).

Presumably we'd want to at least warn users that `mach watch` won't handle recompilation when running the command with such a build.  Even better might be to avoid warning users until they make such a change.

At first glance, this isn't as simple as removing the `if not conditions.is_artifact_build(self)` conditional in mach_commands.py, since doing so causes this error when running the command:

IOError: [Errno 2] No such file or directory: u'…/obj-x86_64-apple-darwin15.6.0/faster/unified_install_dist_bin'

Comment 1

[Nick Alexander :nalexander [he/him]]

The real reason this is more complicated is that |mach watch| leverages install manifests that are only used in the FasterMake+RecursiveMake situation.  That is, the stanza at https://searchfox.org/mozilla-central/rev/dc6d85680539cb7e5fe2843d38a30a0581bfefe1/python/mozbuild/mozbuild/backend/fastermake.py#167 is really relevant; the set of install manifests available for use in a non-FasterMake situation isn't enough to support |mach watch|.

Now I think that non-artifact builds are FasterMake+RecursiveMake, so this should all Just Work with both guards removed; but I haven't investigated yet.

Comment 2

[:Gijs (he/him)]

(In reply to Nick Alexander :nalexander from comment #1)
> The real reason this is more complicated is that |mach watch| leverages
> install manifests that are only used in the FasterMake+RecursiveMake
> situation.  That is, the stanza at
> https://searchfox.org/mozilla-central/rev/
> dc6d85680539cb7e5fe2843d38a30a0581bfefe1/python/mozbuild/mozbuild/backend/
> fastermake.py#167 is really relevant; the set of install manifests available
> for use in a non-FasterMake situation isn't enough to support |mach watch|.
> 
> Now I think that non-artifact builds are FasterMake+RecursiveMake, so this
> should all Just Work with both guards removed; but I haven't investigated
> yet.

With the guard removed, I still get:

[watch 2018-06-14T20:15:37.155441Z] Connecting to watchman
[watch 2018-06-14T20:15:37.156886Z] Checking watchman capabilities
[watch 2018-06-14T20:15:37.179969Z] Subscribing to /Users/gkruitbosch/dev/mozilla-unified
[watch 2018-06-14T20:15:37.192915Z] Watching /Users/gkruitbosch/dev/mozilla-unified
Error running mach:

    ['watch']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

IOError: [Errno 2] No such file or directory: u'/Users/gkruitbosch/dev/builds/opt/faster/unified_install_dist_bin'

  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozbuild/mach_commands.py", line 114, in watch
    return daemon.watch()
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozbuild/faster_daemon.py", line 282, in watch
    for change in self.output_changes(verbose=verbose):
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozbuild/faster_daemon.py", line 251, in output_changes
    for change in self.input_changes(verbose=verbose):
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozbuild/faster_daemon.py", line 200, in input_changes
    input_to_outputs = self.file_copier.input_to_outputs_tree()
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozbuild/util.py", line 962, in __get__
    setattr(instance, name, self.func(instance))
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozbuild/faster_daemon.py", line 90, in file_copier
    'faster', 'unified_install_dist_bin'))
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozpack/manifests.py", line 119, in __init__
    with _auto_fileobj(path, fileobj, 'rb') as fh:
  File "/usr/local/Cellar/python@2/2.7.15/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/gkruitbosch/dev/mozilla-unified/python/mozbuild/mozpack/manifests.py", line 32, in _auto_fileobj
    fileobj = open(path, mode)

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1391106 - Support `mach watch` for non-artifact builds. r?#build

I manually compared $TOPOBJDIR/faster/unified_install_dist_bin for
an artifact and a non-artifact build, and found only a few
pre-processing differences. I can't recall why this wasn't supported
initially; I think it used to be that manifests overlapped and that
the interaction with the RecursiveMake backend was more complicated.
But I see no reason why this shouldn't work, and would like to get
some folks to test it in the wild.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/29f712502e86
Support `mach watch` for non-artifact builds. r=firefox-build-system-reviewers,mhentges

Comment 5

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/29f712502e86