1581631 - mach watch fails to build Debugger modules

Status: NEW

(Firefox Build System :: General, defect)

Description

[Jason Laster [:jlast]]

STR:

1. run ./mach watch
2. edit a file in devtools/client/debugger/src/

ER: file is built
AR: file is unrecognized. NOTE: the watchman reports the file as unrecognized by prefixing a !

I think the reason for this is that debugger modules are built via CompiledModules

I left some notes in faster_daemon with where I think the intervention is needed:

diff --git a/python/mozbuild/mozbuild/faster_daemon.py b/python/mozbuild/mozbuild/faster_daemon.py
index a70cf63a0f7b..70cf75d9da03 100644
--- a/python/mozbuild/mozbuild/faster_daemon.py
+++ b/python/mozbuild/mozbuild/faster_daemon.py
@@ -197,6 +197,13 @@ class Daemon(object):
                 print_line('watch', 'Watching {}'.format(self.config_environment.topsrcdir))

             input_to_outputs = self.file_copier.input_to_outputs_tree()
+            # includes debugger files, but input/output looks flipped in that the input is in objdir-frontend and the output is `browser/chrome/devtools/modules`
+            # u'/Users/jlaster/src/moz/gecko-dev/objdir-frontend-artifacts/devtools/client/debugger/src/actions/threads.js':
+            # set([u'browser/chrome/devtools/modules/devtools/client/debugger/src/actions/threads.js']),
+
+            # notice how a webconsole file, which is normal has an input which is not in objdir-frontend
+            # /Users/jlaster/src/moz/gecko-dev/devtools/client/webconsole/actions/ui.js':
+            # set([u'browser/chrome/devtools/modules/devtools/client/webconsole/actions/ui.js']),
             for input, outputs in input_to_outputs.items():
                 if not outputs:
                     raise Exception("Refusing to watch input ({}) with no outputs".format(input))
@@ -212,6 +219,8 @@ class Daemon(object):
                     result = FasterBuildChange()

                     for change in changed:
+                        # /Users/jlaster/src/moz/gecko-dev/devtools/client/debugger/src/actions/threads.js
+                        print(change)
                         if change in input_to_outputs:
                             result.input_to_outputs[change] = set(input_to_outputs[change])
                         else:

Comment 1

[Brian Grinstead [:bgrins]]

Nick: Jason and I looked into this but got stuck. Would you be able to check and see if there's a fix you could make, or some pointers towards what would have to change to add watch support to the CompiledModules (where the source files are processed via node through the build system)?

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Brian Grinstead [:bgrins] from comment #1)

Nick: Jason and I looked into this but got stuck. Would you be able to check and see if there's a fix you could make, or some pointers towards what would have to change to add watch support to the CompiledModules (where the source files are processed via node through the build system)?

I took a look and I can certainly do the latter. What is happening is that mach watch piggy-backs on part of the internals of mach build faster to do its work. In particular, it uses the install manifests that the FasterMake build backend produces to know how to handle some build changes. (This is particularly useful on Windows where source files need to be copied, not just symlinked, into the object directory.) However, those install manifests only handle copying files and preprocessing files; they don't handle the general case of GENERATED_FILES. Under the hood, that's what CompiledModules is doing: it declares a GENERATED_FILES, and then the build system stitches the "copy this from that" using the install manifests. This is why the inputs-to-outputs map looks odd -- it's really:

[source file in GENERATED_FILES declaration->] input `$topobjdir/generated` -> output `dist/bin/...`

but the part in [] doesn't show up in this graph. mach build faster knows how to ensure that the GENERATED_FILES piece happens before the install manifests is processed, so everything works. mach watch does not, causing this problem.

It would be plausible to move the GENERATED_FILES stanza into the install manifest parsing, which would solve this issue. Perhaps 12 or 18 months ago that might have made sense to me, and potentially even to others; but right now, it's not clear to me what the future of mach build faster really is, and whether the install manifest approach is here to stay. Certainly the energy to move the build system to non-Make expressions is almost completely gone.

Given that, I think the best approach would be to detect if any of the input changes listed are themselves in the object directory and to have mach watch run mach build faster in that situation. It's likely that we'd need to "damp down" the file watching to not have a cascade loop of mach watch -> mach build faster -> mach watch ....

Sorry that this is awkward.

Comment 3

[Jason Laster [:jlast]]

Attachment: Bug 1581631 - mach watch fails to build Debugger modules. r=bgrins