885727 - make -C $DIR rebuilds files unnecessarily after global make

Status: RESOLVED DUPLICATE of bug 877308

(Firefox Build System :: General, defect)

Description

[Nathan Froyd [:froydnj]]

I've noticed this happening at least twice now:

1. make at toplevel.
2. Touch some test files in $DIR/tests/unit/
3. make -C $DIR so test files get appropriately installed.

In the above, DIR is toolkit/components/telemetry/ and I have seen Telemetry.cpp get rebuilt at step 3 even though I haven't touched anything that will depend on it.  This has also happened to me when touching $DIR/TelemetryPing.js (so it can get preprocessed and installed correctly).

I find this most peculiar.

Comment 1

[Gregory Szorc [:gps]]

Are you touching a Makefile.in or moz.build file? If you change the contents of one of these files, that will invalidate all targets in the Makefile and cause rebuilds.

This may be a dupe of bug 877308.

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to Gregory Szorc [:gps] from comment #1)
> Are you touching a Makefile.in or moz.build file? If you change the contents
> of one of these files, that will invalidate all targets in the Makefile and
> cause rebuilds.
> 
> This may be a dupe of bug 877308.

Any Makefile.in or moz.build, or just the ones in the relevant directory?

Comment 3

[Gregory Szorc [:gps]]

Any Makefile.in or moz.build touch will trigger "reticulating splines." A touch to any Makefile/backend.mk will invalidate all targets in that specific Makefile. Makefile/backend.mk is only touched by "reticulating splines" if file content actually changes *or* if it is the first Makefile being invoked.

e.g. the following will invalidate all targets for toolkit:

1) touch dom/moz.build
2) make -C toolkit/components/telemetry

See rules.mk:632 for an explanation.

I concede this can suck. There is no way to make it perfect.

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Gregory Szorc [:gps] from comment #3)
> Any Makefile.in or moz.build touch will trigger "reticulating splines." A
> touch to any Makefile/backend.mk will invalidate all targets in that
> specific Makefile. Makefile/backend.mk is only touched by "reticulating
> splines" if file content actually changes *or* if it is the first Makefile
> being invoked.
> 
> e.g. the following will invalidate all targets for toolkit:
> 
> 1) touch dom/moz.build
> 2) make -C toolkit/components/telemetry
> 
> See rules.mk:632 for an explanation.
> 
> I concede this can suck. There is no way to make it perfect.

That really sucks ...

Comment 5

[Gregory Szorc [:gps]]

Read the comment. I don't believe it's possible to make it work in all cases. You need to make a choice between excessive invalidation of the local Makefile or not picking up changes impacting the local Makefile. We chose the former for hopefully obvious reasons.

Perhaps there is more Makefile voodoo that can be employed here. If there is, we should add that.

Comment 6

[Nathan Froyd [:froydnj]]

(In reply to Gregory Szorc [:gps] from comment #5)
> Read the comment. I don't believe it's possible to make it work in all
> cases. You need to make a choice between excessive invalidation of the local
> Makefile or not picking up changes impacting the local Makefile. We chose
> the former for hopefully obvious reasons.

The rules.mk comment doesn't really make things any clearer, IMHO.  But the argument seems to go something like this:

1. Editing local build bits requires the global build bits to be reconstructed;
2. Global build bits changes require recompilation of objects located at a distance, because build configuration for those objects depends on aspects of the global system, and those aspects may have changed due to (1).

Is that correct?

Comment 7

[Gregory Szorc [:gps]]

1) is mostly correct. 2) is kinda correct.

It's probably best to read bug 874543 and bug 848530.

This should probably be documented on MDN somewhere. Now, which article...

Comment 8

[Mike Hommey [:glandium]]

I still see no reason why a change in foo/bar/Makefile.in should rebuild all Makefiles and backend.mk in the objdir.

Comment 9

[Gregory Szorc [:gps]]

(In reply to Mike Hommey (high latency until June 25) [:glandium] from comment #8)
> I still see no reason why a change in foo/bar/Makefile.in should rebuild all
> Makefiles and backend.mk in the objdir.

We went through this when we implemented the current behavior. In the old world, every Makefile depended on just Makefile.in. In the new world, any Makefile/backend.mk could potentially depend on any moz.build file. e.g. the XPIDL work in bug 850380 takes the inputs from multiple moz.build files and combines them into a single output .mk file. This is how we accomplish nonrecursive building.

Sure, you could have each Makefile list the exact moz.build files that contributed to it. But if a *new* moz.build file is created that would provide data for that Makefile, a leaf |make| may not pick that up because said moz.build file is not in its dependencies. Only a global all-of-the-moz.build dependency would pick up these changes. So, we chose to be more correct at the expense of being slower. That being said, the goal is to make moz.build traversal and backend generation fast. On my machine, it takes less than 4s. On magnetic drives with the trees not in page cache, it can take much longer. That's what happens when you perform I/O on hundreds of files. I will continue to tell people that if you are serious about building m-c, you should do so on an SSD.

Comment 10

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #9)
> (In reply to Mike Hommey (high latency until June 25) [:glandium] from
> comment #8)
> > I still see no reason why a change in foo/bar/Makefile.in should rebuild all
> > Makefiles and backend.mk in the objdir.
> 
> In the new world, any
> Makefile/backend.mk could potentially depend on any moz.build file. e.g. the
> XPIDL work in bug 850380 takes the inputs from multiple moz.build files and
> combines them into a single output .mk file. This is how we accomplish
> nonrecursive building.

And it's a .mk file, not a Makefile. Makefiles still have a small list of fixed input. Bug 850380-type things don't need to be mixed with the rest of the backend.mk stuff.

Comment 11

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #9)
> On my machine, it takes less than 4s.

That's actually unbearably slow when you're doing changes in a single moz.build/Makefile.in to adjust a few things over and over, especially when you know it could take a few milliseconds, and further more when it *did* take a few milliseconds and you were used to it. One-offs in a single directory are becoming a PITA to do.

Comment 12

[Gregory Szorc [:gps]]

Duping into older bug.