874923 - The build system doesn't know what it should do when you touch a webidl file

Status: RESOLVED DUPLICATE of bug 928195

(Firefox Build System :: General, defect)

Description

[(no longer active)]

See this patch <https://hg.mozilla.org/integration/mozilla-inbound/rev/2718954757e1> which burned on Windows: <https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=2718954757e1>.  This can only happen if AudioNodeBinding.cpp is not properly regenerated when AudioNode.webidl is touched.

Comment 1

[Gregory Szorc [:gps]]

Likely regression from bug 861587. Chances are high this is a pymake bug.

Comment 2

[Boris Zbarsky [:bzbarsky]]

I haven't managed to reproduce so far, even with pymake (on Mac, though).

Comment 3

[Mike Hommey [:glandium]]

I wonder if it's not a stupid / vs. \ thing. Like dependencies using / and targets using \, or the other way around.

Comment 4

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #3)
> I wonder if it's not a stupid / vs. \ thing. Like dependencies using / and
> targets using \, or the other way around.

(or some other kind of path differences)

Comment 5

[Boris Zbarsky [:bzbarsky]]

Attachment: Patch for reproducing the bug

OK, so slow progress.

I have managed to reproduce this on Windows, building with pymake, using the following steps:

1)  Pull https://hg.mozilla.org/integration/mozilla-inbound/rev/18f85a091e65
2)  Either clobber or at least rm -r $objdir/dom/bindings
3)  Build: build/pymake/make.py -f client.mk
4)  Apply this patch.
5)  Build: build/pymake/make.py -f client.mk

This fails to compile because HTMLInputElementBinding.cpp never gets regenerated.

Step 5 is very important: if you just "make.py -C $objdir/dom/bindings" or "make.py -C $objdir/dom/bindings export" it will regenerate the .cpp file properly (!).

When this happens, I have observed the following things:

* $objdir/dom/bindings/.changed-dependency-list contains only
  "ParserResults.pkl".  That's bad; it should be
  "ParserResults.pkl HTMLInputElementBinding".

* This happens because the timestamp on HTMLInputElementBinding never got
  updated, even though it's older than HTMLInputElement.webidl

* $objdir/dom/bindings/.deps/HTMLInputElementBinding.pp contains this line:

HTMLInputElementBinding: c:\Users\bzbarsky\mozilla\vanilla\obj-firefox\dom\bindings\HTMLInputElement.webidl

Comment 6

[Boris Zbarsky [:bzbarsky]]

Hmm, bugzilla lost the rest of my comment:

which on the face of it seems reasonable and really should make sure that this rule:

 $(binding_dependency_trackers):
 	@$(TOUCH) $@

should get triggered and we should never have a HTMLInputElementBinding file that's older than HTMLInputElement.webidl.  However that's exactly the situation we end up with.

I added some $(info) calls to dom/bindings/Makefile.in and it looks like it is in fact, during a full build, invoked with $(MAKECMDGOALS) set to "export", and that during this invocation ".deps/HTMLInputElementBinding.pp" is in $(MDDEPEND_FILES).

So what I don't understand is why this is not working.  ;)

Comment 7

[Boris Zbarsky [:bzbarsky]]

Oh, and I just did a build in which I added some echo calls in the .BindingGen rule to echo what $(MAKECMDGOALS) and $(MDDEPEND_FILES) are at that point.  They are as expected: "export" and the correct list of files, including ".deps/HTMLInputElementBinding.pp".

I've filed bug 875620 on a workaround for this problem which will cause a bit of extra work to be done when this bug is hit, but should ensure that we rebuild the right stuff.  The steps in comment 5 should continue to work to reproduce this bug, of course.

Comment 8

[Boris Zbarsky [:bzbarsky]]

Oh, and this point I have no idea where to debug this stuff in pymake, and would love it if this bug gained an owner who might.  ;)

Comment 9

[Boris Zbarsky [:bzbarsky]]

Note that it's not like the attached patch necessarily _builds_.  But it should build in dom/bindings, and more importantly get_checked in HTMLInputElementBinding.cpp should get updated.

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I finally managed to reproduce this.  Going to try to fix it.

Comment 11

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

So in the failure mode pymake decides it doesn't need to rebuild HTMLInputElementBinding before it rebuilds HTMLInputElement.webidl (by copying it to the objdir).  That's really busted ...

Comment 12

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #11)
> So in the failure mode pymake decides it doesn't need to rebuild
> HTMLInputElementBinding before it rebuilds HTMLInputElement.webidl (by
> copying it to the objdir).  That's really busted ...

So even when the build succeeds, it makes the decision about HTMLInputElementBinding before rebuilding HTMLInputElement.webidl.  This also means that every second build succeeds, which explains some of the trouble we've had reproducing this.

Comment 13

[Boris Zbarsky [:bzbarsky]]

Is it possible that it does not realize that the thing in $(webidl_files) is the same as the thing in the .deps/HTMLInputElementBinding.pp file, since the former is a relative path and the latter is an absolute path?

Comment 14

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Boris Zbarsky (:bz) from comment #13)
> Is it possible that it does not realize that the thing in $(webidl_files) is
> the same as the thing in the .deps/HTMLInputElementBinding.pp file, since
> the former is a relative path and the latter is an absolute path?

Yes, I think that's part of what's going on here.

Here is everywhere that "HTMLInputElement.webidl" appears in the pymake log.

Line 929804: INFO:pymake.data: Considering target 'HTMLInputElement.webidl'
Line 930712: INFO:pymake.data: Considering target 'c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\HTMLInputElement.webidl'
Line 930713: INFO:pymake.data: Searching for implicit rule to make 'c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\HTMLInputElement.webidl'
Line 930714: INFO:pymake.data:  Terminal rule at c:\dev\mozilla-inbound\config\rules.mk:1187:0 doesn't match: prerequisite 'c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\HTMLInputElement.webidl.pl' not mentioned and doesn't exist.
Line 930715: INFO:pymake.data:  Terminal rule at c:\dev\mozilla-inbound\config\rules.mk:1192:0 doesn't match: prerequisite 'c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\HTMLInputElement.webidl.sh' not mentioned and doesn't exist.
Line 930716: INFO:pymake.data: Couldn't find implicit rule to remake 'c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\HTMLInputElement.webidl'
Line 934554: INFO:pymake.data:Not remaking HTMLInputElementBinding using rule at c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:151:0 because c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\HTMLInputElement.webidl is older.
Line 938442: INFO:pymake.data: Considering target 'c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl'
Line 938443: INFO:pymake.data: Searching for implicit rule to make 'c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl'
Line 938444: INFO:pymake.data:  Terminal rule at c:\dev\mozilla-inbound\config\rules.mk:1187:0 doesn't match: prerequisite 'c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl.pl' not mentioned and doesn't exist.
Line 938445: INFO:pymake.data:  Terminal rule at c:\dev\mozilla-inbound\config\rules.mk:1192:0 doesn't match: prerequisite 'c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl.sh' not mentioned and doesn't exist.
Line 938446: INFO:pymake.data: Couldn't find implicit rule to remake 'c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl'
Line 940054: INFO:pymake.data:Remaking HTMLInputElement.webidl using rule at c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:140:0 because c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl is newer.
Line 940054: INFO:pymake.data:Remaking HTMLInputElement.webidl using rule at c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:140:0 because c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl is newer.
Line 940856: c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:141:0$ c:/dev/mozilla-inbound/obj-i686-pc-mingw32/_virtualenv/Scripts/python.exe c:/dev/mozilla-inbound/config/nsinstall.py -t -m 644 c:/dev/mozilla-inbound/dom/webidl/HTMLInputElement.webidl .
Line 943108: INFO:pymake.data:Remaking ParserResults.pkl using rule at c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:183:0 because HTMLInputElement.webidl is newer.
Line 943121:     AnalyserNode.webidl AnimationEvent.webidl ArchiveReader.webidl ArchiveRequest.webidl Attr.webidl AudioBuffer.webidl AudioBufferSourceNode.webidl AudioContext.webidl AudioDestinationNode.webidl AudioListener.webidl AudioNode.webidl AudioParam.webidl AudioStreamTrack.webidl AudioProcessingEvent.webidl BatteryManager.webidl BeforeUnloadEvent.webidl BiquadFilterNode.webidl Blob.webidl CanvasRenderingContext2D.webidl CaretPosition.webidl CDATASection.webidl CFStateChangeEvent.webidl ChannelMergerNode.webidl ChannelSplitterNode.webidl CharacterData.webidl ChildNode.webidl ClientRect.webidl ClientRectList.webidl ClipboardEvent.webidl CameraManager.webidl CommandEvent.webidl Comment.webidl CompositionEvent.webidl Coordinates.webidl CSS.webidl CSSPrimitiveValue.webidl CSSStyleDeclaration.webidl CSSStyleSheet.webidl CSSValue.webidl CSSValueList.webidl DataContainerEvent.webidl DelayNode.webidl DesktopNotification.webidl DeviceMotionEvent.webidl DeviceStorage.webidl Document.webidl DocumentFragme...
Line 943144:     AnalyserNode.webidl AnimationEvent.webidl ArchiveReader.webidl ArchiveRequest.webidl Attr.webidl AudioBuffer.webidl AudioBufferSourceNode.webidl AudioContext.webidl AudioDestinationNode.webidl AudioListener.webidl AudioNode.webidl AudioParam.webidl AudioStreamTrack.webidl AudioProcessingEvent.webidl BatteryManager.webidl BeforeUnloadEvent.webidl BiquadFilterNode.webidl Blob.webidl CanvasRenderingContext2D.webidl CaretPosition.webidl CDATASection.webidl CFStateChangeEvent.webidl ChannelMergerNode.webidl ChannelSplitterNode.webidl CharacterData.webidl ChildNode.webidl ClientRect.webidl ClientRectList.webidl ClipboardEvent.webidl CameraManager.webidl CommandEvent.webidl Comment.webidl CompositionEvent.webidl Coordinates.webidl CSS.webidl CSSPrimitiveValue.webidl CSSStyleDeclaration.webidl CSSStyleSheet.webidl CSSValue.webidl CSSValueList.webidl DataContainerEvent.webidl DelayNode.webidl DesktopNotification.webidl DeviceMotionEvent.webidl DeviceStorage.webidl Document.webidl DocumentFragme...
Line 946845: DEBUG:pymake.process:c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:198:0: using shell: command contains shell-special character '>': 'echo AnalyserNode.webidl AnimationEvent.webidl ArchiveReader.webidl ArchiveRequest.webidl Attr.webidl AudioBuffer.webidl AudioBufferSourceNode.webidl AudioContext.webidl AudioDestinationNode.webidl AudioListener.webidl AudioNode.webidl AudioParam.webidl AudioStreamTrack.webidl AudioProcessingEvent.webidl BatteryManager.webidl BeforeUnloadEvent.webidl BiquadFilterNode.webidl Blob.webidl CanvasRenderingContext2D.webidl CaretPosition.webidl CDATASection.webidl CFStateChangeEvent.webidl ChannelMergerNode.webidl ChannelSplitterNode.webidl CharacterData.webidl ChildNode.webidl ClientRect.webidl ClientRectList.webidl ClipboardEvent.webidl CameraManager.webidl CommandEvent.webidl Comment.webidl CompositionEvent.webidl Coordinates.webidl CSS.webidl CSSPrimitiveValue.webidl CSSStyleDeclaration.webidl CSSStyleSheet.webidl CSSValue.webidl CSSValueList...
Line 946846: c:\dev\mozilla-inbound\obj-i686-pc-mingw32\dom\bindings\Makefile:198:0$ echo AnalyserNode.webidl AnimationEvent.webidl ArchiveReader.webidl ArchiveRequest.webidl Attr.webidl AudioBuffer.webidl AudioBufferSourceNode.webidl AudioContext.webidl AudioDestinationNode.webidl AudioListener.webidl AudioNode.webidl AudioParam.webidl AudioStreamTrack.webidl AudioProcessingEvent.webidl BatteryManager.webidl BeforeUnloadEvent.webidl BiquadFilterNode.webidl Blob.webidl CanvasRenderingContext2D.webidl CaretPosition.webidl CDATASection.webidl CFStateChangeEvent.webidl ChannelMergerNode.webidl ChannelSplitterNode.webidl CharacterData.webidl ChildNode.webidl ClientRect.webidl ClientRectList.webidl ClipboardEvent.webidl CameraManager.webidl CommandEvent.webidl Comment.webidl CompositionEvent.webidl Coordinates.webidl CSS.webidl CSSPrimitiveValue.webidl CSSStyleDeclaration.webidl CSSStyleSheet.webidl CSSValue.webidl CSSValueList.webidl DataContainerEvent.webidl DelayNode.webidl DesktopNotification.webidl Dev...

Lines 2-5 suggest that pymake thinks "HTMLInputElement.webidl" and "objdir/dom/bindings/HTMLInputElement.webidl" are not the same thing.

Comment 15

[Gregory Szorc [:gps]]

Is anyone still actively looking at this? Presumably this will require excessive clobbers. I'm still catching up on bug mail. Has this been fixed elsewhere?

Comment 16

[Boris Zbarsky [:bzbarsky]]

See comment 7 for the clobbers end of things.

Past that, I am not looking into it; I'm hoping someone with more pymake knowledge than I have will do it.

Comment 17

[Boris Zbarsky [:bzbarsky]]

The patch in bug 838146 might fix it, since it adds an explicit dependency from $(CURDIR)/foo.webidl to foo.webidl.

Comment 18

[Boris Zbarsky [:bzbarsky]]

Is this reproducible now if the patch we checked in for bug 875620 is backed out?

Comment 19

[Boris Zbarsky [:bzbarsky]]

Ehsan, see comment 18?

Comment 20

[Gregory Szorc [:gps]]

And is this bug any different from bug 890744?

Comment 21

[Boris Zbarsky [:bzbarsky]]

Bug 890744 is about the fact that the exported PrototypeList.h does not get updated properly when a brand new WebIDL binding is added.

This bug was about the fact that when an existing binding is modified its .cpp is not regenerated correctly.  This bug is not observable on trunk today because we checked in a workaround for it in bug 875620, but that workaround means more rebuilding on Windows.  So if we can remove the workaround, that would be good.

Comment 22

[Boris Zbarsky [:bzbarsky]]

Also, in case anyone other than Ehsan decides to check whether this is still a problem, here's what I sent him last night on reproducing this bug:

I think the basic steps to reproduce are (these are a modified version of the steps in comment 5):

1) Locally back out bug 875620.
2) Clobber.
3) Build from toplevel.
4) Apply attachment 753601 [details] [diff] [review].
5) Build from toplevel.

If your build succeeds, I think we can declare the bug fixed.

If we want to be super-sure that what I think fixed it actually fixed it, we could actually follow the instructions in comment 5 to the letter (including updating to the rev listed there), verify that the bug appear swhen doing steps 2-5 above, then go back to a clean rev 18f85a091e65, add these two lines to dom/bindings/Makefile.in:

all_webidl_files_absolute = $(addprefix $(CURDIR)/,$(all_webidl_files))
$(all_webidl_files_absolute): $(CURDIR)/%: %

and then follow steps 2 through 5 and make sure that the build succeeds this time.

No matter what, a lot of building on Windows.

Comment 23

[(no longer active)]

I did follow the steps in comment 22.  Without the patch to dom/bindings/Makefile.in, I can reproduce the build bustage.  With those changes applied though, I can still reproduce, so I think this bug is not fixed.  The compilation error in the second build is also about 'Checked' not being a member of mozilla::dom::HTMLInputElement.

Comment 24

[Boris Zbarsky [:bzbarsky]]

Ehsan, thank you for checking that!  Sounds like this bug is indeed still present.  :(

Comment 25

[Gregory Szorc [:gps]]

I'm going to call this one a dupe unless someone proves otherwise.