Closed Bug 854225 Opened 9 years ago Closed 9 years ago

Frequent Linux32 PGO build failures in TextDecoderBinding.o with error: coverage mismatch for function...

Categories

(Firefox Build System :: General, defect)

x86
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla22

People

(Reporter: philor, Assigned: khuey)

References

Details

Attachments

(2 files, 1 obsolete file)

Starting with https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=392c55ce512d we've been getting frequent build failures like

https://tbpl.mozilla.org/php/getParsedLog.php?id=21025397&tree=Mozilla-Inbound

/usr/bin/ccache /tools/gcc-4.5-0moz3/bin/g++ -o TransitionEventBinding.o -c -I../../dist/stl_wrappers -I../../dist/system_wrappers -include /builds/slave/m-in-lx-pgo-000000000000000000/build/config/gcc_hidden.h -DMOZ_GLUE_IN_PROGRAM -DMOZILLA_INTERNAL_API -D_IMPL_NS_COM -DEXPORT_XPT_API -DEXPORT_XPTC_API -D_IMPL_NS_GFX -D_IMPL_NS_WIDGET -DIMPL_XREAPI -DIMPL_NS_NET -DIMPL_THEBES  -DSTATIC_EXPORTABLE_JS_API -DNO_NSPR_10_SUPPORT -DEXCLUDE_SKIA_DEPENDENCIES  -DOS_POSIX=1 -DOS_LINUX=1  -D_IMPL_NS_LAYOUT -I/builds/slave/m-in-lx-pgo-000000000000000000/build/ipc/chromium/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/ipc/glue -I../../ipc/ipdl/_ipdlheaders  -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/base -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/battery -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/encoding -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/file -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/power -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/quota -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/media -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/network/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/settings -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/phonenumberutils -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/contacts -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/permission -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/alarm -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/events -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/storage -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/offline -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/geolocation -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/notification -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/workers -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/time -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xbl/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xul/document/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/events/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/base/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/html/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/html/document/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/media/webaudio -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/svg/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/generic -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/style -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/xul/base/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/xul/tree -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/camera -I/builds/slave/m-in-lx-pgo-000000000000000000/build/js/xpconnect/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/js/xpconnect/wrappers -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/canvas/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/html/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/media/webrtc/signaling/src/peerconnection -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/base -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/battery -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/indexedDB -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xslt/src/base -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xslt/src/xpath -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xml/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xul/content/src  -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/bindings -I. -I../../dist/include  -I/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dist/include/nspr -I/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dist/include/nss      -fPIC  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -g -fprofile-use -fprofile-correction -Wcoverage-mismatch -O3 -fno-omit-frame-pointer  -Werror -Wno-error=uninitialized -Wno-error=deprecated-declarations -Wno-uninitialized   -DMOZILLA_CLIENT -include ../../mozilla-config.h -MD -MF .deps/TransitionEventBinding.o.pp  /builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TransitionEventBinding.cpp
TreeWalkerBinding.cpp
cc1plus: warnings being treated as errors
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptionsWorkers::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom24TextDecodeOptionsWorkers8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is ad3d836a instead of 3fbf548f
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: coverage mismatch ignored due to -Wcoverage-mismatch
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: execution counts estimated
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptionsWorkers::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom25TextDecoderOptionsWorkers8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 62fb11f5 instead of a022d4be
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In static member function 'static bool mozilla::dom::TextDecodeOptions::InitIds(JSContext*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom17TextDecodeOptions7InitIdsEP9JSContext' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 897a641d instead of 58d0e67c
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptions::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom17TextDecodeOptions8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 41bc3204 instead of 93dd036c
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In static member function 'static bool mozilla::dom::TextDecoderOptions::InitIds(JSContext*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom18TextDecoderOptions7InitIdsEP9JSContext' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 774bd159 instead of b2214a40
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptions::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom18TextDecoderOptions8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is b285fd95 instead of 85a06053
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptions::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom18TextDecoderOptions4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 6d20f2d0 instead of 83c80764
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptions::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom17TextDecodeOptions4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 2beac69b instead of f68686e1
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptionsWorkers::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom25TextDecoderOptionsWorkers4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 8fcbf69c instead of 4b486f7a
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptionsWorkers::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom24TextDecodeOptionsWorkers4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is f93610a1 instead of db121b3e
make[7]: *** [TextDecoderBinding.o] Error 1

Two of the first three builds on that push hit it (the third hit our existing intermittent, bug 820796); zero of the first two builds on the push before hit it. Unfortunately, the horse is already out of the barn, that push has been merged to mozilla-central, so both mozilla-central and mozilla-inbound are closed.
Trev backed out the push where it first appeared, https://hg.mozilla.org/integration/mozilla-inbound/rev/392c55ce512d, but we still got the same failure on the backout, not surprisingly since while that was building I also got one on a retrigger on the push before it.

Current retriggers are in

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=939f0488b0f7
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=a5170d1c3bac
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=76adf81398bc
Blocks: 847120
Blocks: 853692
At least temporarily averted by the backout in https://hg.mozilla.org/integration/mozilla-inbound/rev/a3aef14dc53a, but I don't know whether that was somehow actually at fault, or just made something broken show that it was broken.
Severity: blocker → normal
The dom/bindings .cpp files generated by BindingGen.py are removed between the profile-generate and profile-use passes, so if somehow they end up being different because of some race...
> The dom/bindings .cpp files generated by BindingGen.py are removed between the
> profile-generate and profile-use passes

Er, they are?  That's not good!  Why?  Kyle, can we avoid that?

> so if somehow they end up being different because of some race...

That's bad too.  I'll see if I can reproduce this part.
Flags: needinfo?(khuey)
(In reply to Boris Zbarsky (:bz) from comment #4)
> > The dom/bindings .cpp files generated by BindingGen.py are removed between the
> > profile-generate and profile-use passes
> 
> Er, they are?  That's not good!  Why?  Kyle, can we avoid that?

That's ... unexpected.

> > so if somehow they end up being different because of some race...
> 
> That's bad too.  I'll see if I can reproduce this part.

I assume we have a dependency on a dict's order somewhere.
Flags: needinfo?(khuey)
(In reply to Boris Zbarsky (:bz) from comment #4)
> > The dom/bindings .cpp files generated by BindingGen.py are removed between the
> > profile-generate and profile-use passes
> 
> Er, they are?  That's not good!  Why?  Kyle, can we avoid that?

We're essentially doing make clean between the two passes. Bug 659311 tried to remove that need, but it was backed out.
> I assume we have a dependency on a dict's order somewhere.

I do too; I'll hunt it down if so.
So I found one issue in the hasInstance codegen for SVGUnitTypes and SVGZoomAndPan, but it seems odd that this would affect TextDecoder...

We should still try to preserve the .cpp files even if we fix whatever makes them change, since it's too easy for new things like that to sneak in.  :(
This doesn't affect TextDecoderBinding.cpp, though... could it still cause the build fail?
Attachment #729005 - Flags: feedback?(khuey)
Comment on attachment 729005 [details] [diff] [review]
These seem to be the only ordering issues I can find

Review of attachment 729005 [details] [diff] [review]:
-----------------------------------------------------------------

Fixing these won't hurt, but it seems like there should be something else lurking.
Attachment #729005 - Flags: feedback?(khuey) → feedback+
Right.  Is there a principled way of finding such things in python (e.g. forcing it into a mode where it explicitly randomizes all iterations over dicts and sets)?

If not, seems like we really need to not delete these C++ files.  Could we have a target called "pgoclean" or something that normally does the same thing as clean but in this one dir does something different?
Perhaps we can just delete the files from distclean instead of clean?  That would be reasonable, imo.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #12)
> Perhaps we can just delete the files from distclean instead of clean?  That
> would be reasonable, imo.

That was going to be my suggestion.
Assignee: nobody → khuey
Attached patch Patch (obsolete) — Splinter Review
Attachment #729035 - Flags: review?(mh+mozilla)
Comment on attachment 729035 [details] [diff] [review]
Patch

Actually we probably need globalgen_targets and ParserResults.pkl too.
Attachment #729035 - Flags: review?(mh+mozilla)
Attached patch PatchSplinter Review
Attachment #729035 - Attachment is obsolete: true
Attachment #729063 - Flags: review?(mh+mozilla)
Comment on attachment 729063 [details] [diff] [review]
Patch

Review of attachment 729063 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/bindings/Makefile.in
@@ +237,5 @@
> +    $(binding_header_files) \
> +    $(binding_cpp_files) \
> +    $(all_webidl_files) \
> +    $(globalgen_targets) \
> +    ParserResults.pkl

Indentation looks weird.
Attachment #729063 - Flags: review?(mh+mozilla) → review+
Got sick of starring this. Pushed with indentation fixed.
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a7a2e50308f
https://hg.mozilla.org/mozilla-central/rev/2a7a2e50308f
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla22
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.