854225 - Frequent Linux32 PGO build failures in TextDecoderBinding.o with error: coverage mismatch for function...

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Phil Ringnalda (:philor)]

Starting with https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=392c55ce512d we've been getting frequent build failures like

https://tbpl.mozilla.org/php/getParsedLog.php?id=21025397&tree=Mozilla-Inbound

/usr/bin/ccache /tools/gcc-4.5-0moz3/bin/g++ -o TransitionEventBinding.o -c -I../../dist/stl_wrappers -I../../dist/system_wrappers -include /builds/slave/m-in-lx-pgo-000000000000000000/build/config/gcc_hidden.h -DMOZ_GLUE_IN_PROGRAM -DMOZILLA_INTERNAL_API -D_IMPL_NS_COM -DEXPORT_XPT_API -DEXPORT_XPTC_API -D_IMPL_NS_GFX -D_IMPL_NS_WIDGET -DIMPL_XREAPI -DIMPL_NS_NET -DIMPL_THEBES  -DSTATIC_EXPORTABLE_JS_API -DNO_NSPR_10_SUPPORT -DEXCLUDE_SKIA_DEPENDENCIES  -DOS_POSIX=1 -DOS_LINUX=1  -D_IMPL_NS_LAYOUT -I/builds/slave/m-in-lx-pgo-000000000000000000/build/ipc/chromium/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/ipc/glue -I../../ipc/ipdl/_ipdlheaders  -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/base -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/battery -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/encoding -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/file -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/power -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/quota -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/media -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/network/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/settings -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/phonenumberutils -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/contacts -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/permission -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/alarm -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/events -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/storage -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/offline -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/geolocation -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/src/notification -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/workers -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/time -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xbl/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xul/document/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/events/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/base/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/html/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/html/document/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/media/webaudio -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/svg/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/generic -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/style -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/xul/base/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/layout/xul/tree -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/camera -I/builds/slave/m-in-lx-pgo-000000000000000000/build/js/xpconnect/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/js/xpconnect/wrappers -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/canvas/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/html/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/media/webrtc/signaling/src/peerconnection -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/base -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/battery -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/indexedDB -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xslt/src/base -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xslt/src/xpath -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xml/content/src -I/builds/slave/m-in-lx-pgo-000000000000000000/build/content/xul/content/src  -I/builds/slave/m-in-lx-pgo-000000000000000000/build/dom/bindings -I. -I../../dist/include  -I/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dist/include/nspr -I/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dist/include/nss      -fPIC  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -Wcast-align -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -std=gnu++0x -pthread -pipe  -DNDEBUG -DTRIMMED -g -fprofile-use -fprofile-correction -Wcoverage-mismatch -O3 -fno-omit-frame-pointer  -Werror -Wno-error=uninitialized -Wno-error=deprecated-declarations -Wno-uninitialized   -DMOZILLA_CLIENT -include ../../mozilla-config.h -MD -MF .deps/TransitionEventBinding.o.pp  /builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TransitionEventBinding.cpp
TreeWalkerBinding.cpp
cc1plus: warnings being treated as errors
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptionsWorkers::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom24TextDecodeOptionsWorkers8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is ad3d836a instead of 3fbf548f
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: coverage mismatch ignored due to -Wcoverage-mismatch
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: execution counts estimated
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptionsWorkers::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom25TextDecoderOptionsWorkers8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 62fb11f5 instead of a022d4be
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In static member function 'static bool mozilla::dom::TextDecodeOptions::InitIds(JSContext*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom17TextDecodeOptions7InitIdsEP9JSContext' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 897a641d instead of 58d0e67c
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptions::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom17TextDecodeOptions8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 41bc3204 instead of 93dd036c
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In static member function 'static bool mozilla::dom::TextDecoderOptions::InitIds(JSContext*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom18TextDecoderOptions7InitIdsEP9JSContext' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 774bd159 instead of b2214a40
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptions::ToObject(JSContext*, JSObject*, JS::Value*)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom18TextDecoderOptions8ToObjectEP9JSContextP8JSObjectPN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is b285fd95 instead of 85a06053
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptions::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom18TextDecoderOptions4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 6d20f2d0 instead of 83c80764
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptions::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom17TextDecodeOptions4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 2beac69b instead of f68686e1
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecoderOptionsWorkers::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom25TextDecoderOptionsWorkers4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is 8fcbf69c instead of 4b486f7a
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp: In member function 'bool mozilla::dom::TextDecodeOptionsWorkers::Init(JSContext*, JSObject*, const JS::Value&)':
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: error: coverage mismatch for function '_ZN7mozilla3dom24TextDecodeOptionsWorkers4InitEP9JSContextP8JSObjectRKN2JS5ValueE' while reading counter 'arcs'
/builds/slave/m-in-lx-pgo-000000000000000000/build/obj-firefox/dom/bindings/TextDecoderBinding.cpp:1094:1: note: checksum is f93610a1 instead of db121b3e
make[7]: *** [TextDecoderBinding.o] Error 1

Two of the first three builds on that push hit it (the third hit our existing intermittent, bug 820796); zero of the first two builds on the push before hit it. Unfortunately, the horse is already out of the barn, that push has been merged to mozilla-central, so both mozilla-central and mozilla-inbound are closed.

Comment 1

[Phil Ringnalda (:philor)]

Trev backed out the push where it first appeared, https://hg.mozilla.org/integration/mozilla-inbound/rev/392c55ce512d, but we still got the same failure on the backout, not surprisingly since while that was building I also got one on a retrigger on the push before it.

Current retriggers are in

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=939f0488b0f7
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=a5170d1c3bac
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=76adf81398bc

Comment 2

[Phil Ringnalda (:philor)]

At least temporarily averted by the backout in https://hg.mozilla.org/integration/mozilla-inbound/rev/a3aef14dc53a, but I don't know whether that was somehow actually at fault, or just made something broken show that it was broken.

Comment 3

[Mike Hommey [:glandium]]

The dom/bindings .cpp files generated by BindingGen.py are removed between the profile-generate and profile-use passes, so if somehow they end up being different because of some race...

Comment 4

[Boris Zbarsky [:bzbarsky]]

> The dom/bindings .cpp files generated by BindingGen.py are removed between the
> profile-generate and profile-use passes

Er, they are?  That's not good!  Why?  Kyle, can we avoid that?

> so if somehow they end up being different because of some race...

That's bad too.  I'll see if I can reproduce this part.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Boris Zbarsky (:bz) from comment #4)
> > The dom/bindings .cpp files generated by BindingGen.py are removed between the
> > profile-generate and profile-use passes
> 
> Er, they are?  That's not good!  Why?  Kyle, can we avoid that?

That's ... unexpected.

> > so if somehow they end up being different because of some race...
> 
> That's bad too.  I'll see if I can reproduce this part.

I assume we have a dependency on a dict's order somewhere.

Comment 6

[Mike Hommey [:glandium]]

(In reply to Boris Zbarsky (:bz) from comment #4)
> > The dom/bindings .cpp files generated by BindingGen.py are removed between the
> > profile-generate and profile-use passes
> 
> Er, they are?  That's not good!  Why?  Kyle, can we avoid that?

We're essentially doing make clean between the two passes. Bug 659311 tried to remove that need, but it was backed out.

Comment 7

[Boris Zbarsky [:bzbarsky]]

> I assume we have a dependency on a dict's order somewhere.

I do too; I'll hunt it down if so.

Comment 8

[Boris Zbarsky [:bzbarsky]]

So I found one issue in the hasInstance codegen for SVGUnitTypes and SVGZoomAndPan, but it seems odd that this would affect TextDecoder...

We should still try to preserve the .cpp files even if we fix whatever makes them change, since it's too easy for new things like that to sneak in.  :(

Comment 9

[Boris Zbarsky [:bzbarsky]]

Attachment: These seem to be the only ordering issues I can find

This doesn't affect TextDecoderBinding.cpp, though... could it still cause the build fail?

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Comment on attachment 729005 [details] [diff] [review]
These seem to be the only ordering issues I can find

Review of attachment 729005 [details] [diff] [review]:
-----------------------------------------------------------------

Fixing these won't hurt, but it seems like there should be something else lurking.

Comment 11

[Boris Zbarsky [:bzbarsky]]

Right.  Is there a principled way of finding such things in python (e.g. forcing it into a mode where it explicitly randomizes all iterations over dicts and sets)?

If not, seems like we really need to not delete these C++ files.  Could we have a target called "pgoclean" or something that normally does the same thing as clean but in this one dir does something different?

Comment 12

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Perhaps we can just delete the files from distclean instead of clean?  That would be reasonable, imo.

Comment 13

[Mike Hommey [:glandium]]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #12)
> Perhaps we can just delete the files from distclean instead of clean?  That
> would be reasonable, imo.

That was going to be my suggestion.

Comment 14

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: Patch

Comment 15

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Comment on attachment 729035 [details] [diff] [review]
Patch

Actually we probably need globalgen_targets and ParserResults.pkl too.

Comment 16

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: Patch

Comment 17

[Mike Hommey [:glandium]]

Comment on attachment 729063 [details] [diff] [review]
Patch

Review of attachment 729063 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/bindings/Makefile.in
@@ +237,5 @@
> +    $(binding_header_files) \
> +    $(binding_cpp_files) \
> +    $(all_webidl_files) \
> +    $(globalgen_targets) \
> +    ParserResults.pkl

Indentation looks weird.

Comment 18

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=21101673&tree=Mozilla-Inbound#error0

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Got sick of starring this. Pushed with indentation fixed.
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a7a2e50308f

Comment 20

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

ryanvm++

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/2a7a2e50308f