579131 - Keep track of .ctors to fight global initializers

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[(dormant account)]

I posted about this
http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/abf60ad5fd03a708?pli=1

Basic idea is that we need to keep track of .ctors size for libxul.so.

then number is in 
readelf -S libxul.so
in the size column. We need to have a regression/improvement email every time that number changes. This would be done on linux and maybe possibly OSX. I'm not yet sure what the equivalent is on Windows. 

I think this is high-ish priority as this should be pretty easy and it would start catching regressions immediately.

Comment 1

[[e-mail] (:bhearsum)]

As alternative implementation I'd suggest that this be TinderboxPrint'ed and sent to the graph server, like we do with codesighs/leak tests numbers. It can then be monitored the same way those tests are.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Dumpbin may be able to do the same on Windows.

Comment 3

[Chris AtLee [:catlee]]

Attachment: Script to return the global constructor count on linux

Comment 4

[Chris AtLee [:catlee]]

Attachment: Count global constructors on linux

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 460589 [details] [diff] [review]
Count global constructors on linux

This looks like it'll count the ctors, but I don't see them being TinderboxPrint'ed or sent to the graph server. What's the plan for monitoring these?

Comment 6

[Chris AtLee [:catlee]]

(In reply to comment #5)
> Comment on attachment 460589 [details] [diff] [review]
> Count global constructors on linux
> 
> This looks like it'll count the ctors, but I don't see them being
> TinderboxPrint'ed or sent to the graph server. What's the plan for monitoring
> these?

I can TinderboxPrint it pretty easily.  Graph server posting will have to wait until Rail's work on doing that on the slaves is finished, which will make posting this data super easy.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 460588 [details] [diff] [review]
Script to return the global constructor count on linux

Looks fine to me

Comment 8

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> (In reply to comment #5)
> > Comment on attachment 460589 [details] [diff] [review] [details]
> > Count global constructors on linux
> > 
> > This looks like it'll count the ctors, but I don't see them being
> > TinderboxPrint'ed or sent to the graph server. What's the plan for monitoring
> > these?
> 
> I can TinderboxPrint it pretty easily.  Graph server posting will have to wait
> until Rail's work on doing that on the slaves is finished, which will make
> posting this data super easy.

WFM

Comment 9

[(dormant account)]

> I can TinderboxPrint it pretty easily.  Graph server posting will have to wait
> until Rail's work on doing that on the slaves is finished, which will make
> posting this data super easy.

I'm mostly interested in regression/improvement emails, will those be available before graph server?

Comment 10

[Chris AtLee [:catlee]]

(In reply to comment #9)
> > I can TinderboxPrint it pretty easily.  Graph server posting will have to wait
> > until Rail's work on doing that on the slaves is finished, which will make
> > posting this data super easy.
> 
> I'm mostly interested in regression/improvement emails, will those be available
> before graph server?

No

Comment 11

[Chris AtLee [:catlee]]

Attachment: Count global constructors on linux

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 460660 [details] [diff] [review]
Count global constructors on linux

Cool!

Comment 13

[Chris AtLee [:catlee]]

Comment on attachment 460588 [details] [diff] [review]
Script to return the global constructor count on linux

changeset:   709:6c1a3110a600

Comment 14

[Chris AtLee [:catlee]]

Comment on attachment 460660 [details] [diff] [review]
Count global constructors on linux

changeset:   846:8372cb6ea0d9

Comment 15

[(dormant account)]

is there an eta for getting warning emails on this?

Comment 16

[Chris AtLee [:catlee]]

(In reply to comment #15)
> is there an eta for getting warning emails on this?

nope; everybody is tied up with q3 goals, and the dependent bug for this needs to be fixed first.

Comment 17

[Mike Hommey [:glandium]]

Attachment: Enhance the script counting the number of ctors on linux

Here is a modification that will return the exact number of ctors (not a number of bytes of the section, but an actual count). It should also work on arm-based ports (maemo, android), where the ctors are in a .init_array section.

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 489476 [details] [diff] [review]
Enhance the script counting the number of ctors on linux

Please leave it us to land this. It affects infra, so we need to do it in a scheduled reconfig window.

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 489476 [details] [diff] [review]
Enhance the script counting the number of ctors on linux

changeset:   1818:0ee69e37e575

Comment 20

[[e-mail] (:bhearsum)]

Nothing to check in here for now. Removing needs-reconfig flag.

Comment 21

[Chris AtLee [:catlee]]

Attachment: Send constructors count to graph server

Comment 22

[Chris AtLee [:catlee]]

Attachment: New test name on graph server

We need a new name in the tests table so graph server knows where to put stuff.

Comment 23

[Chris AtLee [:catlee]]

Comment on attachment 508914 [details] [diff] [review]
Send constructors count to graph server

http://hg.mozilla.org/build/buildbotcustom/rev/70277cb3fb30

Comment 24

[Armen [:armenzg]]

Merged and deployed in production:
http://hg.mozilla.org/build/buildbotcustom/rev/09e7dbe6c7b1

Comment 25

[Aki Sasaki (not active)]

Comment on attachment 508914 [details] [diff] [review]
Send constructors count to graph server

http://hg.mozilla.org/build/buildbotcustom/rev/93d1b3d2fda9

Backed out on default+production-0.8 branches due to tryserver bustage.  (self.graphBranch needs to be Tryserver, not MozillaTry)

Comment 26

[Chris AtLee [:catlee]]

Sorry :(

This wasn't caught in staging or in preproduction because we the use the same branch name for submission to graph server as we do for submission to tinderbox, and in both staging and preproduction the branch is hardcoded for everything to MozillaTest and Releng-Preproduction.

Not sure what the correct fix here is, there are a few options:
- Disable this for try like we do for leak test data and codesighs

- Specify that try's graph server branch isn't the same as its tinderbox branch...maybe something like graphBranch = config.get('graphs_branch', config['tinderbox_tree']), with corresponding changes to buildbot-configs

Comment 27

[Chris AtLee [:catlee]]

Attachment: Send constructors count to graph server

Added support for optional graph_branch key in config, which will be set for try to Tryserver.

Move addBuildInfoSteps to where we're doing graph server posts. In particular, addLeakTestSteps and addBuildAnalysis steps don't do it all the time now. The  former is an optimization and the latter fixes busted XR builds.

Comment 28

[Chris AtLee [:catlee]]

Attachment: Send try server results to Tryserver, not MozillaTry

Comment 29

[Chris AtLee [:catlee]]

Comment on attachment 510739 [details] [diff] [review]
Send constructors count to graph server

http://hg.mozilla.org/build/buildbotcustom/rev/e277edf1ae78

Comment 30

[Chris AtLee [:catlee]]

Comment on attachment 511480 [details] [diff] [review]
Send try server results to Tryserver, not MozillaTry

http://hg.mozilla.org/build/buildbot-configs/rev/0749b4b1ad41

Comment 31

[Chris AtLee [:catlee]]

Finally done!

Graphs can be seen at, e.g. http://graphs.mozilla.org/#tests=[[81,1,97]]

Do you want historical data inserted? I can import data for nightlies, it looks like this: http://graphs-stage.mozilla.org/graph.html#tests=[[80,6,8]]

Comment 32

[Mike Hommey [:glandium]]

(In reply to comment #31)
> Finally done!
> 
> Graphs can be seen at, e.g. http://graphs.mozilla.org/#tests=[[81,1,97]]

Awesome
 
> Do you want historical data inserted? I can import data for nightlies, it looks
> like this: http://graphs-stage.mozilla.org/graph.html#tests=[[80,6,8]]

The data before the 8th of December look very wrong.

Comment 33

[Chris AtLee [:catlee]]

> > Do you want historical data inserted? I can import data for nightlies, it looks
> > like this: http://graphs-stage.mozilla.org/graph.html#tests=[[80,6,8]]
> 
> The data before the 8th of December look very wrong.

Probably because of https://bugzilla.mozilla.org/attachment.cgi?id=489476 ?

If I take all that data, and divide by 8, subtract 2, will I get the correct results?

Comment 34

[Mike Hommey [:glandium]]

(In reply to comment #33)
> > > Do you want historical data inserted? I can import data for nightlies, it looks
> > > like this: http://graphs-stage.mozilla.org/graph.html#tests=[[80,6,8]]
> > 
> > The data before the 8th of December look very wrong.
> 
> Probably because of https://bugzilla.mozilla.org/attachment.cgi?id=489476 ?
> 
> If I take all that data, and divide by 8, subtract 2, will I get the correct
> results?

That'd be divided by 4 and substract 2, as it seems these numbers are from x86, not x64. And it looks like that would work, except for the value for commit 8e0fce7d5b49, which is really weird. Is it possible to find the corresponding tarball somewhere, even 6 months later ?

Comment 35

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

The spike is correct, see Bug 594650.

Comment 36

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

And in particular https://bugzilla.mozilla.org/show_bug.cgi?id=276431#c189

Comment 37

[Mike Hommey [:glandium]]

Waw, a static initializer in an idl.

Comment 38

[Chris AtLee [:catlee]]

Ok, try http://graphs-stage.mozilla.org/graph.html#tests=[[80,6,8]] again

And yes, all nightly builds are archived.  Look here: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2010/12/

Comment 39

[Mike Hommey [:glandium]]

(In reply to comment #38)
> Ok, try http://graphs-stage.mozilla.org/graph.html#tests=[[80,6,8]] again

awesome

Comment 40

[(dormant account)]

by the way, this rules!

Comment 41

[Justin Wood (:Callek)]

This broke SeaMonkey comm-1.9.1 linux nightlies. [I just deployed this change yesterday]

I landed a followup fix https://hg.mozilla.org/build/buildbotcustom/rev/250058e36789

We hit that only on 1.9.1 since there we don't use libxul, therefore libxul.so could not be found, and we hit that exception.