863844 - Measure amount of coalescing on inbound right now

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[(not currently active) Ted Mielczarek]

Coalescing makes sheriffing harder, but helps with capacity. We should measure how much coalescing happens on inbound right now.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I analyzed all non-merge push changesets to inbound in the range https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=74354f979ea8&tochange=cad82c3b69bc.

For each changeset, I grabbed all the mozilla-inbound build data for the changeset, discarded any build info for builds with "-pgo" or "_pgo" in the builder name, and assumed any missing builds in the changeset x buildtype matrix were missing because of coalescing (this assumption might over-estimate the gains from coalescing). I filled in the missing values with the average time for that buildtype to "de-coalesce" the builds.

The results:
Actual time taken: 228717299, in 128777 builds
w/o coalescing: 373751505 (63.0%), in 215117 builds

There's some buildtypes (they look JS-related, like spidermonkey/dromaeo benchmarks) that I'm including here even though they ran on very few builds. If I exclude those, the 63% goes down to 58%. So I'd ballpark it and say resource usage would increase by 60% over the current value on mozilla-inbound if we removed coalescing.

I also did some analysis and found that if we eliminated all of the backout changesets and all of the changesets that got backed out, we would reduce resource usage by 24% over the same range on mozilla-inbound. If we removed backouts *and* removed coalescing then resource usage goes up by 27% from current levels. I have more details from my analysis that I may or may not end up summarizing in a post to dev-platform.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

> For each changeset, I grabbed all the mozilla-inbound build data for the changeset,

That should be "For each *push* changeset"

Comment 3

[Kartikaya Gupta (email:[e-mail])]

https://groups.google.com/d/msg/mozilla.dev.platform/Dil1gWjrDKo/aJLI7xXwYpYJ

Comment 4

[Phil Ringnalda (:philor)]

Probably doesn't matter since I don't think anyone had an actual actionable need for a real number, but, that 60% is a number between 0 and 100, and the actual percentage is between 0 and 100, and because the 60% ignored the effects of only building Android/b2g/desktop for pushes that only touch mobile/b2g/browser, and the effects of only doing SpiderMonkey shell builds for pushes that touch js/src/, and the effects of including more than 2/7ths weekend days, and the effects of including the travel weekend and half of the work week where b2g moved to Birch, that's about all you can say about the relationship between actual coalescing and 60% - they are both between 0 and 100.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Actually in this case 100% is not an upper bound; removing coalescing could increase resource usage requirements by > 100%. However most the things you pointed out (e.g. weekends, travel, Birch), if accounted for, should all have the effect of increasing that 60% number to something even higher. So we should be able to say with some confidence that removing coalescing would increase resource requirements by _at least_ 60%, which is already too high for me. So for the purposes of my original proposal, I don't think we can go ahead with that without definitively addressing the coalescing issue.