1358042 - 2.06% bloom_basic http: (osx-10-10) regression on push 1b7da41c6b3a4f104ef5c5a5985aa2db825b7460 (Wed Apr 19 2017)

Status: RESOLVED WONTFIX

(Firefox :: Untriaged, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push 1b7da41c6b3a4f104ef5c5a5985aa2db825b7460. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  2%  bloom_basic http: osx-10-10 opt e10s     711.11 -> 725.77


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=6103

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

:Ehsan Akhgari, :chutten Could you please confirm bug 1350765 is related to this issue? Until previous data is backfilled, I cannot certainly assume this.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

thanks Ionut for filing this bug and collecting more data.  Keep in mind this is a new test- while it looks fairly stable, we are not sure what affects it fully, as a new test there have been a lot of changes in the values.

Comment 4

[Chris H-C :chutten]

Isn't this more likely to be https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=ad85c3fedd6750b955af6ef6219fa411391ae8f1&tochange=278c3bf8f7f4a1a8260e1563f18bf93ccb097d24 which is the range when autoland first started showing the regression? (looked for the first purple dots at https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bautoland,1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&series=%5Bmozilla-inbound,1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&zoom=1492562138780.3806,1492572556982.4304,657.4803149606299,799.2125984251968)

To me it would make more sense as that range has a mac-specific compiler change (Bug 1356927)

Whereas bug 1350765 is a cross-platform telemetry change which should actually be a mild improvement, for tests that accumulate telemetry.

Comment 5

[Ionuț Goldan [:igoldan]]

(In reply to Chris H-C :chutten from comment #4)
Thank you for you quick notice.

> Isn't this more likely to be
> https://hg.mozilla.org/integration/autoland/
> pushloghtml?fromchange=ad85c3fedd6750b955af6ef6219fa411391ae8f1&tochange=278c
> 3bf8f7f4a1a8260e1563f18bf93ccb097d24 which is the range when autoland first
> started showing the regression? (looked for the first purple dots at
> https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bautoland,
> 1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&series=%5Bmozilla-inbound,
> 1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&zoom=1492562138780.3806,
> 1492572556982.4304,657.4803149606299,799.2125984251968)
> 
> To me it would make more sense as that range has a mac-specific compiler
> change (Bug 1356927)
> 
> Whereas bug 1350765 is a cross-platform telemetry change which should
> actually be a mild improvement, for tests that accumulate telemetry.

That is a different bigger regression, which we are aware of. We treat that separately, on bug 1357703.

This filed bug is precisely about the lesser regression, that later followed it: https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&series=%5Bautoland,1d83527507421790c3e598b6fde76955bce2467d,1,1%5D&zoom=1492610415476.1904,1492628202142.8572,654.6468401486989,824.1635687732341

Comment 6

[Chris H-C :chutten]

Oh, I see. Sorry for the confusion.

Is this a platform-specific regression, or is the test only run on osx?

Re-ni?ing :ehsan as I he might have a better idea of how a perf-improvement patch might have resulted in a regression.

Comment 7

[Ionuț Goldan [:igoldan]]

It looks like its OSX specific.

Comment 8

[(no longer active)]

Hmm, it's hard to say really.  Should we back out bug 1350765 temporarily to see if it fixes the regression?  I'm pretty sure it got backed out the last time I tried to land it anyway...

If this bug turns out to be the culprit, it would be interesting to also investigate whether telemetry should be enabled for Talos tests.  :-)

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Ionut, can you test this on try if the backout would remove the regression?

Comment 10

[Ionuț Goldan [:igoldan]]

Yes, will do.

Comment 11

[(no longer active)]

Thanks, maintaining the needinfo for now then until you tell us whether the backout helped or not.  :-)

Comment 12

[Ionuț Goldan [:igoldan]]

The try results have arrived.

The baseline job results are listed here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9404517c7e49425ab835c43fa44f67f478728ec4

The backout job results of 1b7da41c6b3a are listed here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2b23f8d214fbbddc21569ee297cb27de86174db

The Comparison view between the two can be analyzed here: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=9404517c7e49&newProject=try&newRevision=b2b23f8d214fbbddc21569ee297cb27de86174db&framework=1&showOnlyImportant=0

Looks like the backout of 1b7da41c6b3a doesn't remove the regression, but actually worsens it by more than 27%.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I am concerned here that the baseline is from a push that is 2 days prior to the code base for the backout push, can we get two pushes that has the same base revision?

i.e:
./mach try
<backout>
./mach try

Comment 14

[Ionuț Goldan [:igoldan]]

Yep, I ran another pair of retriggers after seeing those strange results.

Comment 15

[Ionuț Goldan [:igoldan]]

The new results are here.

Baseline job results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=42f2038d4ec5570f2364bda47acad1cbd8386fd2&selectedJob=93790461

Backout job results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=45aeceb2edc8be521b744d077749e398f49f3d26&selectedJob=93836804

The Comparison view: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=42f2038d4ec5&newProject=try&newRevision=45aeceb2edc8&framework=1&showOnlyImportant=0

With the new and correct results, it seems that a backout of 1b7da41c6b3a wouldn't do any help on this issue.

Comment 16

[Benjamin Smedberg]

Ionut, is it correct that you own next steps to identify the actual change that caused this regression?

Comment 17

[Ionuț Goldan [:igoldan]]

That is correct. If you have more questions regarding the bloom test, :bholley and :jmaher can explain more about it.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

it seems that we get many alerts for osx and the "root cause" seems to be unrelated to anything that would cause problems.  I have opened bug 1359361 to track this.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

we have adjusted the bloom test to be 5% threshold and realized that the ideal implementation of it (comparing baseline to the ref data) was not done originally.  With that said we should not spend more time on this specific alert.  Apologies for any randomization.