1421276 - 2.37% tsvgr_opacity (osx-10-10) regression on push fbf50b919f3c19840335d119d1e5acaddaeda0ae (Tue Nov 28 2017)

Status: RESOLVED INVALID

(Core :: JavaScript: Internationalization API, defect)

Description

[Ionuț Goldan [:igoldan]]

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=fbf50b919f3c19840335d119d1e5acaddaeda0ae

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  2%  tsvgr_opacity osx-10-10 opt e10s     330.86 -> 338.72


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=10762

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Buildbot/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Buildbot/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Buildbot/Talos/RegressionBugsHandling

Comment 1

[Ionuț Goldan [:igoldan]]

:anba Please consider fixing this performance regression.

Comment 2

[André Bargull [:anba]]

When looking at the subtests [1] of tsvgr_opacity, the regression is linked to "tsvgr_opacity big-optimizable-group-opacity-2500.svg" and when we then look at the graph for that subtest [2], I don't see a clear indication that a regression is present:
- around Nov 15-17 most data points are around 600
- around Nov 17-18 the data points are at ~580
- then there's the weekend with too few data points
- around Nov 21-23 it's back to 580-600
- around Nov 23-27 none or too few data points are present
- and after that it's back to 580-600

So when comparing all data points in the last 14 days, I don't see a clear sign that bug 1319843 regressed this benchmark. 

[1] https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=mozilla-inbound&originalRevision=dd961743507f11147da0a782cf539a20c74ed9de&newProject=mozilla-inbound&newRevision=fbf50b919f3c19840335d119d1e5acaddaeda0ae&originalSignature=e36d4e6025ecbe2c591d4347294065bc90c453cf&newSignature=e36d4e6025ecbe2c591d4347294065bc90c453cf&framework=1
[2] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-inbound,1376765,1,1&highlightAlerts=0

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to André Bargull [:anba] from comment #2)
> When looking at the subtests [1] of tsvgr_opacity, the regression is linked
> to "tsvgr_opacity big-optimizable-group-opacity-2500.svg" and when we then
> look at the graph for that subtest [2], I don't see a clear indication that
> a regression is present:
> - around Nov 15-17 most data points are around 600
> - around Nov 17-18 the data points are at ~580
> - then there's the weekend with too few data points
> - around Nov 21-23 it's back to 580-600
> - around Nov 23-27 none or too few data points are present
> - and after that it's back to 580-600
> 
> So when comparing all data points in the last 14 days, I don't see a clear
> sign that bug 1319843 regressed this benchmark.

You are right. This looks more like noise and may actually resemble what we call a weekend case:
https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/Noise_FAQ#Why_do_we_need_12_future_data_pointsoriginal#Weekends
Marking this bug as invalid.

Comment 4

[André Bargull [:anba]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #3)
> You are right. This looks more like noise and may actually resemble what we
> call a weekend case:
> https://wiki.mozilla.org/Buildbot/Talos/Sheriffing/
> Noise_FAQ#Why_do_we_need_12_future_data_pointsoriginal#Weekends

Whew, lucky me. :-D

> Marking this bug as invalid.

Thanks!