1453329 - 3.42 - 7.61% Strings PerfHasRTLCharsDE / Strings PerfHasRTLCharsExample3 / Strings PerfHasRTLCharsRU / Strings PerfIsUTF8Example3 (macosx64-nightly) regression on push d653ab9ff7622839a4fc7cc38dbfef7f2bc8098f (Fri Apr 6 2018)

Status: RESOLVED INVALID

(Firefox :: Untriaged, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a platform microbenchmarks regression from push:

https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?changeset=d653ab9ff7622839a4fc7cc38dbfef7f2bc8098f

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  8%  Strings PerfIsUTF8Example3 macosx64-nightly opt      7,695.17 -> 8,281.08
  6%  Strings PerfHasRTLCharsRU macosx64-nightly opt       931,136.42 -> 988,954.25
  4%  Strings PerfHasRTLCharsDE macosx64-nightly opt       948,214.83 -> 988,512.83
  3%  Strings PerfHasRTLCharsExample3 macosx64-nightly opt 17,079.21 -> 17,662.92


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=12623

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Platform_Microbenchmarks

Comment 1

[Ionuț Goldan [:igoldan]]

One of these bugs caused OS X perf regressions on Beta:

bug 1448132
bug 1450773
bug 1443958
bug 1448705
bug 1436482
bug 1451337
bug 1451502
bug 1450795
bug 1441348

I ask the owners to look over the nature of this issue and state whether their bugs are responsible or not, until I finish the Try bisection.

Comment 2

[:Gijs (he/him)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)
> bug 1451502

https://hg.mozilla.org/releases/mozilla-beta/rev/4b1c94bc56dd

This is a devtools-only minor change, and for sure it's not this.

Comment 3

[Benjamin Bouvier [:bbouvier] (inactive)]

Highly unlikely to be bug 1450795, which was a correctness fix about WebAssembly calls.

Comment 4

[Tooru Fujisawa [:arai]]

bug 1448132 patch touches OS X specific code (around window handling), but I'm not sure how it can affect gtest.

Comment 5

[Julian Descottes [:jdescottes]]

bug 1451337 is a devtools/debugger change, it can't affect this test.

Comment 6

[Olli Pettay [:smaug][[e-mail]]]

highly unlikely bug 1443958, which is about <input type=date>

Comment 7

[Kris Maglione [:kmag]]

Bug 1436482 is highly unlikely. It's a pretty trivial change to how we decide which extension content scripts to run. It will never be called in a gtest.

Comment 8

[David Parks [:handyman]]

Its not bug 1450773.  That change is Windows only.

Comment 9

[Daniel Holbert [:dholbert]]

It's not bug 1441348; that bug's change was not part of the build. (It was just a tweak to a reftest manifest (to adjust the fuzziness threshold on Windows 10).

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

Almost certainly not bug 1448705; that's in the audio resampler code

Comment 11

[Ionuț Goldan [:igoldan]]

Try bisection results are now here.

bug 1441348: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=5a46f6926427&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1450795: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=65c446050739&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1451502: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=4d4b6d898930&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1451337: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=b4eafb5754d2&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1436482: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=812e30b331cc&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1448705: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=7f455e8dfaa1&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1443958: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=80addee0f7aa&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1450773: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=b1bd1210184f&framework=6&filter=strings%20perfhasrtlchars%20osx

bug 1448132: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=0f6950d69542&newProject=try&newRevision=5fc618886c7b&framework=6&filter=strings%20perfhasrtlchars%20osx

Comment 12

[Julian Descottes [:jdescottes]]

Mentioned it on IRC, but I think the tests mentioned here have been unstable on OSX beta for a while

https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=mozilla-beta,1665372,1,6&highlightedRevisions=0f6950d69542&highlightedRevisions=65c446050739&zoom=1521199040704.3896,1523473627390.9465,756716.144618703,1114925.0998425838

It's been oscillating between 950 and 1000 ms since March (which is the regression detected here)

In this case the difference must have been slightly bigger and triggered the alert, but that would explain why none of the bugs in the pushlog seems connected to this.

Comment 13

[Julian Descottes [:jdescottes]]

Ignore my comment: it seems I changed the platform to Linux while trying to select the correct test. Sorry about the confusion.

Comment 14

[Ionuț Goldan [:igoldan]]

:hsivonen Please look over this tracking bug and state whether the Strings Perf* regression alerts are valid or not.
I myself am confused with the results of the bisection.

Comment 15

[Henri Sivonen (:hsivonen)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #0)
> https://hg.mozilla.org/releases/mozilla-beta/
> pushloghtml?changeset=d653ab9ff7622839a4fc7cc38dbfef7f2bc8098f
> 
> As author of one of the patches included in that push, we need your help to
> address this regression.
> 
> Regressions:
> 
>   8%  Strings PerfIsUTF8Example3 macosx64-nightly opt      7,695.17 ->
> 8,281.08
>   6%  Strings PerfHasRTLCharsRU macosx64-nightly opt       931,136.42 ->
> 988,954.25
>   4%  Strings PerfHasRTLCharsDE macosx64-nightly opt       948,214.83 ->
> 988,512.83
>   3%  Strings PerfHasRTLCharsExample3 macosx64-nightly opt 17,079.21 ->
> 17,662.92

None of those patches has any logical connection to these benchmarks. Is there a chance that the upgrade to Rust 1.25.0 got mistakenly attributed to these changesets?

Comment 16

[Ionuț Goldan [:igoldan]]

(In reply to Henri Sivonen (:hsivonen) from comment #15)
> (In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #0)
> > https://hg.mozilla.org/releases/mozilla-beta/
> > pushloghtml?changeset=d653ab9ff7622839a4fc7cc38dbfef7f2bc8098f
> > 
> > As author of one of the patches included in that push, we need your help to
> > address this regression.
> > 
> > Regressions:
> > 
> >   8%  Strings PerfIsUTF8Example3 macosx64-nightly opt      7,695.17 ->
> > 8,281.08
> >   6%  Strings PerfHasRTLCharsRU macosx64-nightly opt       931,136.42 ->
> > 988,954.25
> >   4%  Strings PerfHasRTLCharsDE macosx64-nightly opt       948,214.83 ->
> > 988,512.83
> >   3%  Strings PerfHasRTLCharsExample3 macosx64-nightly opt 17,079.21 ->
> > 17,662.92
> 
> None of those patches has any logical connection to these benchmarks. Is
> there a chance that the upgrade to Rust 1.25.0 got mistakenly attributed to
> these changesets?

I don't know about that. :chmanchester, can you answer :hsivonen's question?

Comment 17

[Chris Manchester (limited bugmail, email directly)]

No, beta is still on rust 1.24.

Comment 18

[Ionuț Goldan [:igoldan]]

(In reply to Chris Manchester (:chmanchester) from comment #17)
> No, beta is still on rust 1.24.

Can we conclude the generated perf alerts are invalid?

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

Yes, we can conclude that.  None of the code in the range could have caused this, and the tests are unstable, but useful.