1447573 - 8.72% Strings PerfStripCharsWhitespace (osx-10-10) regression on push ea926eb29682 (Mon Mar 12 2018)

Status: RESOLVED WORKSFORME

(Core :: Disability Access APIs, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a platform microbenchmarks regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=ea926eb2968226a0c58a2f7e2fc173035cd49ca8

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  9%  Strings PerfStripCharsWhitespace osx-10-10 opt      314,041.33 -> 341,420.08


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=12154

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Platform_Microbenchmarks

Comment 1

[Ionuț Goldan [:igoldan]]

:surkov Bug 1257669 seems to have caused this perf regression on March 12. I filed this tracking bug to check whether this makes sense or not. What are your thoughts?

Fortunately, a recent change canceled this regression.

Comment 2

[Ionuț Goldan [:igoldan]]

FYI Strings * tests are highly sensitive and very often raise false alerts.

Comment 3

[alexander :surkov (:asurkov)]

that patch was all about assertions in debug builds. It should have zero effect for release builds. Is it a false positive indeed?

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to alexander :surkov from comment #3)
> that patch was all about assertions in debug builds. It should have zero
> effect for release builds. Is it a false positive indeed?

Thanks for sharing this.

Well, for the last month, the test ran as bi-modal.

Then either bug 1257669 or other bug which landed around that time made the test return the upper values.
Switching from multi-modal to uni-modal sometimes is even desired.

:jmaher I'm into resolving this as WORKSFORME. Do you have other thoughts?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

sometimes a switch in modality is a regression or improvement, often it is just random luck.  I am fine with worksforme.