2.37 - 26.9% Explicit Memory / Heap Unclassified / Images / Resident Memory (linux64, linux64-stylo-sequential, osx-10-10, windows10-64, windows7-32) regression on push 90ad57245468 (Thu Mar 8 2018)

RESOLVED FIXED in Firefox 60

Status

()

defect
RESOLVED FIXED
Last year
20 days ago

People

(Reporter: igoldan, Assigned: Mardak)

Tracking

(Blocks 2 bugs, {perf, regression})

56 Branch
Firefox 60
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox58 unaffected, firefox59 unaffected, firefox60 fixed)

Details

Attachments

(1 attachment)

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9322f0ce835f554a0108f59dba244f32ebd8d492&tochange=90ad57245468c398be0be3fcdb2cdec1b9333232

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

 23%  Images windows10-64 opt stylo     7,105,802.51 -> 8,765,477.90
 22%  Images windows10-64 pgo stylo     7,462,464.75 -> 9,090,809.38
  4%  Heap Unclassified windows10-64 pgo stylo41,276,303.39 -> 43,113,613.28
  4%  Heap Unclassified windows10-64 opt stylo41,327,290.70 -> 43,066,678.10
  4%  Resident Memory windows10-64 opt stylo473,359,269.95 -> 490,130,107.88
  4%  Explicit Memory windows10-64 opt stylo311,803,364.88 -> 322,762,749.88
  3%  Resident Memory windows10-64 pgo stylo464,851,298.54 -> 480,887,147.40
  3%  Explicit Memory windows10-64 pgo stylo312,704,988.06 -> 323,460,034.89


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=11997

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests
Other platforms are affected as well:

Regressions:

 27%  Images windows7-32 opt stylo     5,693,513.40 -> 7,225,337.57
 26%  Images linux64-stylo-sequential opt stylo-sequential6,140,888.39 -> 7,729,311.24
 24%  Images osx-10-10 opt stylo       6,498,574.26 -> 8,042,625.19
  8%  Explicit Memory osx-10-10 opt stylo322,331,602.85 -> 348,534,520.90
  6%  Heap Unclassified windows7-32 opt stylo34,326,129.16 -> 36,303,555.68
  6%  Heap Unclassified windows7-32 pgo stylo34,171,244.88 -> 36,113,536.48
  4%  Explicit Memory windows7-32 opt stylo252,345,702.18 -> 262,951,473.47
  4%  Explicit Memory windows7-32 pgo stylo252,441,930.67 -> 262,428,847.34
  3%  Explicit Memory linux64-stylo-sequential opt stylo-sequential310,074,352.13 -> 319,732,109.16
  2%  Explicit Memory linux64 opt stylo313,311,254.14 -> 320,752,266.83
:Mardak It starting to look like bug 1383599 is responsible for these serious AWSY perf regressions. Please look over this issue.

I've retriggered the perf tests, but the results just started to show up.
Flags: needinfo?(edilee)
It looks like the cause for the regression is that AWSY is running with browser.newtabpage.enabled = false. Where before bug 1383599, it was running as if browser.newtabpage.enabled = true (because the pref wasn't being checked).

erahm -> jmaher, does the test need to run with enabled = true, and is this where it's being set:

https://searchfox.org/mozilla-central/rev/588d8120aa11738657da93e09a03378bcd1ba8ec/testing/marionette/components/marionette.js#89-94

And should it be flipped to true (just remove it as the default is true) as that's what it's been running with?
Flags: needinfo?(edilee) → needinfo?(jmaher)
I am not sure here and :erahm is on pto for another week.  if setting this =True fixes this regression we should do that and then leave a pending issue for :erahm.
Flags: needinfo?(jmaher)
Blocks: 1444498
Comment on attachment 8957665 [details]
Bug 1444404 - Use default (true) value for browser.newtabpage.enabled for testing.

https://reviewboard.mozilla.org/r/226586/#review232440

I am not sure what other tests will be affected by this, I suspect quite a few.  Have you tested this on all unittests (try -bo -p all -u all)?
Attachment #8957665 - Flags: review?(jmaher) → review+
Pushed by edilee@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/9a200b344efe
Use default (true) value for browser.newtabpage.enabled for testing. r=jmaher
https://hg.mozilla.org/mozilla-central/rev/9a200b344efe
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → Firefox 60
This patch made a change to files for automation tooling that targets
out-of-tree release channels without the review from a
geckodriver/Marionette peer.  I’m adding some notes to the relevant
files to prevent this from happening in the future as part of bug
1401129.

As I understand it browser.newtabpage.enable = false was broken at
least since Firefox 56?  Do we know about Firefox 55?

My assumption is that it we don’t need to add this pref back for
Firefox 55 compatibility, but I’d like to check to be sure.
Flags: needinfo?(edilee)
Just checked 55.0.3, and browser.newtabpage.enabled does set the page to be mostly blank for the default old Tiles about:newtab, and activity stream isn't packaged, so browser.newtabpage.activity-stream.enabled does nothing.

And in 56.0.2, browser.newtabpage.enabled also shows a mostly blank page for old Tiles about:newtab, and now activity stream is packaged

For 57.0.4, activity stream is the default about:newtab, and browser.newtabpage.enabled doesn't do anything there.

So the pref was not entirely functional since 57 (although one could trigger the blank by disabling both activity stream and newtabpage).
Flags: needinfo?(edilee)
Assignee: nobody → edilee
Thanks for checking, mardak!  I don’t think we should have to worry
about this for the geckodriver 0.20.0 release.
I confirm the push from comment 9 resolved this bug's issue:

== Change summary for alert #12052 (as of Fri, 09 Mar 2018 21:17:16 GMT) ==

Improvements:

 24%  Images windows7-32 pgo stylo     7,615,412.06 -> 5,771,764.67
 21%  Images linux64-stylo-sequential opt stylo-sequential7,713,135.16 -> 6,126,343.62
 20%  Images linux64 opt stylo         7,760,082.68 -> 6,170,993.01
 20%  Images windows10-64 opt stylo    9,022,122.53 -> 7,213,663.75
 19%  Images windows10-64 pgo stylo    9,182,175.45 -> 7,436,880.46
 19%  Images windows7-32 opt stylo     7,119,213.66 -> 5,766,443.51
 17%  Images osx-10-10 opt stylo       7,792,453.23 -> 6,439,950.40
  5%  Heap Unclassified windows7-32 pgo stylo35,995,663.55 -> 34,084,058.05
  5%  Heap Unclassified windows7-32 opt stylo36,214,134.73 -> 34,314,121.08
  4%  Heap Unclassified windows10-64 pgo stylo43,076,576.72 -> 41,142,084.77
  4%  Explicit Memory windows7-32 pgo stylo263,656,416.52 -> 252,222,456.27
  4%  Heap Unclassified windows10-64 opt stylo42,931,024.60 -> 41,118,374.97
  4%  Explicit Memory windows10-64 pgo stylo325,134,727.97 -> 312,914,421.07
  4%  Explicit Memory windows7-32 opt stylo262,229,848.37 -> 252,390,323.57
  4%  Explicit Memory windows10-64 opt stylo323,345,299.33 -> 311,483,309.89
  3%  Explicit Memory osx-10-10 opt stylo352,670,241.26 -> 340,546,597.13
  3%  Explicit Memory linux64-stylo-sequential opt stylo-sequential320,146,774.54 -> 309,550,103.27
  3%  Resident Memory windows10-64 pgo stylo481,836,990.81 -> 466,890,741.34
  3%  Resident Memory windows10-64 opt stylo490,748,101.23 -> 475,774,845.05
  3%  Heap Unclassified osx-10-10 opt stylo74,111,632.79 -> 71,902,783.67
  3%  Explicit Memory linux64 opt stylo321,628,888.69 -> 313,598,628.41

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=12052
Component: Activity Streams: Newtab → New Tab Page
You need to log in before you can comment on or make changes to this bug.