1470831 - 4.4 - 4.61% Resident Memory (windows7-32) regression on push 9253a827f52d262efee2f659095d38cf0b44fd0f (Fri Jun 22 2018)

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected an awsy regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=9253a827f52d262efee2f659095d38cf0b44fd0f

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  5%  Resident Memory windows7-32 pgo stylo     482,513,195.27 -> 504,745,754.46
  4%  Resident Memory windows7-32 opt stylo     488,047,113.58 -> 509,543,927.01

Improvements:

  5%  Heap Unclassified windows7-32 opt stylo     32,213,794.82 -> 30,530,923.12
  5%  Heap Unclassified windows7-32 pgo stylo     32,506,967.19 -> 30,824,038.82
  4%  Heap Unclassified windows10-64 pgo stylo    41,411,305.64 -> 39,640,443.79
  4%  Heap Unclassified windows10-64 opt stylo    40,841,103.42 -> 39,157,636.41


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=13989

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/AWSY/Tests

Comment 1

[Ionuț Goldan [:igoldan]]

:erahm I believe the regressions were caused by harness updates. If I'm right and these were expected, can you please mark this as WONTFIX?

Comment 2

[Ionuț Goldan [:igoldan]]

:erahm Also, can you please link this bug to the right one, out of this list: bug 1470268, bug 1395540, bug 1442361?

Comment 3

[Eric Rahm [:erahm]]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #2)
> :erahm Also, can you please link this bug to the right one, out of this
> list: bug 1470268, bug 1395540, bug 1442361?

I'll update it once I figure out what's going on. The only thing I can think of is bug 1395540, but that doesn't make a ton of sense.

Comment 4

[Eric Rahm [:erahm]]

My best guess is this has something to do with disabling sandboxing during the AWSY tests. I confirmed that DMD itself is not enabled (there's not output, no entries in the about:memory dumps). I'll do some test runs w/o disabling sandboxing to see if that fixes the regression.

Comment 5

[Eric Rahm [:erahm]]

Okay, it looks like disabling sandboxing was the culprit [1]. I can update my patches to only disable sandboxing if dmd is enabled. For the record this will reverse the heap unclassified wins, which implies there might be some unreported memory in sandboxing.

[1] https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=c48100d92a3278c1bd5871170f4b12c127811257&framework=4&selectedTimeRange=172800

Comment 6

[Ionuț Goldan [:igoldan]]

(In reply to Eric Rahm [:erahm] (please no mozreview requests) from comment #3)
> (In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #2)
> > :erahm Also, can you please link this bug to the right one, out of this
> > list: bug 1470268, bug 1395540, bug 1442361?
> 
> I'll update it once I figure out what's going on. The only thing I can think
> of is bug 1395540, but that doesn't make a ton of sense.

> Okay, it looks like disabling sandboxing was the culprit

So I guess bug 1395540 is more related to this perf tracking bug?

Comment 7

[Eric Rahm [:erahm]]

Attachment: Only disable sandboxing in dmd builds

Disabling sandboxing seems to cause a memory regression for win32 builds which
affects our AWSY numbers. This switches over to only disabling sandboxing if
using a DMD build so that we can still run without crashing.

Comment 8

[Bob Clary [:bc] (inactive)]

Comment on attachment 8987970 [details] [diff] [review]
Only disable sandboxing in dmd builds

Review of attachment 8987970 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with answer and resolution to the question.

::: testing/awsy/conf/prefs.json
@@ -7,5 @@
>      "network.proxy.socks_remote_dns": true,
>      "network.proxy.type": 1,
>      "plugin.disable": true,
> -    "startup.homepage_override_url": "",
> -    "startup.homepage_welcome_url": "",

I didn't see a discussion on removing startup.homepage_welcome_url. Was that intentional?

Comment 9

[Eric Rahm [:erahm]]

(In reply to Bob Clary [:bc:] from comment #8)
> Comment on attachment 8987970 [details] [diff] [review]
> Only disable sandboxing in dmd builds
> 
> Review of attachment 8987970 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with answer and resolution to the question.
> 
> ::: testing/awsy/conf/prefs.json
> @@ -7,5 @@
> >      "network.proxy.socks_remote_dns": true,
> >      "network.proxy.type": 1,
> >      "plugin.disable": true,
> > -    "startup.homepage_override_url": "",
> > -    "startup.homepage_welcome_url": "",
> 
> I didn't see a discussion on removing startup.homepage_welcome_url. Was that
> intentional?

Good catch! That was not intentional.

Comment 10

[Eric Rahm [:erahm]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e8664442920191ef7536bda215681281d6daaa56
Bug 1470831 - Only disable sandboxing in dmd builds. r=bc

Comment 11

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/e86644429201

Comment 12

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Eric Rahm [:erahm] (please no mozreview requests) from comment #5)
> which implies there might be some unreported memory in sandboxing.

There probably is unreported memory in Windows sandboxing; it's a nontrivial piece of code and it's imported from Chromium.  :bobowen is probably the best person to talk to to find out more.

Comment 13

[Ionuț Goldan [:igoldan]]

Push from comment 10 reverted back all perf results from comment 0:

== Change summary for alert #14158 (as of Fri, 29 Jun 2018 16:03:47 GMT) ==

Regressions:

  7%  Heap Unclassified windows7-32 pgo stylo     30,584,363.95 -> 32,583,166.36
  5%  Heap Unclassified windows7-32 opt stylo     30,738,689.62 -> 32,316,411.08
  5%  Heap Unclassified windows10-64 opt stylo    38,871,939.00 -> 40,830,586.42
  4%  Heap Unclassified windows10-64 pgo stylo    39,206,249.33 -> 40,868,592.74

Improvements:

  5%  Resident Memory windows7-32 opt stylo     508,877,788.70 -> 483,808,846.69
  5%  Resident Memory windows7-32 pgo stylo     503,084,265.41 -> 478,858,345.01

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14158

Comment 14

[Liz Henry (:lizzard)]

Eric, what is the status here for 62? Is this a regression you think we should accept or is there work that you think we should uplift to beta?

Comment 15

[Eric Rahm [:erahm]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #14)
> Eric, what is the status here for 62? Is this a regression you think we
> should accept or is there work that you think we should uplift to beta?

It's a regression caused by the test harness, I'm okay with accepting it.