1283900 - big page load regression in Nightly 48

Status: RESOLVED FIXED

(Core :: General, defect)

Description

[Brad Lassey [:blassey] (use needinfo?)]

[Tracking Requested - why for this release]: regression in 48

looking at https://mzl.la/2996XIQ it looks like we had a significant regression in page load times at the end of he 48 (April 24) nightly cycle. It also looks like we fixed most of the regression on June 17.

We should figure out what regressed and what fixed it and get that fix uplifted to 48 (assuming it isn't already).

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

[Tracking Requested - why for this release]:

I don't see it in the beta chart:

https://mzl.la/29hwmUV

But aurora 49 seems to show it:

https://mzl.la/29bCrPL

Maybe something thats just #ifndef RELEASE_BUILD?

Comment 2

[Ben Kelly [:bkelly, not reviewing]]

Sorry, I didn't use permalinks.

Beta:

https://mzl.la/296gZco

Aurora:

https://mzl.la/296hrrf

Comment 3

[Gerry Chang [:gchang]]

Hi Brad,
Is there any update or is there anyone can help on this?

Comment 4

[Gerry Chang [:gchang]]

Track this as there is a possible regression in 48.

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

No I don't have an update. Some one should look at what landed in the regression range and the fix range to figure out what caused this.

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

From comment 1 looks like this isn't a problem currently in 48 beta.

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Andrei, can someone from your team help with the regression range checking here?

Comment 8

[Bogdan Maris, Desktop QA]

We tried to manually check for a regression based on the chart from comment 0 (https://mzl.la/2996XIQ) using app.telemetry Page Speed Monitor add-on (https://addons.mozilla.org/ro/firefox/addon/apptelemetry/) but the results were inconcusive. The difference between a good build and a bad one (ex: good=2016-04-18 and bad=2016-05-08) were either very large in favour of the bad build, good build or equal while loading different websites with high content or not.

Based on the fact that manually we could not find a clear regression, we could identify two ranges from the nightly chart:

Regression where the issue started 2016-04-22 - 2016-04-23
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=0891f0fa044cba28024849803e170ed7700e01e0&tochange=37f04460ddb76d6ef4e7c32a8a6b2fbc44cb8776

Regression where the issue was fixed 2016-06-16 - 2016-06-17
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b9f4f38063951cd5a8b249911aea61869f40fd1f&tochange=5f95858f8ddf21ea2271a12810332efd09eff138

Comment 9

[Gerry Chang [:gchang]]

Hi Joel,
I'm not sure if you can help for this perf regression. Might need your input here.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I don't have any big page specific things, there is svg_opacity which is a larger page and usually hits network issues:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=31536000&series=%5Bmozilla-inbound,b7df5ad27d6ab0f1f5651e5ceab20471cd34dd2d,1,1%5D

there is no pattern there about regressing and getting fixed.


a possible pattern would be in tp5o:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=31536000&series=%5Bmozilla-inbound,7d6d0a62f7a2126944a5e7f463ac524c6f771fd3,1,1%5D&series=%5Bmozilla-inbound,e9786d3bc11a5fff1fa013567494164638d623e9,1,1%5D&zoom=1450222787158.0518,1469410642000,186.79058542925185,283.44486052218866&selected=%5Bmozilla-inbound,e9786d3bc11a5fff1fa013567494164638d623e9,33492,30261054,1%5D

a fix on june 17:
https://treeherder.mozilla.org/perf.html#/alerts?id=1555


there are a series of issues which caused regressions, although I don't see anything specific that is a main cause here.  Feel free to poke around at the regressions and look at the bugs associated with them on the graphs.

I am unclear from this bug title/comments if this is on all platforms, e10s, non e10s, which page, local page vs remote page, etc.

Comment 11

[Gerry Chang [:gchang]]

Hi Andrei,
Since Bogdan is on PTO, can you help to find someone in your team to check if this issue is still in 49 based on comment #8 and comment #10?

Comment 12

[Alexandru Simonca [:asimonca], Release Desktop QA]

Hi Gerry,
I tried testing this issue manually but there are too many factors that come into play here. I installed the same add-on as Bogdan in comment #8 [ https://addons.mozilla.org/ro/firefox/addon/apptelemetry/ ] and tested a few sites for page load times using the Nightly 48.0a1 (id: 20160417030601) build as a baseline because it is not in the affected builds range. I've tried out a build of Nightly (id: 20160424030601) that was in the affected range and one that was in the fixed range (id: 20160618030212), but the results are inconclusive. I have also made a spreadsheet with the results and you can find it here: https://goo.gl/kh2hyf

I guess the only way to see if the issue is still in 49 is to check the graphs.
Please let me know if I can help with anything else.

Comment 13

[Gerry Chang [:gchang]]

Hi Nicholas,
Based on Joel's performance results in comment #10, it seems that your fix enhance the performance. Not sure if your fix is related to this bug. Can you help to take a look at it?

Comment 14

[u408661]

(In reply to Gerry Chang [:gchang] from comment #13)
> Hi Nicholas,
> Based on Joel's performance results in comment #10, it seems that your fix
> enhance the performance. Not sure if your fix is related to this bug. Can
> you help to take a look at it?

It may well be related. However, the regressing feature (bug 1016628, which landed in 48) is currently enabled only if NIGHTLY_BUILD is defined (there is an intermittent leak I'm still trying to track down before letting the feature ride the trains). Additionally, the performance fix (bug 1273882) landed in 50. So if there's still a regression, its cause is somewhere else.

Comment 15

[Liz Henry (:lizzard) (relman/hg->git project)]

Is there still a regression in 51?  Since this is still nightly only, I'm marking 49/50 as disabled.

Comment 16

[u408661]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #15)
> Is there still a regression in 51?  Since this is still nightly only, I'm
> marking 49/50 as disabled.

The regression caused by my feature was fixed in bug 1273882, so unless there was a separate regression caused by another landing, this should be fixed as of 50 (though also disabled there for other reasons).

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

I would agree with :hurley.

Comment 18

[Liz Henry (:lizzard) (relman/hg->git project)]

Thank you all! It sounds like we can close this bug now since it is disabled in 49 and fixed in 50/51.