Talos regression tsvgx_nochrome 5% on Android 4.0.4, 8% on Android 2.2, Aug 23 2013

RESOLVED INCOMPLETE

Status

()

Firefox for Android
General
P5
normal
RESOLVED INCOMPLETE
5 years ago
3 years ago

People

(Reporter: gbrown, Unassigned)

Tracking

({perf, regression})

Trunk
x86
Android
perf, regression
Points:
---

Firefox Tracking Flags

(fennec+)

Details

(Reporter)

Description

5 years ago
From dev-tree-management Digest, Vol 56, Issue 120:

Subject: <Regression> mobile - SVG-ASAP, NoChrome - Android 4.0.4 -
        5.48%

Regression: mobile - SVG-ASAP, NoChrome - Android 4.0.4 - 5.48% increase
------------------------------------------------------------------------
    Previous: avg 1056.810 stddev 15.183 of 12 runs up to revision d136c8999d96
    New     : avg 1114.743 stddev 5.850 of 12 runs since revision b2486721572e
    Change  : +57.933 (5.48% / z=3.816)
    Graph   : http://mzl.la/17PFSmi

Changeset range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d136c8999d96&tochange=b2486721572e


From dev-tree-management Digest, Vol 56, Issue 120:

Subject: <Regression> mobile - SVG-ASAP,        NoChrome - Android 2.2
        (Native) - 8.18%

Regression: mobile - SVG-ASAP, NoChrome - Android 2.2 (Native) - 8.18% increase
-------------------------------------------------------------------------------
    Previous: avg 1074.850 stddev 3.398 of 12 runs up to revision d136c8999d96
    New     : avg 1162.780 stddev 40.306 of 12 runs since revision b2486721572e
    Change  : +87.930 (8.18% / z=25.876)
    Graph   : http://mzl.la/16lB4lQ

Changeset range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d136c8999d96&tochange=b2486721572e
(Reporter)

Comment 1

5 years ago
Note that this is the new svg test, tsvgx_nochrome -- not to be confused with tsvg_nochrome.  Currently this shows on tbpl only on mozilla-central, only for Android 4.0.4 (not Android 2.2), under the letter T.
This is the Fig merge as well.

Comment 3

4 years ago
(In reply to Geoff Brown [:gbrown] from comment #1)
> Note that this is the new svg test, tsvgx_nochrome -- not to be confused
> with tsvg_nochrome.  Currently this shows on tbpl only on mozilla-central,
> only for Android 4.0.4 (not Android 2.2), under the letter T.

Supposedly, the new test is better at detecting regressions than the old test, so I'd like to think that it detected a real regression. Is there a reason to suspect that this is a false positive?

(In reply to Kevin Brosnan [:kbrosnan] from comment #2)
> This is the Fig merge as well.

What does this mean/imply?
Flags: needinfo?(kbrosnan)
(Reporter)

Comment 4

4 years ago
(In reply to Avi Halachmi (:avih) from comment #3)
> Is there a reason to suspect that this is a false positive?

I think this is a genuine regression. If you look at the 30-day graph:

http://graphs.mozilla.org/graph.html#tests=[[285,11,29]]&sel=none&displayrange=30&datatype=running

everything before the fig merge (about:home rewrite) is < 1090 (mean of about 1050); everything since is > 1105 (mean of about 1110).
(Reporter)

Comment 5

4 years ago
:margaret -- Did you expect this type of regression from Fig? Do you have any suggestions for tracking this down?
Flags: needinfo?(margaret.leibovic)
tracking-fennec: ? → 26+
Merging the work from bug 862793. Which was done on a project branch, fig.
Flags: needinfo?(kbrosnan)
What sorts of things does tsvgx_nochrome test? It seems pretty difficult to narrow down a changeset range within the Fig changesets because we didn't run that test on Fig TBPL.

Comment 8

4 years ago
https://wiki.mozilla.org/Buildbot/Talos/Tests#tsvg.2C_tsvgx

SVG pages load times, and SVG completion durations of fixed animations/rendering-iterations, while iterating in unlimited frame rate mode (ASAP).

Comment 9

4 years ago
(In reply to Geoff Brown [:gbrown] from comment #5)
> :margaret -- Did you expect this type of regression from Fig? Do you have
> any suggestions for tracking this down?

Sorry for the slow response. No, I wasn't expecting this. I wonder if this might also be caused by infrastructural issues, such as the ones we're suspecting in bug 908823.
Flags: needinfo?(margaret.leibovic)
(In reply to Chenxia Liu [:liuche] from comment #7)
> What sorts of things does tsvgx_nochrome test? It seems pretty difficult to
> narrow down a changeset range within the Fig changesets because we didn't
> run that test on Fig TBPL.

Running tsvgx on intermediate fig builds/csets should be possible with some help from the a-team IMO.

(In reply to :Margaret Leibovic from comment #9)
> Sorry for the slow response. No, I wasn't expecting this. I wonder if this
> might also be caused by infrastructural issues, such as the ones we're
> suspecting in bug 908823.

Bug 90888 indicates a very big variation due to inconsistent runs of robopan (crashes, etc), but the graph on this bug at least looks healthily noisy and consistent. If I didn't know better (and I don't know better), I'd say it looks like a real regression.

Here are several overlaid SVG-ASAP graphs on different platforms:
http://graphs.mozilla.org/graph.html#tests=[[285,11,20],[285,11,29],[281,131,31],[281,131,33],[281,131,35],[281,131,37],[281,131,25]]&sel=1376243694031,1378835694031&displayrange=30&datatype=running

The android 2.2 and 4 regressions are visible on Aug-21, but the other non-mobile platforms are absolutely flat (they changed their name on aug-20-ish, but the invisible prior history is also stable).
While looking into this we made some discoveries that should probably be addressed:
1. This is a nochrome style of talos pageloader test. It uses the pageloader add-on:
   http://hg.mozilla.org/build/talos/file/ca2229a32cb6/talos/pageloader
2. The pageloader "tp-cmdline.js" component will substitute "pageloader.xul" for the normal "browser.xul". This worked OK for XUL Fennec where the entire UI was defined in browser.xul, but native Fennec uses a completely Java UI. This means that UI is still created, even for nochrome.

Notes:
The Java UI expects certain JSON message to be sent from browser.js to let it know Gecko is ready. Since pageloader.js does not send these, I'm not sure if something bad is happening or not.
The Java UI creates a pure Java "home page" if Fennec is not told to explicitly load a page. This home page is not HTML or XUL, but Java. It displays itself over the Gecko content area. I believe this is happening with the pageloader based tests. This will introduce some "chrome" affects to the "nochrome" tests.

Perhaps we could try to load about:blank into the Java UI by using the test.py tploadaboutblank flag?
We recently added the tploadaboutblank flag to talos/pageloader.  This is a viable option to explore.  I am glad there is some attention on this and would be happy to help modify talos to be as realistic as possible with the existing tests.

let me see if tploadaboutblank works with tsvgx -noChrome on android.
(In reply to Joel Maher (:jmaher) from comment #12)
> We recently added the tploadaboutblank flag to talos/pageloader.  This is a
> viable option to explore.

I don't think this would relate to nochrome fennec, and regardless, tploadaboutblank is a partially broken hack and we should replace it. It works on the talos slaves because they're fast, but it hangs on slower machines frequently. We already have a proper fix to replace tploadaboutblank with the real reason for its existence: reload the page while ignoring any cache (tart.html#auto). The fix is on MattN's local talos repo which is used to run TART on low-end machines. I didn't file a bug on it yet since it's lower priority than unblocking the fx-team.

This was off topic, but the bottom line is that tploadaboutblank should not be used elsewhere because it's broken, and regardless, I cannot see how it fits this fennec/fig scenario.
(In reply to Avi Halachmi (:avih) from comment #13)

> This was off topic, but the bottom line is that tploadaboutblank should not
> be used elsewhere because it's broken, and regardless, I cannot see how it
> fits this fennec/fig scenario.

When you start Fennec, does not matter that it's Talos or not, the Java front-end code will ask: "Did we get a URL passed into the Intent?" If so, open the URL in a tab. If not, display a pure Java UI in place of where we normally display web content.

This happens regardless of nochrome, talos, whatever. It will always happen. Fennec is the only browser that does this. This essentially pollutes the test. The situation we are trying to simulate, "no chrome", in fact has a ton of chrome.
mark, what messages are required to be sent to the javaUI to display the page?  I would like to make this as accurate as possible.
Given that this seems to be an wider issue with how talos tests run on Fennec, does it still make sense to track this for 26?
Flags: needinfo?(mark.finkle)
(In reply to Lucas Rocha (:lucasr) from comment #16)
> Given that this seems to be an wider issue with how talos tests run on
> Fennec, does it still make sense to track this for 26?

No. I am re-noming and we can adjust the tracking level in the next triage.

(In reply to Joel Maher (:jmaher) from comment #15)
> mark, what messages are required to be sent to the javaUI to display the
> page?  I would like to make this as accurate as possible.

I need to get back to you on that Joel.
tracking-fennec: 26+ → ?
Flags: needinfo?(mark.finkle)
tracking-fennec: ? → 26+
We probably need someone to work on this or simply make it a tracking+
tracking-fennec: 26+ → ?
These tests have not been run since Sept 18th. Does anyone know why we stopped running them? Or maybe we are just not reporting the data anymore?
(Reporter)

Comment 20

4 years ago
I think we switched to svg-ASAP -- bug 903681.
(In reply to Matt Brubeck (:mbrubeck) from comment #21)
> Indeed, we switched from "SVG-ASAP NoChrome" to "SVG-ASAP":
> http://graphs.mozilla.org/graph.html#tests=[[285,11,29],[281,11,
> 29]]&sel=1350416698837,1381952698837&displayrange=365&datatype=running

And it looks like we improved the regression a bit too. I still think we should try using -tploadaboutblank to minimize the effect of the fennec about:home Java code.
(In reply to Mark Finkle (:mfinkle) from comment #22)
> ... I still think we
> should try using -tploadaboutblank to minimize the effect of the fennec
> about:home Java code.

This flag doesn't do what you think it does (it loads about:blank very briefly before [re]loading the tested content). It's a hack which was added specifically to workaround some issue which talos has with running the TART test.

We should/will remove this specific hack from talos and replace it with a flag to reload the page with no caching.
(In reply to Avi Halachmi (:avih) from comment #23)

> This flag doesn't do what you think it does (it loads about:blank very
> briefly before [re]loading the tested content). It's a hack which was added
> specifically to workaround some issue which talos has with running the TART
> test.
> 
> We should/will remove this specific hack from talos and replace it with a
> flag to reload the page with no caching.

Then we still need something else to fix this issue.
tracking-fennec: ? → +
(In reply to Mark Finkle (:mfinkle) from comment #22)
> ... I still think we
> should try using -tploadaboutblank to minimize the effect of the fennec
> about:home Java code.

(In reply to Mark Finkle (:mfinkle) from comment #24)
> we still need something else to fix this issue.

Sounds reasonable. If you can define the talos change you need, then jmaher could probably help you implement it.
2 actions from this:
* nochrome (current): ensure we pass about:blank to the browser instead of loading the home page which in java land does a lot
* chrome (future): figure out a way to use a single browser and load the uri's from pageloader which will test gecko+java (metro mode does this, possible reuse here)
even passing in about:blank on the command line yields us with about:home.  Is this a bug in fennec? the commandline handler?
filter on [mass-p5]
Priority: -- → P5
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.