Closed
Bug 833000
Opened 12 years ago
Closed 12 years ago
Talos Regression trobopan on Android, Jan 19
Categories
(Firefox for Android Graveyard :: General, defect)
Tracking
(firefox21+ wontfix, fennec21+)
RESOLVED
WONTFIX
People
(Reporter: gbrown, Assigned: jfkthame)
References
Details
From dev-tree-management Digest, Vol 49, Issue 86:
Message: 3
Date: Sat, 19 Jan 2013 02:21:19 -0000
From: nobody@cruncher.build.mozilla.org
To: dev-tree-management@lists.mozilla.org
Subject: <Regression> Mozilla-Inbound - Robocop Pan Benchmark -
Android 2.2 (Native) - 37.6%
Message-ID:
<20130119022119.A20221042C0@cruncher.srv.releng.scl3.mozilla.com>
Content-Type: text/plain; charset="us-ascii"
Regression: Mozilla-Inbound - Robocop Pan Benchmark - Android 2.2 (Native) - 37.6% increase
-------------------------------------------------------------------------------------------
Previous: avg 8754.542 stddev 1047.943 of 30 runs up to revision c01ed477136c
New : avg 12043.640 stddev 649.181 of 5 runs since revision 46726c3ab4e1
Change : +3289.098 (37.6% / z=3.139)
Graph : http://mzl.la/XKwZWE
Changeset range: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=c01ed477136c&tochange=46726c3ab4e1
Changesets:
* http://hg.mozilla.org/integration/mozilla-inbound/rev/3bf0fc40df43
: Alexander Surkov <surkov.alexander@gmail.com> - Bug 569356, part1 - add support of event scenarios in testing, r=marcoz
: http://bugzilla.mozilla.org/show_bug.cgi?id=569356
* http://hg.mozilla.org/integration/mozilla-inbound/rev/46726c3ab4e1
: Brian Smith <bsmith@mozilla.com> - Bug 624514: Make PSM access the network.ntlm.send-lm-response pref only on the main thread, r=honzab
: http://bugzilla.mozilla.org/show_bug.cgi?id=624514
Bugs:
* http://bugzilla.mozilla.org/show_bug.cgi?id=624514 - PSM accesses the pref service off the main thread
* http://bugzilla.mozilla.org/show_bug.cgi?id=569356 - Coalescence of state change events is bad
Message: 1
Date: Sat, 19 Jan 2013 01:21:55 -0000
From: nobody@cruncher.build.mozilla.org
To: dev-tree-management@lists.mozilla.org
Subject: <Regression> Mozilla-Inbound - Robocop Pan Benchmark -
Android 2.2 (Native) - 40.6%
Message-ID:
<20130119012155.9F1271040FC@cruncher.srv.releng.scl3.mozilla.com>
Content-Type: text/plain; charset="us-ascii"
Regression: Mozilla-Inbound - Robocop Pan Benchmark - Android 2.2 (Native) - 40.6% increase
-------------------------------------------------------------------------------------------
Previous: avg 8489.092 stddev 496.581 of 30 runs up to revision 86e23bc67300
New : avg 11938.940 stddev 659.812 of 5 runs since revision ea6e7fbcd3a5
Change : +3449.848 (40.6% / z=6.947)
Graph : http://mzl.la/XKqfIm
Changeset range: http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=86e23bc67300&tochange=ea6e7fbcd3a5
Changesets:
* http://hg.mozilla.org/integration/mozilla-inbound/rev/bc71821fcb9f
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - test fix 7 - explicitly use Droid Serif for the greek-uppercase-1 test on Android, as default fonts may not be suitable. r=dbaron
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/integration/mozilla-inbound/rev/0d6975d654a6
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - test 'fix' 8 - mark remaining problematic tests as fuzzy on android. r=blassey
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/integration/mozilla-inbound/rev/ea3d953f806d
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - Ship fonts (Open Sans and Charis SIL Compact) for content in Firefox for Android. r=mfinkle,blassey
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/integration/mozilla-inbound/rev/823ab8a1c9c0
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - test fix 6 - canvas and svg language-font tests pass on android with the Open Sans font prefs. r=dbaron
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/integration/mozilla-inbound/rev/4d919195cdf0
: Sriram Ramasubramanian <sriram@mozilla.com> - Bug 832433: Use Android spinners on tabs UI for phones. [r=mfinkle]
: http://bugzilla.mozilla.org/show_bug.cgi?id=832433
* http://hg.mozilla.org/integration/mozilla-inbound/rev/ea6e7fbcd3a5
: Sriram Ramasubramanian <sriram@mozilla.com> - Bug 832433: Remove Android Tabs widget in tabs ui. [r=mfinkle] [needs-clobber]
: http://bugzilla.mozilla.org/show_bug.cgi?id=832433
Bugs:
* http://bugzilla.mozilla.org/show_bug.cgi?id=832433 - Phones should use Android spinners
* http://bugzilla.mozilla.org/show_bug.cgi?id=831354 - Ship fonts for content in Firefox for Android
| Reporter | ||
Updated•12 years ago
|
tracking-fennec: --- → ?
| Reporter | ||
Comment 1•12 years ago
|
||
Regression from bug 831354?
https://hg.mozilla.org/integration/mozilla-inbound/rev/0d4ce26f2c28 / https://hg.mozilla.org/integration/mozilla-inbound/rev/f849e0aa18d6 :
trobopan: 11344.0
Previous:
https://hg.mozilla.org/integration/mozilla-inbound/rev/abafe7350c6f /
https://hg.mozilla.org/integration/mozilla-inbound/rev/e5321a44b63c :
trobopan: 8204.5
OS: Mac OS X → Android
| Assignee | ||
Comment 2•12 years ago
|
||
I'm a bit confused by those regression messages; is it claiming there are -two- substantial regressions in quick succession (40.6% -and- 37.6%)? The graph links aren't much help in trying to visualize what's happened, as they yield a graph where the y-axis is so huge that all the data points basically sit at the bottom.
What exactly does Robopan test, and how does it test it?
| Reporter | ||
Comment 3•12 years ago
|
||
(In reply to Jonathan Kew (:jfkthame) from comment #2)
> I'm a bit confused by those regression messages; is it claiming there are
> -two- substantial regressions in quick succession (40.6% -and- 37.6%)?
It is just one regression. Note that both regression alert messages indicate that the test metric has changed from approx 8000 to approx 12000. I do not know why the regression is reported twice some times, but I have seen it before.
> What exactly does Robopan test, and how does it test it?
See test source at https://hg.mozilla.org/mozilla-central/annotate/a63eeede2d15/mobile/android/base/tests/testPan.java.in. This is a robocop-based Talos test. It loads a copy of a long wikipedia page, starts monitoring frame times, scrolls (drags) up to 1000 times, then reports the sum of the squares of the frame delays (where a delay is considered a frame rate worse than 40 fps). I think of it as a measure of how fast we can pan/scroll/drag.
More relevant code at https://hg.mozilla.org/mozilla-central/file/91b9995f9ac7/build/mobile/robocop/FennecNativeDriver.java.in#l186 and https://hg.mozilla.org/mozilla-central/file/91b9995f9ac7/mobile/android/base/gfx/PanningPerfAPI.java#l43
| Reporter | ||
Comment 4•12 years ago
|
||
Detected on mozilla-central in dev-tree-management Digest, Vol 49, Issue 90:
Message: 3
Date: Mon, 21 Jan 2013 13:22:03 -0000
From: nobody@cruncher.build.mozilla.org
To: dev-tree-management@lists.mozilla.org
Subject: <Regression> mobile - Robocop Pan Benchmark - Android 2.2
(Native) - 34.6%
Message-ID:
<20130121132203.B68271042A2@cruncher.srv.releng.scl3.mozilla.com>
Content-Type: text/plain; charset="us-ascii"
Regression: mobile - Robocop Pan Benchmark - Android 2.2 (Native) - 34.6% increase
----------------------------------------------------------------------------------
Previous: avg 8391.592 stddev 618.751 of 30 runs up to revision a97eb4f09112
New : avg 11295.640 stddev 639.275 of 5 runs since revision 1d122eaa9070
Change : +2904.048 (34.6% / z=4.693)
Graph : http://mzl.la/UKXnzg
Changeset range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a97eb4f09112&tochange=1d122eaa9070
Changesets:
* http://hg.mozilla.org/mozilla-central/rev/b784ce7fd90f
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - Ship [but don't actually use] fonts (Open Sans and Charis SIL Compact) for content in Firefox for Android. r=mfinkle,blassey
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/mozilla-central/rev/0cc2c67234c6
: Marco Bonardo <mbonardo@mozilla.com> - Bug 831725 - Fix wrong merge on toolkit/components/places/tests/browser/head.js
r=me
: http://bugzilla.mozilla.org/show_bug.cgi?id=831725
* http://hg.mozilla.org/mozilla-central/rev/ec597237c8fe
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - test fix 9 - give the CGJ reftest a large line-height, so it's less sensitive to the metrics of any fallback font that happens to be found.
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/mozilla-central/rev/8a294f877514
: Daniel Glazman <daniel@glazman.org> - Bug 832025 - Clear the cached styles and type-in state props when pressing Enter inside a heading or a list item; r=ehsan
: http://bugzilla.mozilla.org/show_bug.cgi?id=832025
* http://hg.mozilla.org/mozilla-central/rev/8472e5898021
: Filippo Cristofoletti <filippo987@gmail.com> - Bug 832280 - Disable MSVC warning C4482: nonstandard extension used: enum 'xyz' used in qualified name. r=ted
: http://bugzilla.mozilla.org/show_bug.cgi?id=832280
* http://hg.mozilla.org/mozilla-central/rev/323d01be9d0d
: Ehsan Akhgari <ehsan@mozilla.com> - Merge mozilla-central into mozilla-aurora
* http://hg.mozilla.org/mozilla-central/rev/c059c73cd16c
: Ms2ger <ms2ger@gmail.com> - Merge m-c to m-i.
* http://hg.mozilla.org/mozilla-central/rev/1f42182ac5b3
: Joel Maher <jmaher@mozilla.com> - Bug 823165 - fix robocop tests to work on panda boards. r=gbrown
: http://bugzilla.mozilla.org/show_bug.cgi?id=823165
* http://hg.mozilla.org/mozilla-central/rev/6d14b59b7ce5
: Fabrice Desr? <fabrice@mozilla.com> - Bug 814226 - Permission checks for "webapps-manage" could probably be friendlier r=sicking
: http://bugzilla.mozilla.org/show_bug.cgi?id=814226
* http://hg.mozilla.org/mozilla-central/rev/87fe8d808537
: Raymond Lee <raymond@raysquare.com> - Bug 820763 - Stop using addvisit() in toolkit tests. r=mak
: http://bugzilla.mozilla.org/show_bug.cgi?id=820763
* http://hg.mozilla.org/mozilla-central/rev/a5f5694ad2c0
: Steve Fink <sfink@mozilla.com> - Bug 828753 - jsid rooting, mostly in jsinfer.*. Also switch JSObject from struct to class. r=terrence
: http://bugzilla.mozilla.org/show_bug.cgi?id=828753
* http://hg.mozilla.org/mozilla-central/rev/e5321a44b63c
: David Zbarsky <dzbarsky@gmail.com> - Bug 832169 - Convert SVGAnimatedLength to WebIDL r=bz
: http://bugzilla.mozilla.org/show_bug.cgi?id=832169
* http://hg.mozilla.org/mozilla-central/rev/abafe7350c6f
: David Zbarsky <dzbarsky@gmail.com> - Bug 831673, r=bz
: http://bugzilla.mozilla.org/show_bug.cgi?id=831673
* http://hg.mozilla.org/mozilla-central/rev/f849e0aa18d6
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - update default font prefs to use the bundled Open Sans and Charis SIL Compact on Android, but not on Gonk. r=dbaron
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* http://hg.mozilla.org/mozilla-central/rev/0d4ce26f2c28
: Jonathan Kew <jkew@mozilla.com> - bug 831354 - test fix 6 - canvas and svg language-font tests pass on android with the Open Sans font prefs. r=dbaron
: http://bugzilla.mozilla.org/show_bug.cgi?id=831354
* and 6 more
Bugs:
* http://bugzilla.mozilla.org/show_bug.cgi?id=823872 - [OS.File] Add sanity tests for OS.Constants.{libc, Win}
* http://bugzilla.mozilla.org/show_bug.cgi?id=832425 - gcPreserveCode() should not depend on JS_GC_ZEAL
* http://bugzilla.mozilla.org/show_bug.cgi?id=831623 - Move handleSymbolResponse and fetchSymbol to a new class so it can be used for late write stacks too
* http://bugzilla.mozilla.org/show_bug.cgi?id=832169 - Convert SVGAnimatedLength to WebIDL
* http://bugzilla.mozilla.org/show_bug.cgi?id=831725 - Wrong merge on toolkit/components/places/tests/browser/head.js
* http://bugzilla.mozilla.org/show_bug.cgi?id=830926 - AZPC TrackTouch x/y displacement should not be rounded to int
* http://bugzilla.mozilla.org/show_bug.cgi?id=832589 - B2G reftest runs have errors like "E/GeckoConsole( 812): [JavaScript Error: "The character encoding of the HTML document was not declared..." between each test, from data URI for "<!--CLEAR-->"
* http://bugzilla.mozilla.org/show_bug.cgi?id=828753 - jsid rooting, mostly in jsinfer.*
* http://bugzilla.mozilla.org/show_bug.cgi?id=814226 - Permission checks for "webapps-manage" could probably be friendlier
* http://bugzilla.mozilla.org/show_bug.cgi?id=831673
* http://bugzilla.mozilla.org/show_bug.cgi?id=832280 - Disable MSVC warning C4482: nonstandard extension used: enum 'xyz' used in qualified name
* http://bugzilla.mozilla.org/show_bug.cgi?id=832025 - Major regression in HTML editing rules in CSS mode related to TypeInState
* http://bugzilla.mozilla.org/show_bug.cgi?id=831354 - Ship fonts for content in Firefox for Android
* http://bugzilla.mozilla.org/show_bug.cgi?id=820763 - Stop using addvisit() in toolkit tests
* http://bugzilla.mozilla.org/show_bug.cgi?id=823165 - fix robocop tests to work on panda boards
| Assignee | ||
Comment 5•12 years ago
|
||
Hmm - there doesn't appear to be an equivalent trobopan regression on Android 4.0, afaict; only 2.2 seems to be affected.
I tried panning that long wikipedia article (well, the current version on the web, so it might not be identical, but similar enough) with Aurora and Nightly on my HTC phone that's running Android 2.2, and the performance didn't "feel" any different to me. That's very subjective, of course, but if it truly regressed by 40% I'd have expected to notice some kind of sluggishness in the Nightly build.
Kats, it looks like you were involved with creating trobopan; could you look into this and help me understand whether there's a real regression we should be worrying about?
Comment 6•12 years ago
|
||
(In reply to Jonathan Kew (:jfkthame) from comment #5)
> Hmm - there doesn't appear to be an equivalent trobopan regression on
> Android 4.0, afaict; only 2.2 seems to be affected.
>
Since the graphserver is useless, I scraped the data from tbpl using Firebug:
1aa6bf0a46a2: Android 2.2 opt: 11700.0
c3072e256623: Android 2.2 opt: 11945.75
cc665373ff61: Android 2.2 opt: 11751.0
595daefc448c: Android 2.2 opt: 11859.25
bb5400c00877: Android 2.2 opt: 11275.75
5ea59ba8d604: Android 2.2 opt: 11797.5
1392d241c137: Android 2.2 opt: 11752.25
b8791be70fd0: Android 2.2 opt: 11023.0
ed17c53be925: Android 2.2 opt: 11530.25
332f32b1fad9: Android 2.2 opt: 10654.0
0a25461c8e97: Android 2.2 opt: 12479.0
b2d55f2d4910: Android 2.2 opt: 12029.75
3830ad0eb049: Android 2.2 opt: 12814.75
422d79422232: Android 2.2 opt: 10695.5
3d58243e4cd6: Android 2.2 opt: 12125.5
3ffefc46a8bd: Android 2.2 opt: 12855.75
01c964705be7: Android 2.2 opt: 10870.5
56efd49e70dd: Android 2.2 opt: 11523.0
2106f6631cd6: Android 2.2 opt: 12572.75
d837646f7676: Android 2.2 opt: 11522.75
88daef90f2ab: Android 2.2 opt: 11344.5
1d122eaa9070: Android 2.2 opt: 11878.75
ca1f12ab55c8: Android 2.2 opt: 11751.75
68f8f4b74f63: Android 2.2 opt: 11679.5
ce70deca6e8e: Android 2.2 opt: 11231.0
0d4ce26f2c28: Android 2.2 opt: 11344.0
abafe7350c6f: Android 2.2 opt: 8204.5
a5f5694ad2c0: Android 2.2 opt: 8943.25
87fe8d808537: Android 2.2 opt: 9394.75
6d14b59b7ce5: Android 2.2 opt: 7864.25
1f42182ac5b3: Android 2.2 opt: 9136.25
323d01be9d0d: Android 2.2 opt: 7724.5
8472e5898021: Android 2.2 opt: 8234.25
8a294f877514: Android 2.2 opt: 8702.75
ec597237c8fe: Android 2.2 opt: 8962.75
0cc2c67234c6: Android 2.2 opt: 10410.25
b784ce7fd90f: Android 2.2 opt: 8383.25
b1d4081f0582: Android 2.2 opt: 9249.75
6b280e155484: Android 2.2 opt: 8912.0
f02b8f754b74: Android 2.2 opt: 8786.5
31fea3e0be16: Android 2.2 opt: 8799.25
1591384ce3ad: Android 2.2 opt: 9354.5
c30676933621: Android 2.2 opt: 7985.75
ba89f8698668: Android 2.2 opt: 11967.75
2674257f6117: Android 2.2 opt: 10678.5
d835ea3c8339: Android 2.2 opt: 12073.0
097d044586c1: Android 2.2 opt: 12395.5
fdeb0c833138: Android 2.2 opt: 11208.25
84edc4c47182: Android 2.2 opt: 11568.75
There does appear to be a legitimate regression on Android 2.2.
1aa6bf0a46a2: Android 4.0 opt: 3947.0
c3072e256623: Android 4.0 opt: 3228.0
cc665373ff61: Android 4.0 opt: 2943.5
595daefc448c: Android 4.0 opt: 5246.25
bb5400c00877: Android 4.0 opt: 2864.25
5ea59ba8d604: Android 4.0 opt: 3682.5
1392d241c137: Android 4.0 opt: 3956.5
b8791be70fd0: Android 4.0 opt: 3710.5
ed17c53be925: Android 4.0 opt: 5371.25
332f32b1fad9: Android 4.0 opt: 1843.5
0a25461c8e97: Android 4.0 opt: 5418.25
3830ad0eb049: Android 4.0 opt: 3375.25
833978b3dadf: Android 4.0 opt: 3182.0
422d79422232: Android 4.0 opt: 4299.75
93ca53db2831: Android 4.0 opt: 4627.25
3d58243e4cd6: Android 4.0 opt: 3580.75
3ffefc46a8bd: Android 4.0 opt: 4007.5
01c964705be7: Android 4.0 opt: 3249.5
56efd49e70dd: Android 4.0 opt: 4194.25
2106f6631cd6: Android 4.0 opt: 4316.25
d837646f7676: Android 4.0 opt: 3954.25
88daef90f2ab: Android 4.0 opt: 3391.5
1d122eaa9070: Android 4.0 opt: 5177.5
ca1f12ab55c8: Android 4.0 opt: 3774.0
68f8f4b74f63: Android 4.0 opt: 3657.5
ce70deca6e8e: Android 4.0 opt: 4291.0
0d4ce26f2c28: Android 4.0 opt: 5345.5
abafe7350c6f: Android 4.0 opt: 2640.5
a5f5694ad2c0: Android 4.0 opt: 4668.5
87fe8d808537: Android 4.0 opt: 2260.75
6d14b59b7ce5: Android 4.0 opt: 3417.0
1f42182ac5b3: Android 4.0 opt: 3501.75
323d01be9d0d: Android 4.0 opt: 3488.5
8472e5898021: Android 4.0 opt: 2371.75
8a294f877514: Android 4.0 opt: 3067.5
ec597237c8fe: Android 4.0 opt: 3856.25
0cc2c67234c6: Android 4.0 opt: 5033.0
b784ce7fd90f: Android 4.0 opt: 2735.25
b1d4081f0582: Android 4.0 opt: 2770.75
6b280e155484: Android 4.0 opt: 4252.5
f02b8f754b74: Android 4.0 opt: 3928.75
7dbbb6e3d240: Android 4.0 opt: 3306.5
31fea3e0be16: Android 4.0 opt: 3852.0
1591384ce3ad: Android 4.0 opt: 5148.5
c30676933621: Android 4.0 opt: 3401.75
9dd844b7152e: Android 4.0 opt: 3352.5
ba89f8698668: Android 4.0 opt: 5153.75
2674257f6117: Android 4.0 opt: 4040.75
d835ea3c8339: Android 4.0 opt: 5514.75
097d044586c1: Android 4.0 opt: 2916.25
fdeb0c833138: Android 4.0 opt: 5194.25
84edc4c47182: Android 4.0 opt: 3924.5
4886df89ef45: Android 4.0 opt: 3790.75
This data is all over the place but I think the same regression applies here as well. You can see the value is mostly in the 2000-4000 range without the patch, and in the 3000-5000 range with the patch. But we can focus on the 2.2 version to debug this since it's much more evident there.
The test itself reports the sum of the checkerboard amounts (as a percentage of the screen area) per frame. So the number could go up if we either (1) have more frames with checkboard or (2) increase the checkerboard per frame (or some combination thereof). I suspect the most likely problem here is that painting got a little bit slower because of the different font, and so we are checkerboarding more. The best way to verify this would probably be to push a change to try that prints out all the frame checkerboard values before and after your change and compare them. If the painting got slower then the best way to debug it is probably to use the profiler on a device to see what got slower and optimize it.
I'll push a couple of try builds to get the frame checkerboard values.
Blocks: 831354
Comment 7•12 years ago
|
||
Oh whoops, I got the panning test mixed up with the checkerboard test. The panning test actually reports the sum of the squares of how "late" a frame was (relative to the expected frame timings at 60fps). It is an indicator of jank that goes up if either the number of frames goes up or if frames are delayed more.
I realized my mistake too late to fix my try pushes but I've pushed new try builds at https://tbpl.mozilla.org/?tree=Try&rev=045e78d3362d and https://tbpl.mozilla.org/?tree=Try&rev=828a713b56f7 which will hopefully provide more info by providing the frame delays for the tests.
Comment 8•12 years ago
|
||
Ok, so I grabbed all the frame delays (represented as how far off from the ideal it was) from the above try pushes and ran some stats on them. Before the change:
Samples: 3408
Average: 0.103286
Stddev: 5.11998
Max: 55
Min: -23
After the change:
Samples: 3469
Average: 0.307293
Stddev: 5.38364
Max: 44
Min: -23
The number of samples is about the same, but the average has moved up, which indicates that on average we are drawing frames later than we used to. Whereas before frames were 0.1 ms late on average, they are now 0.3 ms late on average. The stddev hasn't moved much. I'm not entirely sure why switching fonts would cause this.
Comment 9•12 years ago
|
||
BenWa, any ideas on why switching to a different default font would cause average frame jank to increase?
Updated•12 years ago
|
tracking-firefox21:
--- → ?
Comment 10•12 years ago
|
||
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #9)
> BenWa, any ideas on why switching to a different default font would cause
> average frame jank to increase?
I don't think it could be font at all even by some crazy guess. Is bug 832433 included in the regression range? Because that would explain it.
Comment 11•12 years ago
|
||
No, the Android 2.2 data from comment 6 is pretty clear in blaming the font change. i.e. it got better in http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=ba89f8698668&tochange=c30676933621 when the first landing was backed out and it got worse again in http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=abafe7350c6f&tochange=0d4ce26f2c28 when it re-landed.
Updated•12 years ago
|
Updated•12 years ago
|
tracking-fennec: ? → 21+
| Assignee | ||
Comment 12•12 years ago
|
||
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #8)
> Ok, so I grabbed all the frame delays (represented as how far off from the
> ideal it was) from the above try pushes and ran some stats on them. Before
> the change:
>
> Samples: 3408
> Average: 0.103286
> Stddev: 5.11998
> Max: 55
> Min: -23
>
> After the change:
>
> Samples: 3469
> Average: 0.307293
> Stddev: 5.38364
> Max: 44
> Min: -23
>
> The number of samples is about the same, but the average has moved up, which
> indicates that on average we are drawing frames later than we used to.
> Whereas before frames were 0.1 ms late on average, they are now 0.3 ms late
> on average. The stddev hasn't moved much. I'm not entirely sure why
> switching fonts would cause this.
I'm no statistician, but it seems to me that the change here, when compared to the standard deviation, or when considered in the light of the wide scattering of values, is really quite small.
I can imagine a couple of reasons why changing fonts could reasonably be expected affect these figures, at least slightly. First, the new fonts (particularly the serif face) have more generous line-spacing, which means that pages with a large amount of text will often become slightly taller overall, and so they would be expected to require more scroll actions to cover the entire extent. This may be one reason the "after" run shows a larger number of samples. Second, the new fonts are larger in the sense of having a more extensive character set, leading to larger font files; as such, they probably take marginally longer to load, to find character mappings in the cmap table, etc. So there could be a (very slight) slow-down in general text-rendering performance, simply as a result of using larger resources.
However, I think the "40% regression" as reported by trobopan is quite misleading, as it implies that something has gotten -dramatically- worse, and that's simply not true. I can't "sense" any difference in panning performance or responsiveness using builds before/after the font landing.
To try and visualize the change better, I put the frame-delay figures from Kats' try runs (comment 8) into Google charts, and came up with a bar chart showing the distribution of frame delay values; see http://tinyurl.com/a59u2t6.
Looking at this chart, we can see that the red bars (with the new fonts) do indeed show more frames that have a small delay (in the 0-10ms range), but perhaps more striking is the overall similarity between the "before" and "after" figures, with the vast majority of frames falling within the ±6ms range. (It's also curious, though perhaps not significant, that the most extreme outliers, at +49 and +55ms, occur in the "before" figures, not "after".)
ISTM, then, that the regression here is in fact very small, and unlikely to be significant to users. And I think we (i.e. someone with more stats knowledge than me!) should reconsider how trobopan is calculating its numbers, in order to give a more realistic interpretation.
Comment 13•12 years ago
|
||
(In reply to Jonathan Kew (:jfkthame) from comment #12)
> I'm no statistician, but it seems to me that the change here, when compared
> to the standard deviation, or when considered in the light of the wide
> scattering of values, is really quite small.
Agreed.
> I can imagine a couple of reasons why changing fonts could reasonably be
> expected affect these figures, at least slightly. First, the new fonts
> (particularly the serif face) have more generous line-spacing, which means
> that pages with a large amount of text will often become slightly taller
> overall, and so they would be expected to require more scroll actions to
> cover the entire extent. This may be one reason the "after" run shows a
> larger number of samples.
Somebody else had suggested this as well and so I had also looked at the average frame delay for just the first 1000 frames. This should be unaffected by the page getting longer as the amount scrolled would be the same. I don't have the exact results any more but the increase in frame delay was evident there as well.
> Second, the new fonts are larger in the sense of
> having a more extensive character set, leading to larger font files; as
> such, they probably take marginally longer to load, to find character
> mappings in the cmap table, etc. So there could be a (very slight) slow-down
> in general text-rendering performance, simply as a result of using larger
> resources.
Agreed, but none of this should affect the compositor thread, which is what the frame delay is measuring. All of the slowdown should be entirely contained within the gecko main thread. That's the part that concerns me.
> However, I think the "40% regression" as reported by trobopan is quite
> misleading, as it implies that something has gotten -dramatically- worse,
> and that's simply not true. I can't "sense" any difference in panning
> performance or responsiveness using builds before/after the font landing.
Agreed.
> To try and visualize the change better, I put the frame-delay figures from
> Kats' try runs (comment 8) into Google charts, and came up with a bar chart
> showing the distribution of frame delay values; see
> http://tinyurl.com/a59u2t6.
>
> Looking at this chart, we can see that the red bars (with the new fonts) do
> indeed show more frames that have a small delay (in the 0-10ms range), but
> perhaps more striking is the overall similarity between the "before" and
> "after" figures, with the vast majority of frames falling within the ±6ms
> range. (It's also curious, though perhaps not significant, that the most
> extreme outliers, at +49 and +55ms, occur in the "before" figures, not
> "after".)
This is a good visualization, and shows that the difference here is much less than "40%".
> ISTM, then, that the regression here is in fact very small, and unlikely to
> be significant to users. And I think we (i.e. someone with more stats
> knowledge than me!) should reconsider how trobopan is calculating its
> numbers, in order to give a more realistic interpretation.
Agreed.
In summary, I think there is still an unanswered question here, but since the actual regression is minimal and not really noticeable to users, the cost/benefit ratio of digging into it beyond what has already been done is frankly just not worth it. I'm ok with WONTFIXing this bug.
Summary: Talos Regression trobopan on Android, 40%, Jan 19 → Talos Regression trobopan on Android, Jan 19
Comment 14•12 years ago
|
||
Don't think there is anything that needs to happen here , marking this wontfix based on comment 13.Please reopen if needed.
Updated•12 years ago
|
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
Updated•5 years ago
|
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•