889758 - (australis-tpaint) [meta] Australis talos tpaint regression

Status: RESOLVED FIXED

(Firefox :: Theme, defect)

Description

[Matthew N. [:MattN]]

$ python compare.py --revision 26608ffcc23c --branch UX --pgo --testname=tpaint
Linux:
    :( tpaint: 155.0 -> 176.0; 207.0.
Linux64:
    :( tpaint: 143.0 -> 156.0; 188.0.
Win:
    :( tpaint: 140.0 -> 174.0; 178.0.
WinXP:
    :( tpaint: 133.0 -> 143.0; 165.0.
Win8:
    :( tpaint: 162.0 -> 168.0; 179.0.
OSX10.7:
    tpaint: 317.0 -> 423.0; 373.0.
OSX64:
    :( tpaint: 271.0 -> 290.0; 316.0.
OSX10.8:
    tpaint: 190.0 -> 280.0; 208.0.

Joel, is tpaint reported to m.dev.tree-management? I don't see recent posts there about it. Is it a talos test that we should care about?

I don't recall this showing up on June 6th when I filed bug 880611 but the graph in the URL only starts on UX on that day for some reason.

Comment 1

[Matthew N. [:MattN]]

We may as well track the ts_paint regression in the same bug because I don't expect significantly different causes.

$ python compare.py --revision 26608ffcc23c --branch UX --testname=ts_paint
Linux:
    :( ts_paint: 669.053 -> 709.895; 773.474.
Linux64:
    :( ts_paint: 612.737 -> 638.211; 712.579.
Win:
    :( ts_paint: 738.842 -> 20834.2; 778.842.
WinXP:
    :( ts_paint: 519.211 -> 557.0; 577.737.
Win8:
    ts_paint: 683.632 -> 20919.5; 743.789.
OSX10.7:
    ts_paint: 907.947 -> 1112.74; 1025.68.
OSX64:
    ts_paint: 822.053 -> 879.895; 874.842.
OSX10.8:
    ts_paint: 676.105 -> 783.421; 719.053.

Comment 2

[Matthew N. [:MattN]]

Linux is good now while the others also got better. Note that compare.py doesn't seem to work with PGO at the moment (bug 886533).

$ python compare.py --revision b6f8dc3e635b --branch UX --testname=tpaint
Linux:
    tpaint: 155.0 -> 176.0; 168.0.   [N/A] [PGO: N/A]
Linux64:
    tpaint: 143.0 -> 156.0; 154.0.   [N/A] [PGO: N/A]
Win:
    tpaint: 140.0 -> 174.0; 173.0.   [N/A] [PGO: N/A]
WinXP:
    :( tpaint: 133.0 -> 143.0; 160.0.   [N/A] [PGO: N/A]
Win8:
    :( tpaint: 161.0 -> 168.0; 170.0.   [N/A] [PGO: N/A]
OSX10.7:
    tpaint: 317.0 -> 410.0; 360.0.   [N/A] [PGO: N/A]
OSX64:
    :( tpaint: 271.0 -> 290.0; 315.0.   [N/A] [PGO: N/A]
OSX10.8:
    tpaint: 190.0 -> 280.0; 207.0.   [N/A] [PGO: N/A]

$ python compare.py --revision b6f8dc3e635b --branch UX --testname=ts_paint
Linux:
    ts_paint: 669.053 -> 709.895; 676.158.   [N/A] [PGO: N/A]
Linux64:
    ts_paint: 612.737 -> 638.211; 622.263.   [N/A] [PGO: N/A]
Win:
    ts_paint: 738.842 -> 20834.2; 747.474.   [N/A] [PGO: N/A]
WinXP:
    :( ts_paint: 515.263 -> 557.0; 573.263.   [N/A] [PGO: N/A]
Win8:
    ts_paint: 683.632 -> 20919.5; 713.684.   [N/A] [PGO: N/A]
OSX10.7:
    ts_paint: 907.947 -> 1112.74; 990.368.   [N/A] [PGO: N/A]
OSX64:
    ts_paint: 823.0 -> 879.895; 868.0.   [N/A] [PGO: N/A]
OSX10.8:
    ts_paint: 675.632 -> 783.421; 698.579.   [N/A] [PGO: N/A]

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I am not sure if tpaint is reported as a regression.  A while back the tests that we report on was reduced.  After 10 minutes of poking around, I can't find the list of tests we track or don't track.

Comment 4

[Matthew N. [:MattN]]

We're making progress as ts_paint went from about 9% to 6% now.

Worst offender is still XP:
ts_paint
http://graphs.mozilla.org/graph.html#tests=[[83,137,37],[83,94,37]]

tpaint (still around 14%) and isn't budging much. I'm going to bisect tpaint on old changesets to see what caused this (the previous bisection was only for dirty places).
http://graphs.mozilla.org/graph.html#tests=[[82,59,37],[82,1,37]]

We have about a dozen ideas of things to try in the spreadsheet without the bisection.

Comment 5

[Justin Dolske [:Dolske]]

Updating summary for clarification, per discussion with MattN...

Bug 889758 and bug 880611 overlap a bit, involving flavors of window painting regressions.

tpaint is effectively a subset of ts_paint, in that both are measuring time to paint a new window, but ts_paint includes app startup time as well. Any tpaint regression should should up in ts_paint, but the reverse is not always true.

EG: currently on XP-nonPGO, there is a ~19ms regression in tpaint, and a ~27ms regression in ts_paint (plain). [ts_paint also has "medium" and "max" variations, which I'm ignoring for the moment.] These numbers imply that there's actually a common ~19ms regression across both, and an additional 8ms (27 - 19) regression in startup. Or that 8ms might be in the noise now.

Using _this_ bug for tracking tpaint, and the other bug for tracking the difference specific to ts_paint.

Comment 6

[Matthew N. [:MattN]]

Markus, tpaint on XP is the biggest Australis perf regression we're tracking. Would you be able to compare profiles of tpaint (opening a new browser window) between m-c and UX like you did in bug 880611 comment 31 for ts_paint? Thanks

Comment 7

[Matthew N. [:MattN]]

FYI: In my local talos setup on XP, delayedStartup is included in tpaint.

Comment 8

[Markus Stange [:mstange]]

(In reply to Matthew N. [:MattN] from comment #6)
> Markus, tpaint on XP is the biggest Australis perf regression we're
> tracking. Would you be able to compare profiles of tpaint (opening a new
> browser window) between m-c and UX like you did in bug 880611 comment 31 for
> ts_paint?

I'll give it a try.

Comment 9

[Markus Stange [:mstange]]

Hmm, the first attempt didn't work unfortunately, the profiles didn't end up in the log. I'll try to find out why tomorrow.

Comment 10

[Markus Stange [:mstange]]

Got it this time. Here are two profiles (each) for Windows XP:
Nightly: http://tests.themasta.com/cleopatra/?report=d5e8ed9190f33f4f54ee5c59fcc43abff6d5ac4f
UX: http://tests.themasta.com/cleopatra/?report=52274e64e269acab8d54ed9d816348d7ecd678b9

Comment 11

[Matthew N. [:MattN]]

(In reply to Markus Stange [:mstange] from comment #10)
> Got it this time. Here are two profiles (each) for Windows XP:

Thanks Markus. Was the comparison/analysis you did in bug 880611 comment 31 done by you manually comparing the profiles or did you have a python script to analyze that?

Comment 12

[Markus Stange [:mstange]]

I did it manually last time. But now I've got something better:
http://tests.themasta.com/cleopatra/?report=94e362db39ba63637ad5d84e90f661acb3bb3e4c
This profile is a combined profile of both the two mozilla-central profiles and the two UX profiles, and the samples from the mozilla-central runs each have weight -1, so it actually only shows you the difference now.
This required some changes to cleopatra which are only present on my instance at http://tests.themasta.com/cleopatra/ .

As the next step I'm going to work on doing the same for the Win7 profiles. And Mac after that. Hopefully in an even more automated way.

Comment 13

[Matthew N. [:MattN]]

Thanks Markus that will be very helpful. The link in comment 12 gives me the following error though:
Error fetching profile :(. URL: http://profile-store.commondatastorage.googleapis.com/94e362db39ba63637ad5d84e90f661acb3bb3e4c

Comment 14

[Markus Stange [:mstange]]

I've created another profile for Win7 which contains 4 runs each. Unfortunately it's too big to upload the normal way, but you can download it here: http://tests.themasta.com/comparison-profile-win7.txt.zip
and then use it with http://tests.themasta.com/cleopatra/ .
One interesting part is http://dl.dropboxusercontent.com/u/2901861/Screenshots/Bildschirmfoto%202013-07-19%20um%2010.20.58.png

Comment 15

[Markus Stange [:mstange]]

I've published my scripts to https://github.com/mstange/analyze-tryserver-profiles .

Comment 16

[Markus Stange [:mstange]]

Here's a comparison profile for Mac OS 10.6:
http://tests.themasta.com/comparison-profile-snowleopard.txt.zip

Comment 17

[Justin Dolske [:Dolske]]

FYI: I made an improved (I think :) version of the tpaint test in bug 896243. The current attachment works as a standalone webpage. Might be useful for local testing/profiling. (The usual Talos-can-be-different caveats apply!)

Comment 18

[Matthew N. [:MattN]]

(In reply to Markus Stange [:mstange] from comment #15)
Thanks for sharing Markus

Comment 19

[:Gijs (out for now; he/him)]

Fixing bug 894099 took this down to about 7%.

Comment 20

[:Gijs (out for now; he/him)]

Conservative update, but really, update! :-)

Comment 21

[Mike Conley (:mconley) (:⚙️)]

What do we think chaps? Are we deep enough into the noise on this one?

http://graphs.mattn.ca/graph.html#tests=[[82,137,37],[82,94,37]]&sel=none&displayrange=30&datatype=running

Comment 22

[Matthew N. [:MattN]]

Unfortunately, the gap on PGO didn't close as much :(
http://bit.ly/19DhDgO

Comment 23

[Matthew N. [:MattN]]

Here is a new XP tpaint comparison profile I did on
https://tbpl.mozilla.org/?tree=Try&rev=93a0eb9bc3e7 and
https://tbpl.mozilla.org/?tree=Try&rev=404ba05c49cf

Bug 899064 seems to be the cause of the markers being missing in previous attempts as setting the pref browser.pagethumbnails.capturing_disabled to true for this run made it work again.

http://tests.themasta.com/cleopatra/#?report=f2b255b02dd22de41cd371b7941182d3cb36fce3

Comment 24

[Matthew N. [:MattN]]

(In reply to Matthew N. [:MattN] from comment #23)
Note that this was for 40 runs of tpaint

Comment 25

[Matthew N. [:MattN]]

Fixed link (without the #):
http://tests.themasta.com/cleopatra/?report=f2b255b02dd22de41cd371b7941182d3cb36fce3

Comment 26

[Matthew N. [:MattN]]

The worst case tpaint result is now less than 0.5% (less than 1ms) on PGO and Non-PGO so I think we can call this fixed! On average UX is ~2% faster on tpaint than m-c.

Dependencies of bug 902024 which affect the painting times of tabs will likely drop the worst case below 0% anyways.

Comment 27

[Mike Conley (:mconley) (:⚙️)]

(In reply to Matthew N. [:MattN] from comment #26)
> The worst case tpaint result is now less than 0.5% (less than 1ms) on PGO
> and Non-PGO so I think we can call this fixed! On average UX is ~2% faster
> on tpaint than m-c.
> 
> Dependencies of bug 902024 which affect the painting times of tabs will
> likely drop the worst case below 0% anyways.

\o/ Great job all!