Closed Bug 889758 (australis-tpaint) Opened 11 years ago Closed 11 years ago

[meta] Australis talos tpaint regression

Categories

(Firefox :: Theme, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
Firefox 26

People

(Reporter: MattN, Unassigned)

References

()

Details

(Keywords: meta, perf, Whiteboard: [Australis:M8][Australis:P1])

$ python compare.py --revision 26608ffcc23c --branch UX --pgo --testname=tpaint
Linux:
    :( tpaint: 155.0 -> 176.0; 207.0.
Linux64:
    :( tpaint: 143.0 -> 156.0; 188.0.
Win:
    :( tpaint: 140.0 -> 174.0; 178.0.
WinXP:
    :( tpaint: 133.0 -> 143.0; 165.0.
Win8:
    :( tpaint: 162.0 -> 168.0; 179.0.
OSX10.7:
    tpaint: 317.0 -> 423.0; 373.0.
OSX64:
    :( tpaint: 271.0 -> 290.0; 316.0.
OSX10.8:
    tpaint: 190.0 -> 280.0; 208.0.

Joel, is tpaint reported to m.dev.tree-management? I don't see recent posts there about it. Is it a talos test that we should care about?

I don't recall this showing up on June 6th when I filed bug 880611 but the graph in the URL only starts on UX on that day for some reason.
Flags: needinfo?(jmaher)
We may as well track the ts_paint regression in the same bug because I don't expect significantly different causes.

$ python compare.py --revision 26608ffcc23c --branch UX --testname=ts_paint
Linux:
    :( ts_paint: 669.053 -> 709.895; 773.474.
Linux64:
    :( ts_paint: 612.737 -> 638.211; 712.579.
Win:
    :( ts_paint: 738.842 -> 20834.2; 778.842.
WinXP:
    :( ts_paint: 519.211 -> 557.0; 577.737.
Win8:
    ts_paint: 683.632 -> 20919.5; 743.789.
OSX10.7:
    ts_paint: 907.947 -> 1112.74; 1025.68.
OSX64:
    ts_paint: 822.053 -> 879.895; 874.842.
OSX10.8:
    ts_paint: 676.105 -> 783.421; 719.053.
Summary: UX branch tpaint regression of up to 20% → UX branch tpaint/ts_paint regression of up to 20%
Linux is good now while the others also got better. Note that compare.py doesn't seem to work with PGO at the moment (bug 886533).

$ python compare.py --revision b6f8dc3e635b --branch UX --testname=tpaint
Linux:
    tpaint: 155.0 -> 176.0; 168.0.   [N/A] [PGO: N/A]
Linux64:
    tpaint: 143.0 -> 156.0; 154.0.   [N/A] [PGO: N/A]
Win:
    tpaint: 140.0 -> 174.0; 173.0.   [N/A] [PGO: N/A]
WinXP:
    :( tpaint: 133.0 -> 143.0; 160.0.   [N/A] [PGO: N/A]
Win8:
    :( tpaint: 161.0 -> 168.0; 170.0.   [N/A] [PGO: N/A]
OSX10.7:
    tpaint: 317.0 -> 410.0; 360.0.   [N/A] [PGO: N/A]
OSX64:
    :( tpaint: 271.0 -> 290.0; 315.0.   [N/A] [PGO: N/A]
OSX10.8:
    tpaint: 190.0 -> 280.0; 207.0.   [N/A] [PGO: N/A]

$ python compare.py --revision b6f8dc3e635b --branch UX --testname=ts_paint
Linux:
    ts_paint: 669.053 -> 709.895; 676.158.   [N/A] [PGO: N/A]
Linux64:
    ts_paint: 612.737 -> 638.211; 622.263.   [N/A] [PGO: N/A]
Win:
    ts_paint: 738.842 -> 20834.2; 747.474.   [N/A] [PGO: N/A]
WinXP:
    :( ts_paint: 515.263 -> 557.0; 573.263.   [N/A] [PGO: N/A]
Win8:
    ts_paint: 683.632 -> 20919.5; 713.684.   [N/A] [PGO: N/A]
OSX10.7:
    ts_paint: 907.947 -> 1112.74; 990.368.   [N/A] [PGO: N/A]
OSX64:
    ts_paint: 823.0 -> 879.895; 868.0.   [N/A] [PGO: N/A]
OSX10.8:
    ts_paint: 675.632 -> 783.421; 698.579.   [N/A] [PGO: N/A]
Summary: UX branch tpaint/ts_paint regression of up to 20% → UX branch tpaint/ts_paint regression of up to 11%
Depends on: 889597
I am not sure if tpaint is reported as a regression.  A while back the tests that we report on was reduced.  After 10 minutes of poking around, I can't find the list of tests we track or don't track.
Flags: needinfo?(jmaher)
We're making progress as ts_paint went from about 9% to 6% now.

Worst offender is still XP:
ts_paint
http://graphs.mozilla.org/graph.html#tests=[[83,137,37],[83,94,37]]

tpaint (still around 14%) and isn't budging much. I'm going to bisect tpaint on old changesets to see what caused this (the previous bisection was only for dirty places).
http://graphs.mozilla.org/graph.html#tests=[[82,59,37],[82,1,37]]

We have about a dozen ideas of things to try in the spreadsheet without the bisection.
Updating summary for clarification, per discussion with MattN...

Bug 889758 and bug 880611 overlap a bit, involving flavors of window painting regressions.

tpaint is effectively a subset of ts_paint, in that both are measuring time to paint a new window, but ts_paint includes app startup time as well. Any tpaint regression should should up in ts_paint, but the reverse is not always true.

EG: currently on XP-nonPGO, there is a ~19ms regression in tpaint, and a ~27ms regression in ts_paint (plain). [ts_paint also has "medium" and "max" variations, which I'm ignoring for the moment.] These numbers imply that there's actually a common ~19ms regression across both, and an additional 8ms (27 - 19) regression in startup. Or that 8ms might be in the noise now.

Using _this_ bug for tracking tpaint, and the other bug for tracking the difference specific to ts_paint.
Summary: UX branch tpaint/ts_paint regression of up to 11% → UX branch tpaint regression of up to 14%
No longer depends on: 892809
Markus, tpaint on XP is the biggest Australis perf regression we're tracking. Would you be able to compare profiles of tpaint (opening a new browser window) between m-c and UX like you did in bug 880611 comment 31 for ts_paint? Thanks
Flags: needinfo?(mstange)
FYI: In my local talos setup on XP, delayedStartup is included in tpaint.
(In reply to Matthew N. [:MattN] from comment #6)
> Markus, tpaint on XP is the biggest Australis perf regression we're
> tracking. Would you be able to compare profiles of tpaint (opening a new
> browser window) between m-c and UX like you did in bug 880611 comment 31 for
> ts_paint?

I'll give it a try.
Flags: needinfo?(mstange)
Hmm, the first attempt didn't work unfortunately, the profiles didn't end up in the log. I'll try to find out why tomorrow.
(In reply to Markus Stange [:mstange] from comment #10)
> Got it this time. Here are two profiles (each) for Windows XP:

Thanks Markus. Was the comparison/analysis you did in bug 880611 comment 31 done by you manually comparing the profiles or did you have a python script to analyze that?
I did it manually last time. But now I've got something better:
http://tests.themasta.com/cleopatra/?report=94e362db39ba63637ad5d84e90f661acb3bb3e4c
This profile is a combined profile of both the two mozilla-central profiles and the two UX profiles, and the samples from the mozilla-central runs each have weight -1, so it actually only shows you the difference now.
This required some changes to cleopatra which are only present on my instance at http://tests.themasta.com/cleopatra/ .

As the next step I'm going to work on doing the same for the Win7 profiles. And Mac after that. Hopefully in an even more automated way.
Thanks Markus that will be very helpful. The link in comment 12 gives me the following error though:
Error fetching profile :(. URL: http://profile-store.commondatastorage.googleapis.com/94e362db39ba63637ad5d84e90f661acb3bb3e4c
I've created another profile for Win7 which contains 4 runs each. Unfortunately it's too big to upload the normal way, but you can download it here: http://tests.themasta.com/comparison-profile-win7.txt.zip
and then use it with http://tests.themasta.com/cleopatra/ .
One interesting part is http://dl.dropboxusercontent.com/u/2901861/Screenshots/Bildschirmfoto%202013-07-19%20um%2010.20.58.png
FYI: I made an improved (I think :) version of the tpaint test in bug 896243. The current attachment works as a standalone webpage. Might be useful for local testing/profiling. (The usual Talos-can-be-different caveats apply!)
(In reply to Markus Stange [:mstange] from comment #15)
Thanks for sharing Markus
Depends on: 897218
Depends on: 897260, 894099
Fixing bug 894099 took this down to about 7%.
Summary: UX branch tpaint regression of up to 14% → UX branch tpaint regression of up to 7%
Depends on: 897614
Depends on: 897617
Summary: UX branch tpaint regression of up to 7% → UX branch tpaint regression of up to 11%
Depends on: 898012
Depends on: 899576
Depends on: 899608
Depends on: 899867
Summary: UX branch tpaint regression of up to 11% → UX branch tpaint regression of up to 7%
Depends on: 899587
Blocks: australis-ts
Alias: australis-tpaint
Depends on: 900408
Depends on: 901425
Depends on: 901786
Depends on: 901827
Depends on: 902164
Conservative update, but really, update! :-)
Summary: UX branch tpaint regression of up to 7% → UX branch tpaint regression of up to 3.5%
Depends on: 902627
Depends on: 902632
Depends on: 902857
No longer depends on: 902924
Depends on: 903113
Depends on: 906075
Depends on: 903044
Unfortunately, the gap on PGO didn't close as much :(
http://bit.ly/19DhDgO
Here is a new XP tpaint comparison profile I did on
https://tbpl.mozilla.org/?tree=Try&rev=93a0eb9bc3e7 and
https://tbpl.mozilla.org/?tree=Try&rev=404ba05c49cf

Bug 899064 seems to be the cause of the markers being missing in previous attempts as setting the pref browser.pagethumbnails.capturing_disabled to true for this run made it work again.

http://tests.themasta.com/cleopatra/#?report=f2b255b02dd22de41cd371b7941182d3cb36fce3
(In reply to Matthew N. [:MattN] from comment #23)
Note that this was for 40 runs of tpaint
The worst case tpaint result is now less than 0.5% (less than 1ms) on PGO and Non-PGO so I think we can call this fixed! On average UX is ~2% faster on tpaint than m-c.

Dependencies of bug 902024 which affect the painting times of tabs will likely drop the worst case below 0% anyways.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Summary: UX branch tpaint regression of up to 3.5% → [meta] Australis talos tpaint regression
No longer depends on: 902100
(In reply to Matthew N. [:MattN] from comment #26)
> The worst case tpaint result is now less than 0.5% (less than 1ms) on PGO
> and Non-PGO so I think we can call this fixed! On average UX is ~2% faster
> on tpaint than m-c.
> 
> Dependencies of bug 902024 which affect the painting times of tabs will
> likely drop the worst case below 0% anyways.

\o/ Great job all!
Target Milestone: --- → Firefox 26
You need to log in before you can comment on or make changes to this bug.