Closed Bug 660264 Opened 14 years ago Closed 9 years ago

D3D9 layers is a 28% tp4 hit on Talos

Categories

(Core :: Graphics, defect)

Product:
Core
Component:
Graphics
Platform:
All
Windows XP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED INCOMPLETE
Tracking Flags:
Tracking Status
firefox5 - wontfix
firefox6 + wontfix
firefox7 - ---

People

(Reporter: dao, Assigned: BenWa)

References

(
URL
)

Details

(Keywords: perf)

Attachments

(3 files)

Talos R3 XP 005 - DxDiag
36.73 KB, text/plain
Details
Results log
1.37 MB, application/x-tar
Details
Blacklist outdated GeForce 9400 due to D3D9 regression
1.25 KB, patch
joe
: review+
Details | Diff | Splinter Review
layers disabled: http://tbpl.mozilla.org/?tree=Try&rev=1f58c265e1c3 260.98 (±1.12) layers enabled: http://tbpl.mozilla.org/?tree=Try&rev=d856cbfa6c38 334.82 (±5.24) http://perf.snarkfest.net/compare-talos/index.html?oldRevs=1f58c265e1c3&newRev=d856cbfa6c38&tests=tp4&submit=true Note that bug 581212 was a 9% tp4 regression when it landed, which wasn't noticed because of bug 651659.
Keywords: perf
Benoit Girard has offered to look into this!
Assignee: nobody → bgirard
Are you hoping to get this down to zero with little effort? Given that it landed with a 9% regression already, I'd be skeptical about this. So at least on mozilla-beta and mozilla-aurora, I think we should disable D3D9 layers.
I'd be very uncomfortable disabling D3D9 layers without baking that on m-c for at least a week or two. I don't even think that configuration is green on trunk right now.
(In reply to comment #4) > I'd be very uncomfortable disabling D3D9 layers without baking that on m-c > for at least a week or two. I don't even think that configuration is green > on trunk right now. This is very unsettling. That configuration is being used already when the driver isn't whitelisted, and we're exposing a pref for disabling hardware acceleration. How can we ship Firefox at all when disabling D3D9 layers would be a big uncertainty? I'm confused about the greenness, though. We're running tests without layers acceleration at least on Linux, aren't we? If so, I'd suspect and hope that tests failing on Windows without layers acceleration are just fragile rather than exposing a real problem.
We currently test both accelerated and software-only paths on Linux (perhaps only on try? Matt Woodrow knows more), and those are green. On recent pushes to try, though, you'll see reftest-no-d2d-d3d results that are permaorange; nobody has spent the time to investigate those, and so they've stayed permaorange. I also suspect these are just fragile tests, though.
Still investigating, my first attempt using xperf is unsuccessful. This article explains why and gives suggestion on how to approach the problem: http://msdn.microsoft.com/en-us/library/bb172234%28v=vs.85%29.aspx
URL: http://msdn.microsoft.com/en-us/libra...
Discussed this with Jeff. What we think is happening is that state changes are costing us a lot of cycles (See link), as well as texture creation/deletion cause a mode transition to the kernel software driver (including a flush of the D3D runtime command buffer). A possible fix would be to do more careful state management and trying to reuse textures. http://msdn.microsoft.com/en-us/library/bb172234%28v=vs.85%29.aspx#Appendix
> A possible fix would be to do more careful state management and trying to reuse textures. Is this something we'd want to try on aurora and beta? I'm still skeptical. FWIW, I think we should disable this even on trunk and only turn it back on when it doesn't degrade perf.
(In reply to comment #9) > > A possible fix would be to do more careful state management and trying to reuse textures. > > Is this something we'd want to try on aurora and beta? I'm still skeptical. It's hard to say at this point. Once we have pin pointed specific bottlenecks we should have a better idea of the complexity of the fix.
(In reply to comment #9) > > A possible fix would be to do more careful state management and trying to reuse textures. > > Is this something we'd want to try on aurora and beta? I'm still skeptical. > > FWIW, I think we should disable this even on trunk and only turn it back on > when it doesn't degrade perf. You're presuming a correlation between Tp4 results and subjective performance. We know for example that D3D9 improves scrolling performance by quite a bit (easy to measure as well, there's a nice applet out there). This is not necessarily true. It is completely feasible that we make some decisions when composition is cheap that cause us to do some extra rendering on initial page load, to provide smoother page interaction after initial load. However this interaction is not what Tp4 tests. There's also the issue that even if I look at tests with large differences that should be noticable by the human eye (200ms+) I don't see differences anywhere near that (maybe you do BenWa?). I.e. disabling now would be a bit premature, unless there is data that shows there's a better user experience without D3D9 of course. An interesting example is that D2D scores worse on Tp4 than your layers disabled measurement (290 +/- some for D2D). Yet we know, from feedback and obvious subjective differences, that page rendering is perceived as significantly faster when using Direct2D.
> I.e. disabling now would be a bit premature, unless there is data that shows > there's a better user experience without D3D9 of course. This reversal of the burden of proof doesn't seem kosher to me. Bug 581212 probably wouldn't have been allowed to stick if we had known about the regression.
An interesting observation is that the wall-clock time difference between execution of the two test runs is actually only 2 +/- 2 minutes on 20 minutes. Meaning at most a 20% difference. I'm not sure what this means (if anything) though.
(In reply to comment #12) > > I.e. disabling now would be a bit premature, unless there is data that shows > > there's a better user experience without D3D9 of course. > > This reversal of the burden of proof doesn't seem kosher to me. Bug 581212 > probably wouldn't have been allowed to stick if we had known about the > regression. The fact of the matter is in general Firefox 4 is perceived as faster. And so is D2D. i.e. there's considerable evidence the Tp4 measurement is not a good indicator at least in the D2D situation. Targeted measurements on both D3D9 and D2D using for example the window Redraw method (which times painting, although it doesn't work well anymore, sadly) have indicated a speedup. I believe we should understand our measurements before we start possibly regressing our users because some number came out of a test where nobody I've been able to find can actually answer my question of what work is actually included in the Tp4 measurement (other than 'until onload fires').
> The fact of the matter is in general Firefox 4 is perceived as faster. Right, but that's true without D3D9 layers as well. > And so is D2D. i.e. there's considerable evidence the Tp4 measurement is not a > good indicator at least in the D2D situation. D2D + D3D10 layers actually improve Tp4, I think, according to http://perf.snarkfest.net/compare-talos/index.html?oldRevs=1f58c265e1c3&newRev=d856cbfa6c38&tests=tp4&submit=true > I believe we should understand our measurements before we start possibly > regressing our users because some number came out of a test where nobody > I've been able to find can actually answer my question of what work is > actually included in the Tp4 measurement (other than 'until onload fires'). "until onload fires" seems pretty clear to me.
(In reply to comment #15) > > D2D + D3D10 layers actually improve Tp4, I think, according to > http://perf.snarkfest.net/compare-talos/index. > html?oldRevs=1f58c265e1c3&newRev=d856cbfa6c38&tests=tp4&submit=true Hrm, maybe it's the deviation in Tp4 runs that is throwing off my rough comparison. That's nowhere near the difference in the measurements I did last year though. > > I believe we should understand our measurements before we start possibly > > regressing our users because some number came out of a test where nobody > > I've been able to find can actually answer my question of what work is > > actually included in the Tp4 measurement (other than 'until onload fires'). > > "until onload fires" seems pretty clear to me. For what it's worth, I tested this just a minute ago, and 'until onload fires' does not include painting. I actually put a ::Sleep(3000) on each paint (makes the browser a pain to use btw), and local measurements do -not- seem to be affected. i.e. this might be setup overhead from D3D9, which we don't see being offset by the painting being faster. We'd need a measurement that includes painting I'd say. As a matter of fact, considering it -doesn't- include painting, I'm a little surprised this is mattering a lot at all. I wonder why non-painting stuff would be so strongly affected.
onload doesn't including painting of the page. So the Tp4 regression here is probably all about chrome paint time (which makes sense given that bug 651659 hid the regression). The switch from GDI-only to D3D9 means that when something small changes in the window (like say, the spinning "loading" graphic), instead of repainting only a tiny part of the window we recomposite the entire window. That recompositing happens on the GPU but it's still not free. That may be part of the cause of the regression. Given that we don't have user complaints about slow pageload with D3D9 on, I think it's OK to spend some time to understand the problem and hopefully come up with a fix and then decide what to do about branches.
Page load time is hard to quantify with the naked eye and I wouldn't even count on somebody having done a side-by-side comparison with D3D9 on/off, so the lack of concrete complaints isn't surprising. We do know that people prefer certain browsers because they seem to load pages faster.
(In reply to comment #18) > Page load time is hard to quantify with the naked eye and I wouldn't even > count on somebody having done a side-by-side comparison with D3D9 on/off, so > the lack of concrete complaints isn't surprising. We do know that people > prefer certain browsers because they seem to load pages faster. Sounds like a double blind experiment would be a good idea!
(In reply to comment #12) > This reversal of the burden of proof doesn't seem kosher to me. Bug 581212 > probably wouldn't have been allowed to stick if we had known about the > regression. This should probably be discussed in dev.platform instead of here. (In reply to comment #17) > onload doesn't including painting of the page. So the Tp4 regression here is > probably all about chrome paint time (which makes sense given that bug 651659 > hid the regression). Should we be measuring time to paint instead?
I suppose we should really wait for the MozAfterPaint event following the load event, but this likely wouldn't make a substantial difference here, since D3D9 layers would still delay the load event.
I was not able to reproduce the Talos regression locally on my win7 machine or my winxp machine. Either my setup is incorrect or the problem is related to the drivers on the Talos machine. I requested a Talos machine to investigate further.
(In reply to comment #21) > I suppose we should really wait for the MozAfterPaint event following the > load event, but this likely wouldn't make a substantial difference here, > since D3D9 layers would still delay the load event. We should definitely change Tp to wait for MozAfterPaint rather than onload. I filed bug 661918 on this issue.
we aren't going to be taking any changes here for Firefox 5 so setting status-firefox5 accordingly.
status-firefox5: --- → wontfix
I've got tp4 setup on the machine and awaiting results now.
Tested on nightly: http://hg.mozilla.org/mozilla-central/rev/bbbc80a2bd8c With D3D9: 284 (2 runs) Without D3D9: 266 (2 runs) I was able to reproduce the regression on a Talos machine but not on my laptop, it's not clear what the cause is. One thing that stuck out while testing is that Talos is not fully patched (running XP SP2, not SP3) and drivers are likely also out of date. Should we get approval to patch/update drivers on the Talos machine and re runs tp4? Should we compare with another benchmarks?
(In reply to comment #27) > I was able to reproduce the regression on a Talos machine but not on my > laptop, it's not clear what the cause is. One thing that stuck out while > testing is that Talos is not fully patched (running XP SP2, not SP3) and > drivers are likely also out of date. > > Should we get approval to patch/update drivers on the Talos machine and re Depends on what's your goal is. I don't think we want the regression to just go away on the talos machine. If the problem is an outdated driver, then we shouldn't be using D3D9 with that driver, since our users have outdated drivers as well.
(In reply to comment #27) > Should we get approval to patch/update drivers on the Talos machine and re > runs tp4? Should we compare with another benchmarks? I believe after you release the Talos machine it gets re-imaged anyhow. bjacob and/or catlee will know for sure.
(In reply to comment #29) > I believe after you release the Talos machine it gets re-imaged anyhow. bjacob > and/or catlee will know for sure. This is correct. We should see if updating it (locally) changes the numbers, and then go from there.
Attached file Results log —
Here are the result logs including the results before and after patching. The following were patched to the latest version: - Windows Updates - GeForce display drivers - DirectX 9.0c Scores: With D3D9: 248, 249 Without D3D9: 251, 251 After patching both scores improved. Notice that with D3D9 the score are faster.
We discussed the results during the GFX meeting: The current Talos machine drivers are outdated and were whitelisted specifically to get test coverage. I will open a follow up bug with releng to get the machines upgraded. I'm going to wait a shorth while before releasing the Talos machine but will do so soon so please voice any concern ASAP.
Summary: D3D9 layers is a 28% tp4 hit → D3D9 layers is a 28% tp4 hit with outdated drivers on Talos
Depends on: 665773
(In reply to comment #31) > Scores: > With D3D9: 248, 249 > Without D3D9: 251, 251 > > After patching both scores improved. Notice that with D3D9 the score are > faster. Are these numbers correct? The improvement from 248 to 249 is barely noticeable.
(In reply to comment #33) > Are these numbers correct? The improvement from 248 to 249 is barely > noticeable. The improvement by patching is from 284 (original report got 334) to 248 with D3D9 enabled, the numbers 248, 249 are the results from a different run with a patched system and D3D9 enabled. These numbers are nearly identical as expected.
Depends on: 629759
Is there anything we can and need to do for Firefox 6? We're deep into Beta now and unless there's something very low risk on the horizon here, I think we should probably punt again.
As far as I can tell and as indicated earlier, we should block the driver that makes us slow. I'm not sure why this hasn't happened yet.
(In reply to comment #35) > Is there anything we can and need to do for Firefox 6? We're deep into Beta > now and unless there's something very low risk on the horizon here, I think > we should probably punt again. There is nothing we need to do for FF6. We originally tracked it to see if it was a real regression, its not, its only on machines with really old drivers and those drivers are only white listed because they are on our Talos machines. Really we just need to get bug 629759 fixed to get the perf numbers reporting right. Marking wontfix for this reason.
status-firefox6: --- → wontfix
I'm confused as to how this issue is being dealt with -- please enlighten me. Is the old driver some synthetic one that's unlikely to exist on users' machines? If not, how would this not be a real regression for these users?
I can't speak to the likeliehood, but basically we'd blacklist it other than for its use on the Talos machines. Updating the driver fixed the perf issue, see #32. It should be fixed, but it doesn't need to be resolved for FF6 given current info.
So what's needed to get bug 629759 back on track? Can we whitelist the driver exclusively for talos? It baffles me that we're repeatedly pushing this back without knowing that the driver is niche. If it's a stock driver from when these machines were set up, then there's actually a risk that it's widespread.
(In reply to comment #40) > So what's needed to get bug 629759 back on track? > Can we whitelist the driver exclusively for talos? > > It baffles me that we're repeatedly pushing this back without knowing that > the driver is niche. If it's a stock driver from when these machines were > set up, then there's actually a risk that it's widespread. FTR this is what the XP slaves run: Name: NVIDIA GeForce 9400 Main driver: nv4_disp.dll Version: 6.14.001.7756 Date: 7/22/2009 *I* think that stock drivers were installed since I see no mention in the ref documentation and I don't see a C:\NVidia directory as I had seen on the win7 slaves: > https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP
tracking-firefox7: --- → ?
My understanding is the following: 1. This occurs on an older driver that we would blacklist if the talos machines weren't running it 2. Bug 629759 tracks updating the talos machines 3. Once bug 629759 is fixed, there is no reason to not have this driver on the blacklist 4. We can update the downloadable blacklist separate from a Firefox release 5. This bug tracks the blacklisting and is dependent on bug 629759 6. This has been a potential issue ever since D3D9 landed Based on that information we're denying the tracking-firefox7 as this does not have to be tracked for a specific release (but we do really want bug 629759 fixed and this driver version blacklisted)
tracking-firefox7: ? → -
(In reply to comment #42) > 6. This has been a potential issue ever since D3D9 landed 7. We only discovered it recently. (Well, two months ago by now.) I can't imagine that this would have been allowed to stay on mozilla-central even for a day if talos hadn't been broken :/
We did crunch some numbers. D3D9 layers is received by (as a percentage of the users on each of these platforms). Win7 | 8% WinVista | 9% Win2003 | 18% WinXP | 5% Unfortunately Joe couldn't check dll version from crash stats to see how common this driver is. Roughly I think this adds up to 7-8% of our user base (but it could be calculated more strictly). Turning it off completely hurts users without this driver. Since we don't need a code change here, we just need to prioritize the driver upgrade here so we can blocklist.. I spoke with catlee about that on Friday, he is looking into it.
My impression from talking with JP was that this was specific to the linux drivers, which doesn't sound correct at all now. Which platforms need driver updates? What are the minimum version and desired version for each platform? For each platform there's a bunch of work that releng needs to do to figure out how to upgrade the driver. After that we'll want to close all the trees so we can upgrade the drivers and establish a new baseline for test results.
Windows XP needs the update; we're not picky about what version we use, just as long as it's recent and >= 257.21.
(In reply to comment #45) > For each platform there's a bunch of work that releng needs to do to figure > out how to upgrade the driver. After that we'll want to close all the trees > so we can upgrade the drivers and establish a new baseline for test results. These are the things that need to happen: * test/write an OPSI package to deploy and install the drivers * all the XP machines would have to be updated at the same time ** otherwise talos numbers would fluctuate depending on the machine This does not necessarily need a downtime but being very cut and clear when things happen. The steps to take are: disable every XP machine, mark the OPSI package to be installed for all slaves and then reboot the slaves. Landing a dummy change on mozilla-central and other branches with email regression coverage would also be important if we want to blame the correct changeset.
Here's the patch to blacklist the old drivers once the new drivers are deployed.
Attachment #551160 - Flags: review?(joe)
Attachment #551160 - Flags: review?(joe) → review+
(In reply to Christian Legnitto [:LegNeato] from comment #42) > 4. We can update the downloadable blacklist separate from a Firefox release Apparently this doesn't apply here...
(In reply to Dão Gottwald [:dao] from comment #40) > It baffles me that we're repeatedly pushing this back without knowing that > the driver is niche. If it's a stock driver from when these machines were > set up, then there's actually a risk that it's widespread. Indeed it would. I'm really disappointed this has more or less been ignored since it was identified. (In reply to :armenzg -- back on Monday August 8th from comment #47) > This does not necessarily need a downtime but being very cut and clear when > things happen. You will want a downtime since this will change performance numbers.
I'd like to re-emphasize, that we concluded as part of the investigation here that TP4 does -not- measure page painting times. We concluded it possibly measured something in Chrome painting but that would not explain that more complex pages also have a higher absolute number increase. However the bottom line is we do not, repeat, do not know what extra time is being measured here, nor how the user experience is effected here. Wall clock time isn't as much different as Tp4 times, which is another mystery. Obviously blacklisting the old drivers is a good thing in any case, but we need to acknowledge we do not know if the user experience on this platform is actually adversely affected by D3D9 layers.
Sure, tp4 is not primarily a painting perf benchmark. How does this make it less of an issue? Tp4 measures how long it takes for the load event to occur. This is real time spent somewhere before pages are ready -- neither tp4 nor users care where this time is spent. We should fix tp to also wait for a paint event after the load event, but this won't make the extra time spent before the load event go away.
(In reply to JP Rosevear from comment #39) > I can't speak to the likeliehood, but basically we'd blacklist it other than > for its use on the Talos machines. Updating the driver fixed the perf > issue, see #32. It should be fixed, but it doesn't need to be resolved for > FF6 given current info. I checked how often this driver version occurs: In 4 days of crashes 10 crashes out of 125653 crashes with Nvidia devices had this driver version.
(In reply to Dão Gottwald [:dao] from comment #52) > Sure, tp4 is not primarily a painting perf benchmark. How does this make it > less of an issue? Tp4 measures how long it takes for the load event to > occur. This is real time spent somewhere before pages are ready -- neither > tp4 nor users care where this time is spent. We should fix tp to also wait > for a paint event after the load event, but this won't make the extra time > spent before the load event go away. So, that's just the thing. We don't know, as you say, users don't care where time is spent. So suppose that the extra time in Tp4 is caused by some more complex setup because of using an accelerated layer manager, suppose that adds 80ms to the time before paint, but makes the paint 100ms quicker. It would cause an 80ms regression in Tp4, but a 20ms improvement in user-perceived page load time. Hence my point that a better timing than Tp4 is needed to make a good decision. Additionally onLoad firing behaves more interesting, whether it fires before or after the paint, as I understand it, depends on the timing of numerous things in page loading. Theoretically this means it's even possible subtle timing changes can mean onLoad starts firing after a certain paint, again increasing the time until onLoad fires, but not necessarily the time it takes for the page to display.
Bug 612190 added onpaint timing for browser startup. Does the same reasoning apply here?
(In reply to Emanuel Hoogeveen from comment #55) > Bug 612190 added onpaint timing for browser startup. Does the same reasoning > apply here? It's a little bit tricky, we probably want onLoad to have fired, and then after onLoad the first PaintEvent. But I'm not sure if it's possible for onload to fire -after- the last paint event. But this would require some investigation I think.
I describe what I think we should do in bug 661918, comment 10 in regards to painting and the load event. My suggestion is doable in JS right now.
We deployed last week the newer drivers while I was gone in bug 629759. There were Paint and Txul improvements on m-c, m-i & m-a.
(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #58) > We deployed last week the newer drivers while I was gone in bug 629759. > There were Paint and Txul improvements on m-c, m-i & m-a. Seems like we're missing the expected tp improvement then. Here are the relevant tree-management e-mails: Improvement! Paint decrease 8.64% on XP Mozilla-Inbound ------------------------------------------------------- Previous: avg 127.025 stddev 2.130 of 30 runs up to revision 144add433e72 New : avg 116.052 stddev 0.821 of 5 runs since revision cfb447e2f21f Change : -10.972 (8.64% / z=5.151) Graph : http://mzl.la/qvTT8F Improvement! Paint decrease 8.64% on XP Mozilla-Inbound ------------------------------------------------------- Previous: avg 127.025 stddev 2.130 of 30 runs up to revision 144add433e72 New : avg 116.052 stddev 0.821 of 5 runs since revision cfb447e2f21f Change : -10.972 (8.64% / z=5.151) Graph : http://mzl.la/qvTT8F Improvement! Txul decrease 5.89% on XP Mozilla-Inbound ------------------------------------------------------ Previous: avg 112.675 stddev 0.934 of 30 runs up to revision cfb447e2f21f New : avg 106.042 stddev 1.240 of 5 runs since revision 20c37bc31102 Change : -6.633 (5.89% / z=7.106) Graph : http://mzl.la/piTvIy Improvement! Paint decrease 9.05% on XP Firefox ----------------------------------------------- Previous: avg 126.646 stddev 1.664 of 30 runs up to revision 61b2f8646c38 New : avg 115.189 stddev 0.971 of 5 runs since revision 1dddaeb1366b Change : -11.456 (9.05% / z=6.884) Graph : http://mzl.la/p3MZ8R Regression DHTML increase 1.7% on XP Firefox ----------------------------------------------- Previous: avg 530.760 stddev 3.460 of 30 runs up to revision 1dddaeb1366b New : avg 539.765 stddev 1.677 of 5 runs since revision 1dddaeb1366b Change : +9.005 (1.7% / z=2.602) Graph : http://mzl.la/q2mLk6 Improvement! Txul decrease 6% on XP Mozilla-Aurora -------------------------------------------------- Previous: avg 121.856 stddev 1.228 of 30 runs up to revision 17d82fcd27ad New : avg 114.547 stddev 0.326 of 5 runs since revision e259c21107fb Change : -7.309 (6% / z=5.954) Graph : http://mzl.la/rpZfKH
dao - could you rerun the before and after changesets with the new drivers to get the new delta by chance?
To be clear the DirectX runtime did not get installed.
(In reply to Benoit Girard (:BenWa) from comment #31) > Created attachment 540560 [details] > Results log > > Here are the result logs including the results before and after patching. > > The following were patched to the latest version: > - Windows Updates > - GeForce display drivers > - DirectX 9.0c > > Scores: > With D3D9: 248, 249 > Without D3D9: 251, 251 > > After patching both scores improved. Notice that with D3D9 the score are > faster. (In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #61) > To be clear the DirectX runtime did not get installed. Does this mean that this bug isn't about one particular outdated driver but outdated DirectX?
(In reply to Dão Gottwald [:dao] from comment #59) > Seems like we're missing the expected tp improvement then. > Here are the relevant tree-management e-mails: What performance improvement were we expecting? Are we looking for a magic ~28% improvement? We had previously observed a ~28% regression with tp, but I believe all the scores you posted above are tp5 thus it is no longer valid to compare the results with any data we collected from tp4 paint. I'm going to kick off tp5 runs with D3D9 disabled, as Shawn suggests, so that we have numbers to compare to verify how big the remaining regression is (if at all). (In reply to Dão Gottwald [:dao] from comment #62) > (In reply to Benoit Girard (:BenWa) from comment #31) > Does this mean that this bug isn't about one particular outdated driver but > outdated DirectX? We have seen a significant 8% improvement by applying newer drivers which indicates that this bug was, at least partially, caused by outdated driver. The results of tp5 without D3D9 will give us a hint if patching DirectX/XP is also required.
Thanks BenWa. (In reply to Benoit Girard (:BenWa) from comment #63) > The results of tp5 without D3D9 will give us a hint if patching DirectX/XP > is also required. You can get tp4 coverage on the try server if you want to. Aren't the latest DirectX dlls in the firefox.zip already? I see C:\talos-data\firefox\d3dx9_43.dll being unzipped.
Waiting on tp4 + tp5 results. Triggered with D3D9 Disabled from parent c7ea065539d2: http://tbpl.mozilla.org/?tree=Try&rev=375712f68548 Triggered with D3D9 Enabled from parent c7ea065539d2: http://tbpl.mozilla.org/?tree=Try&rev=e403bce91ac8
(In reply to Shawn Wilsher :sdwilsh from comment #60) > dao - could you rerun the before and after changesets with the new drivers > to get the new delta by chance? http://perf.snarkfest.net/compare-talos/index.html?oldRevs=f7c535bd23c4&newRev=0e157955f298&tests=tp4,tp5&submit=true The 28% regression remains for tp4 on XP -- the driver update apparently didn't make difference. For tp5 I get a 2.7% regression on XP (and Win7, interestingly). I hope someone understand the wildly different characteristics of tp4 and tp5, because this isn't the kind of difference that I'd expect from an updated /representative/ page set. My suspicion would be that the page sets are very random and let us overstate some regressions and miss others.
(In reply to Benoit Girard (:BenWa) from comment #63) > (In reply to Dão Gottwald [:dao] from comment #59) > > Seems like we're missing the expected tp improvement then. > > Here are the relevant tree-management e-mails: > > What performance improvement were we expecting? Are we looking for a magic > ~28% improvement? We had previously observed a ~28% regression with tp, but > I believe all the scores you posted above are tp5 thus it is no longer valid > to compare the results with any data we collected from tp4 paint. What I quoted from comment 31 was still tp4.
(In reply to Dão Gottwald [:dao] from comment #66) > (In reply to Shawn Wilsher :sdwilsh from comment #60) > > dao - could you rerun the before and after changesets with the new drivers > > to get the new delta by chance? > > http://perf.snarkfest.net/compare-talos/index. > html?oldRevs=f7c535bd23c4&newRev=0e157955f298&tests=tp4,tp5&submit=true > > The 28% regression remains for tp4 on XP -- the driver update apparently > didn't make difference. For tp5 I get a 2.7% regression on XP (and Win7, > interestingly). > > I hope someone understand the wildly different characteristics of tp4 and > tp5, because this isn't the kind of difference that I'd expect from an > updated /representative/ page set. My suspicion would be that the page sets > are very random and let us overstate some regressions and miss others. Note that in the details, we can see pages that were also in the original set (spiegel.de/cnn.de/etc.) also didn't regress with Tp5, and did with Tp4 (note that CNN practically didn't change recently, nor did the Spiegel website, so a different website version is unlikely, though not impossible). I restate my original suggestion, we don't really know what we're measuring, and are measuring something different in Tp5. Note that the expected the result is no difference, D3D9 is not really going to help static page render times, composition here takes practically no time compared to essential page rendering work. There might be a small regression with static page rendering times in D3D9 (like we see for Tp5), due to additional content getting active layers to speed up user interaction.
(In reply to Bas Schouten (:bas) from comment #68) > I restate my original suggestion, we don't really know what we're measuring, > and are measuring something different in Tp5. Right, but what does this imply? We can't just ignore Tp, since despite all its known and unknown shortcomings we don't have a better page-load time test. > Note that the expected the result is no difference, D3D9 is not really going > to help static page render times, composition here takes practically no time > compared to essential page rendering work. There might be a small regression > with static page rendering times in D3D9 (like we see for Tp5), due to > additional content getting active layers to speed up user interaction. It's good to hear that we should expect no difference, but... Are you saying that the regression cannot exist because it shouldn't exist? Updating stuff on talos made the regression go away (comment 31), so it seems that there is a real problem, even if Tp4 may be overstating it.
(In reply to Bas Schouten (:bas) from comment #68) > I restate my original suggestion, we don't really know what we're measuring, > and are measuring something different in Tp5. dao's point in comment 40 still stands. This would have been backed out immediately had talos not been completely broken by bug 651659. Just because it's in the tree now does not mean it gets a free pass. This regression needs to be understood. > Note that the expected the result is no difference, D3D9 is not really going > to help static page render times, composition here takes practically no time > compared to essential page rendering work. There might be a small regression > with static page rendering times in D3D9 (like we see for Tp5), due to > additional content getting active layers to speed up user interaction. Note that bug 651659 was about us not drawing browser chrome during page runs. The browser chrome is certainly going to change as a result of pages loading, so this isn't just static content. Can someone please actually look into this instead of argue about why this regression either shouldn't exist or isn't real? Numbers don't lie.
(In reply to Dão Gottwald [:dao] from comment #69) > (In reply to Bas Schouten (:bas) from comment #68) > > I restate my original suggestion, we don't really know what we're measuring, > > and are measuring something different in Tp5. > > Right, but what does this imply? We can't just ignore Tp, since despite all > its known and unknown shortcomings we don't have a better page-load time > test. Right, nobody's advocating is ignoring it. I'm advocating understanding what it measures though. If we understand its shortcomings we can decide whether this measurement is indicative of a real user experience issue. > > > Note that the expected the result is no difference, D3D9 is not really going > > to help static page render times, composition here takes practically no time > > compared to essential page rendering work. There might be a small regression > > with static page rendering times in D3D9 (like we see for Tp5), due to > > additional content getting active layers to speed up user interaction. > > It's good to hear that we should expect no difference, but... Are you saying > that the regression cannot exist because it shouldn't exist? Updating stuff > on talos made the regression go away (comment 31), so it seems that there is > a real problem, even if Tp4 may be overstating it. No. Not at all, if you read more carefully what I'm saying is it's perfectly possible (although I'm not saying this is true!). To have a change that makes Tp4 numbers higher, but page loading times faster. By it reducing the work post-Tp4 measurement, and increasing the workload pre-Tp4 management. Again, this does not -have- to be the case, but as long as we don't understand the measurement, it's possible. (In reply to Shawn Wilsher :sdwilsh from comment #70) > (In reply to Bas Schouten (:bas) from comment #68) > > I restate my original suggestion, we don't really know what we're measuring, > > and are measuring something different in Tp5. > dao's point in comment 40 still stands. This would have been backed out > immediately had talos not been completely broken by bug 651659. Just > because it's in the tree now does not mean it gets a free pass. This > regression needs to be understood. Right, but it wasn't backed out, and now we're shipping a product that we know people have reported no issues with related to page loading performance. We cannot consider a change that could possibly increase page loading times (again, it's perfectly possible, although not necessarily true that although Tp4 goes up, the time to load the page goes down) without understanding it completely. Again, I'm all for the regression being -understood- and then possibly acted upon depending on that understanding. > > > Note that the expected the result is no difference, D3D9 is not really going > > to help static page render times, composition here takes practically no time > > compared to essential page rendering work. There might be a small regression > > with static page rendering times in D3D9 (like we see for Tp5), due to > > additional content getting active layers to speed up user interaction. > Note that bug 651659 was about us not drawing browser chrome during page > runs. The browser chrome is certainly going to change as a result of pages > loading, so this isn't just static content. Right, but the browser chrome should be mostly an 'absolute' time change, not relative to page complexity, possibly the recomposition of the rotating animation on page loading is slowed by D3D9 layers? It's not impossible, since that's a super-tiny part which in software would be very fast, but in hardware entails a little more work (hardware acceleration isn't made to change a 10x10 pixel area). That redraw would occur more often with a complex page possibly, although I wouldn't really expect it to be actively rotating during the Tp4 measurement, but maybe it is. > > Can someone please actually look into this instead of argue about why this > regression either shouldn't exist or isn't real? Numbers don't lie. Right, they don't lie but if what they measure is not understood, what they're saying cannot be interpreted. BenWa looked into this bug and could not reproduce the issue on a local machine with up to date drivers. The fact a driver update didn't fix this regression warrants some additional investigation, but I think that investigation should -first- focus on understanding the following: 1. What does Tp4 measure? 2. What does Tp5 measure - and how does it differ from Tp4? (measurements show that it does, as you say, number don't lie) 3. How can we make the tests measure a number which is more strongly indicative of user perceived page loading times. (I believe Timothy Nikkel had an idea on that in another bug) In between 2 and 3 we need to decide whether activation of D3D9 layers needs reconsideration based on the available data. After 3 we can have a really good idea of whether D3D9 layers creates a significant user experience regression in page loading (at least on this hardware, and we will be able to test other hardware in a meaningful way).
Summary: D3D9 layers is a 28% tp4 hit with outdated drivers on Talos → D3D9 layers is a 28% tp4 hit on Talos
This back-and-forth over whether Tp measures something useful, or how it should be changed to be more useful, isn't really helping this bug. Maybe it can be taken to a newsgroup, or at least another bug? What normally happens when we find out about a perf regression is back out the offending changeset once we figure out what caused the regression. We were pretty sure that D3D9 layers caused a huge regression on Tp4, but in our investigation to prove that, we discovered that an up-to-date driver and DirectX runtime made this regression go away entirely. For no particular reason, it was our belief that the driver update would fix this problem, but it seems that (given comment 66) it's the combination of driver update and DirectX runtime update that fixes it. DirectX runtime updates are automatically pushed out through Windows Update, so users should be OK for the most part - it's our artificial test environment that causes these issues. Even given the fact that the number of users affected by this issue is likely vanishingly small, I think we were too hesitant to turn off layers acceleration on D3D9, at least on mozilla-central. I maintain that we'd probably go orange - see comment 4 - but we should have at least started investigations. I'm going to make sure that sort of oversight doesn't happen again. In the short term, Benoit Girard is going to check out a test slave to ensure that updating the DirectX runtime through Windows Update is all that's necessary to restore the performance numbers, as seen in comment 31. If that's the case, we can close out this bug in favour of updating the DirectX runtime on all our test slaves. If that isn't the case, we'll start looking at turning off D3D9 and continue investigation.
(In reply to Joe Drew (:JOEDREW!) from comment #72) > DirectX runtime > updates are automatically pushed out through Windows Update, so users should > be OK for the most part - it's our artificial test environment that causes > these issues. Are you sure about this? I've never seen the DirectX runtimes as a critical or even an optional update on any of my systems. I think Microsoft rely on games developers to ship their games with a copy of the runtimes they need - and Firefox also ships with d3dx9_43.dll and D3DCompiler_43.dll as users don't always have them.
(In reply to Emanuel Hoogeveen from comment #73) > Are you sure about this? I've never seen the DirectX runtimes as a critical > or even an optional update on any of my systems. I think Microsoft rely on > games developers to ship their games with a copy of the runtimes they need - > and Firefox also ships with d3dx9_43.dll and D3DCompiler_43.dll as users > don't always have them. I'm talking about DirectX 9.0c as mentioned in comment 31. The d3dx and D3DCompiler dlls are separate functionality. You might be right, though Microsoft says on http://www.microsoft.com/download/en/details.aspx?id=35 : "Note that the DirectX Runtime (Direct3D, DirectInput, DirectSound) is not part of this package as it is included as part of the Windows operating system, and therefore cannot be installed or uninstalled. Updating the DirectX Runtime is achieved by installing the latest Service Pack or obtaining a newer version of Windows."
For any testing lets use the full offline package as the build network does not have access to most of the internetz: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8109 If anyone points me to an existing build (a try changeset works) to be tested against I can trigger a staging run with the aforementioned DirectX update. Is the June 2010 update the one we want to test?
(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #75) > If anyone points me to an existing build (a try changeset works) to be > tested against I can trigger a staging run with the aforementioned DirectX > update. mozilla-central tip should work for this.
Depends on: 679066
FTR we are running XP SP2 and the current DirectX version on the talos-r3-xp slaves is DirectX 9.0c (4.09.0000.0904 - a.k.a 4.09.00.0904) After I install "DirectX End-User Runtimes (June 2010)" I don't see any version changes on dxdiag. BenWa has triggered a try job to be sure that the driver change is really being picked up.
(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #77) > FTR we are running XP SP2 and the current DirectX version on the talos-r3-xp > slaves is DirectX 9.0c (4.09.0000.0904 - a.k.a 4.09.00.0904) Something here doesn't add up, according to wikipedia SP2 should be 0903 not 0904. I triggered a try job. This should confirm that the newer driver is in fact picked up: http://tbpl.mozilla.org/?tree=Try&rev=a13a4c89aa3e
As mentioned in IRC and IIUC it should add up since SP2 should contain the version I mentioned. 4.09.00.0904 Windows XP SP2, SP3*, Windows Server 2003 SP1, Windows Server 2003 R2 and Xbox 360 August 6, 2004 / April 21, 2008* [1] http://en.wikipedia.org/wiki/DirectX
Ohh right, I read that incorrectly. You are correct.
Just to be clear, the DirectX runtimes do not contain the core d3d9.dll. They contain (found by extracting the offline package): D3DCompiler_n.dll where n goes from 33 to 43, d3dcsx_n.dll where n goes from 42 to 43, d3dx9_n.dll where n goes from 24 to 43, d3dx10.dll, d3dx10_n.dll where n goes from 33 to 43, various microsoft.directx*.dll files, x3daudio1_n.dll where n goes from 0 to 7, xactengine2_n.dll where n goes from 0 to 10, xactengine3_n.dll where n goes from 0 to 7, XAPOFX1_n.dll where n goes from 0 to 5, XAudio2_n.dll where n goes fro 0 to 7, xinput1_n.dll where n goes from 1 to 3, and xinput9_1_0.dll
Just a quick update. I've got a talos machine with standalone on it. I'm starting to run more throughout test that should isolate exactly which patch needs to be applied to fix the regression.
I'm trying to find out why with standalone talos I get the following numbers: D3D9 on: 294 D3D9 off: 287 I'm expecting to get numbers that reproduce a 28% performance hit like we do on Try.
(In reply to Benoit Girard (:BenWa) from comment #83) > I'm trying to find out why with standalone talos I get the following numbers: > D3D9 on: 294 > D3D9 off: 287 > > I'm expecting to get numbers that reproduce a 28% performance hit like we do > on Try. Is that over multiple runs? Otherwise you should do some more probably, the variation in Tp4 is far higher than 7 ms I believe.
Yes it's over multiple run. I've only ever seen variations of +-1 on talos on all the test I've ever ran. Perhaps the standard tp4 variation we see on try is because we're hitting different machines. In my case I'm running my tests on the same machine.
We could retrigger the same job multiple times on try which would make several different slaves to run tp and see how close or far the numbers are from each other.
For comment 0 as well as comment 66, I always triggered multiple re-runs.
I've corrected my setup with anode's help (needed to add plugins.zip to the profile + set resolution to 1280x1024). The numbers I'm seeing on a stock Talos are now more sensible: tp4 D3D9 on: 275 tp4 D3D9 off: 261 5% difference. The numbers with D3D9 off match what we get on try, however the numbers with D3D9 on does not match what we see on try.
We are still working on this. I've managed to reproduce the regression intermittently using the same tp4.manifest used by talos. I've tried bisecting the differences but because its intermittent I haven't been able to get anything conclusive. I will discuss this issue with others on how to proceed to isolate the problem. I don't understand why the issue isn't intermittent on talos.
Any ETA on when you could get back to this? We had a newer Flash deployed this morning so the numbers will be now different (plugins.zip refresh. Should we reevaluate what we want to accomplish? (considering tp4 is phased out and we have a newer plugins.zip)
BenWa any ideas on how to proceed? Anyone you can talk to determine what is the way forward?
I can try what you suggested in Comment 90. At this point I don't have a clear path foward as my attention is on OGL Layers at the moment.
(In reply to Benoit Girard (:BenWa) from comment #92) > I can try what you suggested in Comment 90. At this point I don't have a > clear path foward as my attention is on OGL Layers at the moment. I read this as "we'd rather work on new things than worry about an old regression". Please tell me I'm reading this incorrectly?
(In reply to Shawn Wilsher :sdwilsh from comment #93) > (In reply to Benoit Girard (:BenWa) from comment #92) > > I can try what you suggested in Comment 90. At this point I don't have a > > clear path foward as my attention is on OGL Layers at the moment. > I read this as "we'd rather work on new things than worry about an old > regression". Please tell me I'm reading this incorrectly? It should be read as I wrote it. In any case this regression is important to the GFX team and I have not forgotten about it. The fact that I have not been able to reproduce the regression outside of talos means that I have not dropped working on bugs that clearly do effect users.
Try run for 71e4ffb02f08 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=71e4ffb02f08 Results (out of 17 total builds): success: 17 Builds available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/b56girard@gmail.com-71e4ffb02f08
:BenWa - its been a few months since last update, any progress?
No I haven't had any time. A high GFX priority item is to investigate cases were hardware acceleration makes things worse (often on low end graphics hardware), I'm hoping that it will help us make progress on this issue. We have been testing hardware acceleration on very low end hardware and can reproduce regression. I expect these issues to be worked on when the compositor feature is no longer blocking mobile release.
Just a correction the hardware acceleration being slower is only with D2D, D3D9 apparently still makes an improvements. In any case we will still revisit this bug.
Should we consider disabling accelerated layers on Windows XP? I don't think slowing down the common case of "most web pages" in order to have decent performance on some advanced HTML5-type pages is the right trade-off.
We've discussed in the bug that we can't reproduce this bug outside of our test infrastructure and that we can't find evidence to show that users are affected with this regression. Layers acceleration is important for typical web pages such as gmail and facebook.
Maybe it's just a placebo effect, but when I turn off accel. layers on my Windows XP box, sites seem to display faster for me. Is there a plan to try to measure this outside of our testing infrastructure? Could we do an AB test pilot study asking users to tell us about their perceived performance or something like that? Windows XP is a huge portion of our users and we should be putting as much or more effort into getting things as good as they can be for winXP as we do for our other tier 1 platforms.
That would be great, is someone free to perform the study?
(In reply to Benoit Girard (:BenWa) from comment #102) > That would be great, is someone free to perform the study? I'm looking into it now. If so, it won't be in the next few weeks because there are higher priority items on the UR roadmap.
(In reply to Asa Dotzler [:asa] from comment #103) > (In reply to Benoit Girard (:BenWa) from comment #102) > > That would be great, is someone free to perform the study? > > I'm looking into it now. If so, it won't be in the next few weeks because > there are higher priority items on the UR roadmap. Did anything ever come of this?
Flags: needinfo?(asa)
I'm closing this bug as incomplete due to lack of response. Please reopen if this bug is still relevant.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → INCOMPLETE
Flags: needinfo?(asa)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: