Closed Bug 1371211 Opened 7 years ago Closed 6 years ago

investigate windows 10 config to understand per differences compared to windows 8

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jmaher, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

it appears that changing an operating system from windows 8 to windows 10 has presented some noisier data as well as less desirable numbers.  While this is not a regression in the sense that our firefox code changed it would be good to know if we are measuring something useful as compared to what our end users are seeing.
for reference bug 1358306 is where we worked on getting windows 10 stood up, there are attachments there for about:support.

I have filed a bug to get loaners so I can get information.
When I get the loaners I will get the graphics card version and post it here.
I'm attaching this bug to the Quantum release criteria bug so I don't lose it. Whether or why Windows 10 is slower than Windows 8 doesn't block Quantum, but it does affect how we interpret Talos results for the Quantum release criteria.
Blocks: QRC_FX57
this is the windows 8 about:support.

if I am reading this correctly, I see:
"adapterDescription": "NVIDIA GeForce GT 610 ",
"driverVersion": "9.18.13.3523",
for windows 10 (https://bug1358306.bmoattachments.org/attachment.cgi?id=8870492 ):
"adapterDescription": "NVIDIA GeForce GT 610 ",
"driverVersion": "9.18.13.3523",


this looks to be the same driver for both systems; maybe the old driver doesn't work well on windows 10?
as a note, the win10 stuff is from old data, I need to figure out how to get vnc to work on this loaner.
I am still waiting on releng to get a win10 machine I can access and win8 machine I can get admin access to.
The driver version reported in windows on ix machines is:

21.21.13.7878
2/23/2017
The nvidia graphics driver for w10 got updated per the request from PI in https://bugzilla.mozilla.org/show_bug.cgi?id=1358307#c16
win8: 9.18.13.3523
win10: 21.21.13.7878

:cpeterson- we have graphics info, the drivers are different, are there other things we should look at between the machines to help understand the differences in numbers?
Milan, can a gfx engineer take a look at this bug info to see if the different Nvidia driver versions might explain why Talos is so much slower on win10 than win8 on the same hardware? Are there other win10 graphics or DWM changes that might have caused this?

When Talos upgraded from win8 to win10, there was a clear discontinuity in the results for many tests, but particularly for tp5 [1] and tsvg_static [2]:

[1] https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-central,b68e2b084272409d7def3928a55baf0e00f3888a,1,1%5D&series=%5Bmozilla-central,ac46ba40f08bbbf209a6c34b8c054393bf222e67,1,1%5D&series=%5Bmozilla-central,c00763b23b39207671b795a12ba29d38ddc17f06,1,1%5D

[2] https://treeherder.mozilla.org/perf.html#/graphs?timerange=5184000&series=%5Bmozilla-central,397a484349ec684142dc3b3dab8f882a5d54bc8b,1,1%5D&series=%5Bmozilla-central,18cf40355e5b20164ab9307f83dd6d6eb6184aa8,1,1%5D&series=%5Bmozilla-central,e4e0081ff90530932c463fc917d113936690baa3,1,1%5D
Flags: needinfo?(milan)
(In reply to Chris Peterson [:cpeterson] from comment #11)
> Milan, can a gfx engineer take a look at this bug info to see if the
> different Nvidia driver versions might explain why Talos is so much slower
> on win10 than win8 on the same hardware? Are there other win10 graphics or
> DWM changes that might have caused this?

There were a lot of changes between Win 8 and Win 10 :)

We can see if updating Windows 8 drivers from 335.23 to 378.78 changes the results, but I have a feeling that it's probably a wider system issue.

Can we capture profiles on the two systems for a (random) test that shows the worst regression?
Flags: needinfo?(milan)
:mconley- do you have some docs for getting profiling to work from talos?  I know it is problematic and you got it working on try server recently.
Flags: needinfo?(mconley)
(In reply to Joel Maher ( :jmaher) from comment #13)
> :mconley- do you have some docs for getting profiling to work from talos?  I
> know it is problematic and you got it working on try server recently.

I _believe_ the main problem is that you have to request more than one rebuild with --rebuild-talos > 1 (bug 1370907). The other thing is that a number of the Talos tests are using the old Profiler.js script, which means that they'll sometimes only report a single processes samples, which isn't always what you want.

Also, sometimes symbolication just kinda times out in automation. Sometimes, when you can get profiles from Talos, it's kind of a miracle.

cpeterson, you mentioned tp5 and tsvgx - were there other tests in that list that showed a wide difference between Win 8 and Win 10? I ask because some might be easier than others to get profiles from.
Flags: needinfo?(mconley) → needinfo?(cpeterson)
(In reply to Mike Conley (:mconley) from comment #14)
> cpeterson, you mentioned tp5 and tsvgx - were there other tests in that list
> that showed a wide difference between Win 8 and Win 10? I ask because some
> might be easier than others to get profiles from.

There are discontinuities in most of the other Talos tests, including ts_paint, tpaint, and tart as shown on the dashboards below:

https://health.graphics/quantum/
Flags: needinfo?(cpeterson)
Okay. ts_paint was made to work with profiling very recently in bug 1372361. That's one we can / should gather a profile from from both types of machines for comparison.
windows 8 is not easily accessible anymore- lets move forward.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: