1485757 - Support xperf on x64

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[(No longer employed by Mozilla) Aaron Klotz]

We should get the xperf suite up and running on x64.

The latest version of the Windows Performance Toolkit is available as part of this package:
https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install

(Note that I have been running xtalos against local x64 builds without difficulty, so this should not be too difficult to stand up)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:grenade, the ADK has the xperf tool that we want, I found that the ADK has many other things we do not need (otherwise it would be 5-6GB), instead we need about 100MB of tools/libraries- how can we install this on the test image that we use in AWS?

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

one option would be to create a zip package and use tooltool to just unzip the bits needed at the beginning of the test run.

if we wanted to do it in the occ run, we'd probably have to install the whole adk (unless there are some command line installer options to just install the components we need)

if you want me to go the route of installing the whole adk with occ, it's actually pretty simple to do and i could write and test a patch over the next few days.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I imagine this does more than just copy files since xperf can hook into the core OS- I think the ADK would be interesting- let me look at the option of downloading it to install offline on other computers and see what comes up

Comment 4

[(No longer employed by Mozilla) Aaron Klotz]

Bruce Dawson at Google has managed to extract WPT into its own installer as part of his UIforETW project (see https://github.com/google/UIforETW/releases).

The WPT installer inside the UIforETW ZIP is still signed by Microsoft, he just managed to extract it from the ADK. He also has UIforETW automatically installing WPT as necessary.

Maybe we can use his work as an inspiration?

Comment 5

[(No longer employed by Mozilla) Aaron Klotz]

Oh yeah, it's just an MSI package, so we can probably just trigger a silent install using msiexec.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I installed the WPTx64-x86_en-us.msi from https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install (which is also easy to find in the link from comment 4!, download/unzip/cd "third_party\wpt10")

Doing this I was able to run xperf with our commands to successfully record using the same cli arguments we use for win7 xperf.exe

I installed with |msiexec /i WPTx64-x86_en-us.msi /quiet /qn| from an administrator prompt and it installed silently.

:grenade, is this something you can pick up and run with as it fits into your schedule?

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: https://github.com/mozilla-releng/OpenCloudConfig/pull/170

ok. we're ready to go. i just need a green light/review on the occ pr.
https://github.com/mozilla-releng/OpenCloudConfig/pull/170

i used the WPTx64-x86_en-us.msi extracted from the latest UI for ETW release (https://github.com/google/UIforETW/releases/download/v1.49/etwpackage1.49.zip).

i ran the same install against the win 10 beta workers (gecko-t-win10-64-beta, gecko-t-win10-64-gpu-b) where the install succeeded and xperf was installed to "C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\xperf.exe"

here's a task on a beta worker configured to run `where xperf` and `xperf -help` when workers come online (there's some problem with the tc queue atm):
https://tools.taskcluster.net/groups/JoFmu4xFR02-_CmPunIgDw/tasks/JoFmu4xFR02-_CmPunIgDw/details

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9004493 [details] [diff] [review]
https://github.com/mozilla-releng/OpenCloudConfig/pull/170

Review of attachment 9004493 [details] [diff] [review]:
-----------------------------------------------------------------

awesome

Comment 9

[Pete Moore [:pmoore][:pete]]

Joel,

To run tasks as administrator on win10, you'll need to add

  "runAsAdministrator": true

to task payload.features section.

This isn't needed on win7 as UAC is disabled on win7, but I believe on win10, UAC is enabled.

For more information, search for "runAsAdministrator" on page https://docs.taskcluster.net/docs/reference/workers/generic-worker/docs/payload

Good luck!

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

thanks Pete, I got this going:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6eac8159d5dca093cbdd56b0b15f0d06060bb49a&selectedJob=196255449

There are 4 next steps that I see:
1) getting proper scopes on the job automatically:
  - 'generic-worker:os-group:aws-provisioner-v1/gecko-t-win10-64/Administrators'
  - 'generic-worker:run-as-administrator:aws-provisioner-v1/gecko-t-win10-64'
2) install the mozprofilerprobe.mof file on the core image as referenced in bug 1422018
3) determine how to specify the xperf path in a config file so we can support both win7/win10
4) green up results- we need to determine if we have win7 and win10 unique files, or just a big file that is a join of both configs

:grenade, can you help with step #2 ?

Comment 11

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: https://github.com/mozilla-releng/OpenCloudConfig/pull/171

this pr downloads http://hg.mozilla.org/mozilla-central/raw-file/360ab7771e27/toolkit/components/startup/mozprofilerprobe.mof to C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\mozprofilerprobe.mof
and then runs `mofcomp "C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\mozprofilerprobe.mof"`

i don't know anything about what the mof file or the mofcomp command do, but i tested this on the win 10 beta workers and the command completed without errors.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9004888 [details] [diff] [review]
https://github.com/mozilla-releng/OpenCloudConfig/pull/171

Review of attachment 9004888 [details] [diff] [review]:
-----------------------------------------------------------------

I assume we need to change the componenttype values, but it could be there is a dependency ordering that I am not aware of.

::: userdata/Manifest/gecko-t-win10-64-cu.json
@@ +1400,5 @@
> +        "\"C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\mozprofilerprobe.mof\""
> +      ],
> +      "DependsOn": [
> +        {
> +          "ComponentType": "ChecksumFileDownload",

you mixed up the component type here, this should be msiInstall, not the prevous item.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

the command has not output- it is one of those stealth commands.

Comment 14

[Rob Thijssen [:grenade (EET/UTC+0300)]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #12)
> I assume we need to change the componenttype values, but it could be there
> is a dependency ordering that I am not aware of.
...
> you mixed up the component type here, this should be msiInstall, not the
> prevous item.

yeah, the dependencies are intentional as they are.

- the download depends on the msi install because that component creates the directory we download into
- the mofcomp command depends on the download because it needs the downloaded file

Comment 15

[Rob Thijssen [:grenade (EET/UTC+0300)]]

merged:
https://github.com/mozilla-releng/OpenCloudConfig/commit/20dd55f14327742c322e9f0988e9405eaedd3602

Comment 16

[Rob Thijssen [:grenade (EET/UTC+0300)]]

feel free to reopen if i've missed something

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

I saw this work on a try push from last week, but failed on a fresh try push today:
https://taskcluster-artifacts.net/S6A3UldWSzuiHtG6y0bR3Q/0/public/logs/live_backing.log

I am not sure if this didn't get rolled out, or was rolled out in a limited fashion?

Comment 18

[Rob Thijssen [:grenade (EET/UTC+0300)]]

xperf is rolled out on all win 10 workers. i wonder if the error relates to something not working rather than xperf not being installed. here's the error message:

18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: NT Kernel Logger: The instance name passed was not recognized as valid by a WMI data provider. (0x1069).
18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: talos_ses: The instance name passed was not recognized as valid by a WMI data provider. (0x1069).
18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: test.etl: The system cannot find the file specified. (0x2).
18:10:49     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: test.etl: The system cannot find the file specified. (0x80070002).
18:10:49     INFO -  executing '"c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf.exe" -stop -stop talos_ses'
18:10:49     INFO -  reading etl filename: test.etl
18:10:49     INFO -  etlparser: in readfile: test.etl.csv
18:10:49    ERROR -  Traceback (most recent call last):
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 115, in <module>
18:10:49     INFO -      main()
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 109, in main
18:10:49     INFO -      **args.__dict__)
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 93, in stop_from_config
18:10:49     INFO -      run_session_restore_analysis(csv_filename=csv_base, debug=debug, **kwargs)
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 39, in run_session_restore_analysis
18:10:49     INFO -      output += "%.3f\n" % (interval.get_results()[XPerfAttribute.RESULT])
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\xtalos\xperf_analyzer.py", line 253, in get_results
18:10:49     INFO -      end = self.seen_evtlist[-1]
18:10:49     INFO -  IndexError: list index out of range
18:10:50     INFO -  extending with xperf!
18:10:50     INFO -  TEST-UNEXPECTED-ERROR | tp5n | [Errno 2] No such file or directory: 'etl_output_session_restore_stats.csv'
18:10:50    ERROR -  Traceback (most recent call last):
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\run_tests.py", line 299, in run_tests
18:10:50     INFO -      talos_results.add(mytest.runTest(browser_config, test))
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\ttest.py", line 63, in runTest
18:10:50     INFO -      return self._runTest(browser_config, test_config, setup)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\ttest.py", line 275, in _runTest
18:10:50     INFO -      else None)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 95, in add
18:10:50     INFO -      global_counters=self.global_counters
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 333, in __init__
18:10:50     INFO -      self.counters(self.counter_results, self.global_counters)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 413, in counters
18:10:50     INFO -      self.xperf(global_counters)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 502, in xperf
18:10:50     INFO -      with open(filename, 'r') as contents:
18:10:50     INFO -  IOError: [Errno 2] No such file or directory: 'etl_output_session_restore_stats.csv'
18:10:50     INFO -  TEST-INFO took 129865ms
18:10:50     INFO -  SUITE-END | took 129s
18:10:50    ERROR - Return code: 2

Comment 19

[Rob Thijssen [:grenade (EET/UTC+0300)]]

here's a task that shows xperf is installed:
https://tools.taskcluster.net/groups/U3QybOA0Qfa9uRDfWDBumQ/tasks/U3QybOA0Qfa9uRDfWDBumQ/runs/0/logs/public%2Flogs%2Flive.log

Comment 20

[(No longer employed by Mozilla) Aaron Klotz]

> 18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance
> Toolkit/xperf: error: NT Kernel Logger: The instance name passed was not
> recognized as valid by a WMI data provider. (0x1069).

Is often seen when the mof file was not successfully installed.

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

yeah, I asked for the wrong thing, the .mof file in reference to the comments discussing in comment 14.

Comment 22

[Rob Thijssen [:grenade (EET/UTC+0300)]]

ok. the mof file is present on the instances
(see https://tools.taskcluster.net/groups/PFLgDi6tQh6toTqTfuqSAg/tasks/PFLgDi6tQh6toTqTfuqSAg/runs/0/logs/public%2Flogs%2Flive.log).

papertrail shows no errors when running the mofcomp command
(see https://papertrailapp.com/groups/3538333/events?q=[[Script]CommandRun_mofcomp_mozprofilerprobe)

is there something else i should check for?

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

hmm, looking more I see access denied errors which I didn't see last week- possibly those are the real problems.

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=7820f2bbdc8829d7e36c90f8a8917fc1325eb570

:pmoore, did anything change with scopes and run-as-administrator.

Comment 24

[Rob Thijssen [:grenade (EET/UTC+0300)]]

just a thought:

does the mofcomp command need to be run in the context of the task user (rather than as the system user, during system setup)?

i'm wondering if mofcomp output goes to AppData or some user specific directory. if so, the command will need to be run as part of the task (after the task user is created, maybe in preflight scripts) rather than during setup.

i'm just guessing here. i have no knowledge of mofcomp or xperf beyond the installation steps performed earlier.

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

I believe mofcomp is needed to run as administrator vs regular user.  But if the user is an admin, maybe there is something there.  This did work last Thursday, but I will scour logs and configs to see if something else changed.

Comment 26

[(No longer employed by Mozilla) Aaron Klotz]

This is the output that I get when running
> mofcomp c:\path\to\mozprofilerprobe.mof

Microsoft (R) MOF Compiler Version 10.0.17134.1
Copyright (c) Microsoft Corp. 1997-2006. All rights reserved.
Parsing MOF file: mozprofilerprobe.mof
MOF file has been successfully parsed
Storing data in the repository...
Done!

Comment 27

[Pete Moore [:pmoore][:pete]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #23)
> hmm, looking more I see access denied errors which I didn't see last week-
> possibly those are the real problems.
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.
> com&fromchange=7820f2bbdc8829d7e36c90f8a8917fc1325eb570
> 
> :pmoore, did anything change with scopes and run-as-administrator.

No, but I see from the task definition here, that the step explained in comment 9 has not been applied:

https://tools.taskcluster.net/groups/FmmdjrliQkmDY8EH7qHyVw/tasks/Vncxza1aSiy9xnglhpfk1g/details

Comment 28

[Joel Maher ( :jmaher ) (UTC -8)]

got it, I fixed that in edit/retrigger and things worked- now let me fix up my patch

Comment 29

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

we want to run xperf on windows 10 and need administrator access.

Comment 30

[Joel Maher ( :jmaher ) (UTC -8)]

here is a try push showing it working:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a9980f5a7ce1308f3403f4126aa2ab0b0355d94

the try push has another patch for the talos changes needed...one step at a time.

Comment 31

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

there is a phabricator request, but not a bugzilla request

Comment 32

[Pete Moore [:pmoore][:pete]]

Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

Pete Moore [:pmoore][:pete] has approved the revision.

Comment 33

[Pete Moore [:pmoore][:pete]]

just a couple of nits and questions in the review...

Comment 34

[Pete Moore [:pmoore][:pete]]

Joel, you should probably also get a review from someone else too that knows this code better than me. Perhaps dustin/ahal/Callek.

Comment 35

[Joel Maher ( :jmaher ) (UTC -8)]

ok, asking :dustin for review after addressing the nits!

Comment 36

[Pete Moore [:pmoore][:pete]]

Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

Pete Moore [:pmoore][:pete] has been removed from the revision.

Comment 37

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

Dustin J. Mitchell [:dustin] pronoun: he has approved the revision.

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a7876555f4d4
add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=dustin

Comment 39

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/a7876555f4d4

Comment 40

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/cbbcaf071839