Closed Bug 1485757 Opened Last year Closed Last year

Support xperf on x64

Categories

(Testing :: Talos, enhancement)

x86_64
Windows
enhancement
Not set

Tracking

(firefox-esr60 fixed, firefox64 fixed)

RESOLVED FIXED
mozilla64
Tracking Status
firefox-esr60 --- fixed
firefox64 --- fixed

People

(Reporter: aklotz, Assigned: jmaher)

References

Details

Attachments

(3 files)

We should get the xperf suite up and running on x64.

The latest version of the Windows Performance Toolkit is available as part of this package:
https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install

(Note that I have been running xtalos against local x64 builds without difficulty, so this should not be too difficult to stand up)
Blocks: 1463483
:grenade, the ADK has the xperf tool that we want, I found that the ADK has many other things we do not need (otherwise it would be 5-6GB), instead we need about 100MB of tools/libraries- how can we install this on the test image that we use in AWS?
Flags: needinfo?(rthijssen)
one option would be to create a zip package and use tooltool to just unzip the bits needed at the beginning of the test run.

if we wanted to do it in the occ run, we'd probably have to install the whole adk (unless there are some command line installer options to just install the components we need)

if you want me to go the route of installing the whole adk with occ, it's actually pretty simple to do and i could write and test a patch over the next few days.
Flags: needinfo?(rthijssen)
I imagine this does more than just copy files since xperf can hook into the core OS- I think the ADK would be interesting- let me look at the option of downloading it to install offline on other computers and see what comes up
Bruce Dawson at Google has managed to extract WPT into its own installer as part of his UIforETW project (see https://github.com/google/UIforETW/releases).

The WPT installer inside the UIforETW ZIP is still signed by Microsoft, he just managed to extract it from the ADK. He also has UIforETW automatically installing WPT as necessary.

Maybe we can use his work as an inspiration?
Oh yeah, it's just an MSI package, so we can probably just trigger a silent install using msiexec.
I installed the WPTx64-x86_en-us.msi from https://docs.microsoft.com/en-us/windows-hardware/get-started/adk-install (which is also easy to find in the link from comment 4!, download/unzip/cd "third_party\wpt10")

Doing this I was able to run xperf with our commands to successfully record using the same cli arguments we use for win7 xperf.exe

I installed with |msiexec /i WPTx64-x86_en-us.msi /quiet /qn| from an administrator prompt and it installed silently.

:grenade, is this something you can pick up and run with as it fits into your schedule?
Flags: needinfo?(rthijssen)
ok. we're ready to go. i just need a green light/review on the occ pr.
https://github.com/mozilla-releng/OpenCloudConfig/pull/170

i used the WPTx64-x86_en-us.msi extracted from the latest UI for ETW release (https://github.com/google/UIforETW/releases/download/v1.49/etwpackage1.49.zip).

i ran the same install against the win 10 beta workers (gecko-t-win10-64-beta, gecko-t-win10-64-gpu-b) where the install succeeded and xperf was installed to "C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\xperf.exe"

here's a task on a beta worker configured to run `where xperf` and `xperf -help` when workers come online (there's some problem with the tc queue atm):
https://tools.taskcluster.net/groups/JoFmu4xFR02-_CmPunIgDw/tasks/JoFmu4xFR02-_CmPunIgDw/details
Assignee: nobody → rthijssen
Status: NEW → ASSIGNED
Flags: needinfo?(rthijssen)
Attachment #9004493 - Flags: review?(jmaher)
Comment on attachment 9004493 [details] [diff] [review]
https://github.com/mozilla-releng/OpenCloudConfig/pull/170

Review of attachment 9004493 [details] [diff] [review]:
-----------------------------------------------------------------

awesome
Attachment #9004493 - Attachment is patch: true
Attachment #9004493 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #9004493 - Flags: review?(jmaher) → review+
Joel,

To run tasks as administrator on win10, you'll need to add

  "runAsAdministrator": true

to task payload.features section.

This isn't needed on win7 as UAC is disabled on win7, but I believe on win10, UAC is enabled.

For more information, search for "runAsAdministrator" on page https://docs.taskcluster.net/docs/reference/workers/generic-worker/docs/payload

Good luck!
Flags: needinfo?(jmaher)
thanks Pete, I got this going:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6eac8159d5dca093cbdd56b0b15f0d06060bb49a&selectedJob=196255449

There are 4 next steps that I see:
1) getting proper scopes on the job automatically:
  - 'generic-worker:os-group:aws-provisioner-v1/gecko-t-win10-64/Administrators'
  - 'generic-worker:run-as-administrator:aws-provisioner-v1/gecko-t-win10-64'
2) install the mozprofilerprobe.mof file on the core image as referenced in bug 1422018
3) determine how to specify the xperf path in a config file so we can support both win7/win10
4) green up results- we need to determine if we have win7 and win10 unique files, or just a big file that is a join of both configs

:grenade, can you help with step #2 ?
Flags: needinfo?(jmaher) → needinfo?(rthijssen)
this pr downloads http://hg.mozilla.org/mozilla-central/raw-file/360ab7771e27/toolkit/components/startup/mozprofilerprobe.mof to C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\mozprofilerprobe.mof
and then runs `mofcomp "C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\mozprofilerprobe.mof"`

i don't know anything about what the mof file or the mofcomp command do, but i tested this on the win 10 beta workers and the command completed without errors.
Flags: needinfo?(rthijssen)
Attachment #9004888 - Flags: review?(jmaher)
Comment on attachment 9004888 [details] [diff] [review]
https://github.com/mozilla-releng/OpenCloudConfig/pull/171

Review of attachment 9004888 [details] [diff] [review]:
-----------------------------------------------------------------

I assume we need to change the componenttype values, but it could be there is a dependency ordering that I am not aware of.

::: userdata/Manifest/gecko-t-win10-64-cu.json
@@ +1400,5 @@
> +        "\"C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\mozprofilerprobe.mof\""
> +      ],
> +      "DependsOn": [
> +        {
> +          "ComponentType": "ChecksumFileDownload",

you mixed up the component type here, this should be msiInstall, not the prevous item.
Attachment #9004888 - Attachment is patch: true
Attachment #9004888 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #9004888 - Flags: review?(jmaher) → review+
the command has not output- it is one of those stealth commands.
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #12)
> I assume we need to change the componenttype values, but it could be there
> is a dependency ordering that I am not aware of.
...
> you mixed up the component type here, this should be msiInstall, not the
> prevous item.

yeah, the dependencies are intentional as they are.

- the download depends on the msi install because that component creates the directory we download into
- the mofcomp command depends on the download because it needs the downloaded file
feel free to reopen if i've missed something
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
I saw this work on a try push from last week, but failed on a fresh try push today:
https://taskcluster-artifacts.net/S6A3UldWSzuiHtG6y0bR3Q/0/public/logs/live_backing.log

I am not sure if this didn't get rolled out, or was rolled out in a limited fashion?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
xperf is rolled out on all win 10 workers. i wonder if the error relates to something not working rather than xperf not being installed. here's the error message:

18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: NT Kernel Logger: The instance name passed was not recognized as valid by a WMI data provider. (0x1069).
18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: talos_ses: The instance name passed was not recognized as valid by a WMI data provider. (0x1069).
18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: test.etl: The system cannot find the file specified. (0x2).
18:10:49     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf: error: test.etl: The system cannot find the file specified. (0x80070002).
18:10:49     INFO -  executing '"c:/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/xperf.exe" -stop -stop talos_ses'
18:10:49     INFO -  reading etl filename: test.etl
18:10:49     INFO -  etlparser: in readfile: test.etl.csv
18:10:49    ERROR -  Traceback (most recent call last):
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 115, in <module>
18:10:49     INFO -      main()
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 109, in main
18:10:49     INFO -      **args.__dict__)
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 93, in stop_from_config
18:10:49     INFO -      run_session_restore_analysis(csv_filename=csv_base, debug=debug, **kwargs)
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos/xtalos/parse_xperf.py", line 39, in run_session_restore_analysis
18:10:49     INFO -      output += "%.3f\n" % (interval.get_results()[XPerfAttribute.RESULT])
18:10:49     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\xtalos\xperf_analyzer.py", line 253, in get_results
18:10:49     INFO -      end = self.seen_evtlist[-1]
18:10:49     INFO -  IndexError: list index out of range
18:10:50     INFO -  extending with xperf!
18:10:50     INFO -  TEST-UNEXPECTED-ERROR | tp5n | [Errno 2] No such file or directory: 'etl_output_session_restore_stats.csv'
18:10:50    ERROR -  Traceback (most recent call last):
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\run_tests.py", line 299, in run_tests
18:10:50     INFO -      talos_results.add(mytest.runTest(browser_config, test))
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\ttest.py", line 63, in runTest
18:10:50     INFO -      return self._runTest(browser_config, test_config, setup)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\ttest.py", line 275, in _runTest
18:10:50     INFO -      else None)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 95, in add
18:10:50     INFO -      global_counters=self.global_counters
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 333, in __init__
18:10:50     INFO -      self.counters(self.counter_results, self.global_counters)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 413, in counters
18:10:50     INFO -      self.xperf(global_counters)
18:10:50     INFO -    File "Z:\task_1536083824\build\tests\talos\talos\results.py", line 502, in xperf
18:10:50     INFO -      with open(filename, 'r') as contents:
18:10:50     INFO -  IOError: [Errno 2] No such file or directory: 'etl_output_session_restore_stats.csv'
18:10:50     INFO -  TEST-INFO took 129865ms
18:10:50     INFO -  SUITE-END | took 129s
18:10:50    ERROR - Return code: 2
> 18:10:46     INFO -  c:/Program Files (x86)/Windows Kits/10/Windows Performance
> Toolkit/xperf: error: NT Kernel Logger: The instance name passed was not
> recognized as valid by a WMI data provider. (0x1069).

Is often seen when the mof file was not successfully installed.
yeah, I asked for the wrong thing, the .mof file in reference to the comments discussing in comment 14.
ok. the mof file is present on the instances
(see https://tools.taskcluster.net/groups/PFLgDi6tQh6toTqTfuqSAg/tasks/PFLgDi6tQh6toTqTfuqSAg/runs/0/logs/public%2Flogs%2Flive.log).

papertrail shows no errors when running the mofcomp command
(see https://papertrailapp.com/groups/3538333/events?q=[[Script]CommandRun_mofcomp_mozprofilerprobe)

is there something else i should check for?
hmm, looking more I see access denied errors which I didn't see last week- possibly those are the real problems.

https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&fromchange=7820f2bbdc8829d7e36c90f8a8917fc1325eb570

:pmoore, did anything change with scopes and run-as-administrator.
Flags: needinfo?(pmoore)
just a thought:

does the mofcomp command need to be run in the context of the task user (rather than as the system user, during system setup)?

i'm wondering if mofcomp output goes to AppData or some user specific directory. if so, the command will need to be run as part of the task (after the task user is created, maybe in preflight scripts) rather than during setup.

i'm just guessing here. i have no knowledge of mofcomp or xperf beyond the installation steps performed earlier.
I believe mofcomp is needed to run as administrator vs regular user.  But if the user is an admin, maybe there is something there.  This did work last Thursday, but I will scour logs and configs to see if something else changed.
This is the output that I get when running
> mofcomp c:\path\to\mozprofilerprobe.mof

Microsoft (R) MOF Compiler Version 10.0.17134.1
Copyright (c) Microsoft Corp. 1997-2006. All rights reserved.
Parsing MOF file: mozprofilerprobe.mof
MOF file has been successfully parsed
Storing data in the repository...
Done!
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #23)
> hmm, looking more I see access denied errors which I didn't see last week-
> possibly those are the real problems.
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.
> com&fromchange=7820f2bbdc8829d7e36c90f8a8917fc1325eb570
> 
> :pmoore, did anything change with scopes and run-as-administrator.

No, but I see from the task definition here, that the step explained in comment 9 has not been applied:

https://tools.taskcluster.net/groups/FmmdjrliQkmDY8EH7qHyVw/tasks/Vncxza1aSiy9xnglhpfk1g/details
Flags: needinfo?(pmoore)
got it, I fixed that in edit/retrigger and things worked- now let me fix up my patch
we want to run xperf on windows 10 and need administrator access.
here is a try push showing it working:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a9980f5a7ce1308f3403f4126aa2ab0b0355d94

the try push has another patch for the talos changes needed...one step at a time.
Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

there is a phabricator request, but not a bugzilla request
Attachment #9006985 - Flags: review?(pmoore)
Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

Pete Moore [:pmoore][:pete] has approved the revision.
Attachment #9006985 - Flags: review+
just a couple of nits and questions in the review...
Joel, you should probably also get a review from someone else too that knows this code better than me. Perhaps dustin/ahal/Callek.
Flags: needinfo?(jmaher)
ok, asking :dustin for review after addressing the nits!
Flags: needinfo?(jmaher)
Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

Pete Moore [:pmoore][:pete] has been removed from the revision.
Attachment #9006985 - Flags: review+
Comment on attachment 9006985 [details]
Bug 1485757 - add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=pmoore

Dustin J. Mitchell [:dustin] pronoun: he has approved the revision.
Attachment #9006985 - Flags: review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a7876555f4d4
add run-as-administrator support to taskcluster to support xperf on windows 10-64. r=dustin
https://hg.mozilla.org/mozilla-central/rev/a7876555f4d4
Status: REOPENED → RESOLVED
Closed: Last yearLast year
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Blocks: 1490616
Assignee: nobody → jmaher
Attachment #9006985 - Flags: review?(pmoore) → review+
You need to log in before you can comment on or make changes to this bug.