1473079 - Add gecko profiling support to raptor

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Robert Wood [:rwood]]

i.e. /mach raptor-test --test ... --geckoProfile

Should generate gecko profiles similar to how talos currently does.

Comment 1

[Robert Wood [:rwood]]

Attachment: Bug 1473079 - Add gecko profiling support to raptor (part 1: cmd line args); r?jmaher

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9010358 [details]
Bug 1473079 - Add gecko profiling support to raptor (part 1: cmd line args); r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.

Comment 3

[Robert Wood [:rwood]]

Attachment: Bug 1473079 - Add gecko profiling support to raptor (part 2: limit page cycles); r?jmaher

Depends on D6286

Comment 4

[Robert Wood [:rwood]]

Attachment: Bug 1473079 - Add gecko profiling support to raptor (part 3: enable gecko profiling; wip)

Depends on D6309

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9010408 [details]
Bug 1473079 - Add gecko profiling support to raptor (part 2: limit page cycles); r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.

Comment 6

[Robert Wood [:rwood]]

Attachment: Bug 1473079 - Add gecko profiling support to raptor (part 3: enable gecko profiling); r?jmaher

Depends on D6309

Comment 7

[Robert Wood [:rwood]]

Attachment: profile_raptor-tp6-google-firefox.zip

Attaching a sample gecko profile zip output by Raptor. For some reason when I attempt to load this in perf-html.io, after selecting the pagecycle profile, nothing is displayed and there are errors in the console (see below).

:gregtatum, would you mind having a look at the attached zip from raptor? There must be something wrong with the way it being created or the contents...? Thanks.

11:26:19.455
Failed to process the profile in the zip file with the following error: zipped-profiles.js:72:6
xu/<
zipped-profiles.js:72:6
11:26:19.455
Error: "Attempted to get a thread index before a profile was loaded."
...
11:26:19.455
Previous ZipFileState: 
Object { phase: "VIEW_PROFILE_IN_ZIP_FILE", zip: {…}, pathInZipFile: "profile_raptor-tp6-facebook-firefox/raptor-tp6-facebook-firefox_pagecycle_1.profile" }
zipped-profiles.js:148:4
11:26:19.455
Next ZipFileState: 
Object { phase: "FAILED_TO_PROCESS_PROFILE_FROM_ZIP_FILE", zip: {…}, pathInZipFile: "profile_raptor-tp6-facebook-firefox/raptor-tp6-facebook-firefox_pagecycle_1.profile" }
zipped-profiles.js:149:4
11:26:19.458 Error: Attempted to transition the ZipFileState from the phase “VIEW_PROFILE_IN_ZIP_FILE” to “FAILED_TO_PROCESS_PROFILE_FROM_ZIP_FILE”, however “VIEW_PROFILE_IN_ZIP_FILE” can only transition to “LIST_FILES_IN_ZIP_FILE”.

Comment 8

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e4b9183408bcd9a9663761b2308c56dfec076a5

Comment 9

[Greg Tatum [:gregtatum]]

Looks like the profiles have no threads, which happens to break the profiler. I'll follow-up with a bug on the profiler side to display a better error message.

This can easily happen if the thread selector fails to match any threads.

Comment 10

[Robert Wood [:rwood]]

(In reply to Greg Tatum [:gregtatum] from comment #9)
> Looks like the profiles have no threads, which happens to break the
> profiler. I'll follow-up with a bug on the profiler side to display a better
> error message.
> 
> This can easily happen if the thread selector fails to match any threads.

Ok thank you for the lead there, maybe I need to do something to enable the profiler to run in content as well somehow.

Comment 11

[Greg Tatum [:gregtatum]]

I created this PR to at handle that error case: https://github.com/devtools-html/perf.html/pull/1318

Let me know if you need me to look into why the threads aren't being recorded. I'll probably need more context into what raptor is to help.

Comment 12

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f98e042a4a796644e032a3430bb8a72c079ea6a9

Comment 13

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2b474730f4135d947a1fd520c07c0afe0b7788ed

Comment 14

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=38c7dc4a40ca87fa52ce5d15eb2accf6d7a8761a

Comment 15

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bc455d1c252d2a76dfdc0455ada1046d86147c0a

Comment 16

[Robert Wood [:rwood]]

(In reply to Greg Tatum [:gregtatum] from comment #11)

> Let me know if you need me to look into why the threads aren't being
> recorded. I'll probably need more context into what raptor is to help.

Hi Greg,

Yes please if you could please take a look I'd really appreciate it. I'm not sure what I am doing wrong.

Raptor is a performance framework similar to Talos except it runs as a webextension. General info on Raptor is here [1]. With this patch I am using the browser.GeckoProfiler API, starting, stopping, and getting the gecko profiles inside the main webextension chrome code (runner.js) found here [2]. The patch is pretty big but if you need surrounding code the entire patch is at [3]. I don't see any errors in the console from the browser.GeckoProfiler API.

Do I need to be calling the browser.GeckoProfiler API calls from within the webext content instead maybe? I thought though at least the GeckoMain thread would be working with this code. Any advice would be greatly appreciated.

[1] https://wiki.mozilla.org/Performance_sheriffing/Raptor

[2] https://phabricator.services.mozilla.com/differential/changeset/?ref=172077&whitespace=ignore-most

[3] https://phabricator.services.mozilla.com/D6918

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9012201 [details]
Bug 1473079 - Add gecko profiling support to raptor (part 3: enable gecko profiling); r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

also Greg, if you are not the right person, can you forward the needinfo request to whomever would be the right person to answer it.

Comment 19

[Robert Wood [:rwood]]

Hi :florian, do you have any guidance or advice here? Thanks!

Comment 20

[Greg Tatum [:gregtatum]]

I've been out on PTO for almost 2 weeks, I'll take a look at this bug sometime before the end of the week.

Comment 21

[Greg Tatum [:gregtatum]]

I added a comment on what I think is going on in Phabricator. Feel free to ping me some more if that doesn't work.

Comment 22

[Robert Wood [:rwood]]

(In reply to Greg Tatum [:gregtatum] from comment #21)
> I added a comment on what I think is going on in Phabricator. Feel free to
> ping me some more if that doesn't work.

Thanks Greg yes that fixed it!

Comment 23

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=78a237207038abb3902b175339db5b27094c8e00

Comment 24

[Robert Wood [:rwood]]

In treeherder you can see the gecko profiles from raptor by:

- select a raptor test job, click job details tab
- you'll see the profile_raptor_* zip artifacts listed, beside each click the 'open in perf-html.io' link
- then in perf-html.io you can select one of the pagecycles to view

Locally when running raptor the gecko profile zips are stored in testing/mozharness/build/blobber_upload_dir/.

:mstange, when you have a moment could you please look at some of the profile artifacts on the try run [1]? Looks like the symbols are missing (?) and so I can't tell if they are sufficient. Any direction would be appreciated please as I want to be sure the profiles provided by raptor are of good use! Thanks.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=78a237207038abb3902b175339db5b27094c8e00

Comment 25

[Markus Stange [:mstange]]

You're right, the symbols are missing from at least the Win 10 x64 profiles. I didn't check any others. Do you know if the symbolicator.integrate_symbol_zip_from_url / symbolicator.integrate_symbol_zip_from_file lines are hit? That's where I'd start debugging.

Comment 26

[Robert Wood [:rwood]]

(In reply to Markus Stange [:mstange] from comment #25)

Thanks for the pointer Markus that's the issue, no symbols_path so those methods weren't even being called.

Comment 27

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0787ac93fbb935d1eefdac7da59d1e0e8392a3b

Comment 28

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #27)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0787ac93fbb935d1eefdac7da59d1e0e8392a3b

Profiling working in production ^ ready to land after review of latest updates in phab

Comment 29

[Markus Stange [:mstange]]

Great! The symbols are present now.

However, I think the ss and the tp6 profiles show that the profiler buffer isn't cleared between page cycles. You need to call StopProfiler and StartProfiler between each pagecycle, otherwise the second profile will include the first profile.

For example, here are the first 12 seconds of the two facebook pagecycle profiles:
https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FRWG5SBYyR06RmRodcTNuZQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_raptor-tp6-facebook-firefox.zip/calltree/?file=profile_raptor-tp6-facebook-firefox%2Fraptor-tp6-facebook-firefox_pagecycle_1.profile&globalTrackOrder=0-1-2-3-4-5&hiddenGlobalTracks=1-2-4&localTrackOrderByPid=6392-1-0~5708-0~6428-0~6576-0~&range=0.0000_12.0106&thread=7&v=3
https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FRWG5SBYyR06RmRodcTNuZQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_raptor-tp6-facebook-firefox.zip/calltree/?file=profile_raptor-tp6-facebook-firefox%2Fraptor-tp6-facebook-firefox_pagecycle_2.profile&globalTrackOrder=0-1-2-3-4-5&hiddenGlobalTracks=1-2-3&localTrackOrderByPid=6392-1-0~5708-0~6428-0~6576-0~&range=0.0000_12.0045&thread=0&v=3

They're identical. The second profile additionally contains the second pagecycle at the end, but in this link I've restricted the visible range to the first 12 seconds.

Also, in the speedometer profile, it looks like the buffer isn't big enough to contain the entire benchmark run. It probably needs to be 4 or 5 times the current size.

Comment 30

[Robert Wood [:rwood]]

(In reply to Markus Stange [:mstange] from comment #29)

Thanks Markus. I filed Bug 1502873 for this follow-up work.

Comment 31

[Robert Wood [:rwood]]

Landed:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=70378a21cbf570a359b3ce4f134c45256241c439

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/74af7ef01752
Add gecko profiling support to raptor (part 1: cmd line args); r=jmaher
https://hg.mozilla.org/integration/autoland/rev/64f653c604bb
Add gecko profiling support to raptor (part 2: limit page cycles); r=jmaher

Comment 33

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/74af7ef01752
https://hg.mozilla.org/mozilla-central/rev/64f653c604bb
https://hg.mozilla.org/mozilla-central/rev/70378a21cbf5