Closed Bug 1473079 Opened Last year Closed Last year

Add gecko profiling support to raptor

Categories

(Testing :: Raptor, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox65 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 1 obsolete file)

i.e. /mach raptor-test --test ... --geckoProfile

Should generate gecko profiles similar to how talos currently does.
Blocks: 1473078
Blocks: 1481867
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Comment on attachment 9010358 [details]
Bug 1473079 - Add gecko profiling support to raptor (part 1: cmd line args); r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.
Attachment #9010358 - Flags: review+
Comment on attachment 9010408 [details]
Bug 1473079 - Add gecko profiling support to raptor (part 2: limit page cycles); r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.
Attachment #9010408 - Flags: review+
Blocks: 1492913
Depends on: 1492955
Blocks: 1493243
Attaching a sample gecko profile zip output by Raptor. For some reason when I attempt to load this in perf-html.io, after selecting the pagecycle profile, nothing is displayed and there are errors in the console (see below).

:gregtatum, would you mind having a look at the attached zip from raptor? There must be something wrong with the way it being created or the contents...? Thanks.

11:26:19.455
Failed to process the profile in the zip file with the following error: zipped-profiles.js:72:6
xu/<
zipped-profiles.js:72:6
11:26:19.455
Error: "Attempted to get a thread index before a profile was loaded."
...
11:26:19.455
Previous ZipFileState: 
Object { phase: "VIEW_PROFILE_IN_ZIP_FILE", zip: {…}, pathInZipFile: "profile_raptor-tp6-facebook-firefox/raptor-tp6-facebook-firefox_pagecycle_1.profile" }
zipped-profiles.js:148:4
11:26:19.455
Next ZipFileState: 
Object { phase: "FAILED_TO_PROCESS_PROFILE_FROM_ZIP_FILE", zip: {…}, pathInZipFile: "profile_raptor-tp6-facebook-firefox/raptor-tp6-facebook-firefox_pagecycle_1.profile" }
zipped-profiles.js:149:4
11:26:19.458 Error: Attempted to transition the ZipFileState from the phase “VIEW_PROFILE_IN_ZIP_FILE” to “FAILED_TO_PROCESS_PROFILE_FROM_ZIP_FILE”, however “VIEW_PROFILE_IN_ZIP_FILE” can only transition to “LIST_FILES_IN_ZIP_FILE”.
Flags: needinfo?(gtatum)
Attachment #9012206 - Flags: feedback?(gtatum)
Looks like the profiles have no threads, which happens to break the profiler. I'll follow-up with a bug on the profiler side to display a better error message.

This can easily happen if the thread selector fails to match any threads.
(In reply to Greg Tatum [:gregtatum] from comment #9)
> Looks like the profiles have no threads, which happens to break the
> profiler. I'll follow-up with a bug on the profiler side to display a better
> error message.
> 
> This can easily happen if the thread selector fails to match any threads.

Ok thank you for the lead there, maybe I need to do something to enable the profiler to run in content as well somehow.
I created this PR to at handle that error case: https://github.com/devtools-html/perf.html/pull/1318

Let me know if you need me to look into why the threads aren't being recorded. I'll probably need more context into what raptor is to help.
Flags: needinfo?(gtatum)
(In reply to Greg Tatum [:gregtatum] from comment #11)

> Let me know if you need me to look into why the threads aren't being
> recorded. I'll probably need more context into what raptor is to help.

Hi Greg,

Yes please if you could please take a look I'd really appreciate it. I'm not sure what I am doing wrong.

Raptor is a performance framework similar to Talos except it runs as a webextension. General info on Raptor is here [1]. With this patch I am using the browser.GeckoProfiler API, starting, stopping, and getting the gecko profiles inside the main webextension chrome code (runner.js) found here [2]. The patch is pretty big but if you need surrounding code the entire patch is at [3]. I don't see any errors in the console from the browser.GeckoProfiler API.

Do I need to be calling the browser.GeckoProfiler API calls from within the webext content instead maybe? I thought though at least the GeckoMain thread would be working with this code. Any advice would be greatly appreciated.

[1] https://wiki.mozilla.org/Performance_sheriffing/Raptor

[2] https://phabricator.services.mozilla.com/differential/changeset/?ref=172077&whitespace=ignore-most

[3] https://phabricator.services.mozilla.com/D6918
Flags: needinfo?(gtatum)
Comment on attachment 9012201 [details]
Bug 1473079 - Add gecko profiling support to raptor (part 3: enable gecko profiling); r?jmaher

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.
Attachment #9012201 - Flags: review+
also Greg, if you are not the right person, can you forward the needinfo request to whomever would be the right person to answer it.
Hi :florian, do you have any guidance or advice here? Thanks!
Flags: needinfo?(florian)
I've been out on PTO for almost 2 weeks, I'll take a look at this bug sometime before the end of the week.
I added a comment on what I think is going on in Phabricator. Feel free to ping me some more if that doesn't work.
Flags: needinfo?(gtatum)
Flags: needinfo?(florian)
(In reply to Greg Tatum [:gregtatum] from comment #21)
> I added a comment on what I think is going on in Phabricator. Feel free to
> ping me some more if that doesn't work.

Thanks Greg yes that fixed it!
In treeherder you can see the gecko profiles from raptor by:

- select a raptor test job, click job details tab
- you'll see the profile_raptor_* zip artifacts listed, beside each click the 'open in perf-html.io' link
- then in perf-html.io you can select one of the pagecycles to view

Locally when running raptor the gecko profile zips are stored in testing/mozharness/build/blobber_upload_dir/.

:mstange, when you have a moment could you please look at some of the profile artifacts on the try run [1]? Looks like the symbols are missing (?) and so I can't tell if they are sufficient. Any direction would be appreciated please as I want to be sure the profiles provided by raptor are of good use! Thanks.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=78a237207038abb3902b175339db5b27094c8e00
Flags: needinfo?(mstange)
You're right, the symbols are missing from at least the Win 10 x64 profiles. I didn't check any others. Do you know if the symbolicator.integrate_symbol_zip_from_url / symbolicator.integrate_symbol_zip_from_file lines are hit? That's where I'd start debugging.
Flags: needinfo?(mstange)
Attachment #9012206 - Flags: feedback?(gtatum)
(In reply to Markus Stange [:mstange] from comment #25)

Thanks for the pointer Markus that's the issue, no symbols_path so those methods weren't even being called.
(In reply to Robert Wood [:rwood] from comment #27)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0787ac93fbb935d1eefdac7da59d1e0e8392a3b

Profiling working in production ^ ready to land after review of latest updates in phab
Great! The symbols are present now.

However, I think the ss and the tp6 profiles show that the profiler buffer isn't cleared between page cycles. You need to call StopProfiler and StartProfiler between each pagecycle, otherwise the second profile will include the first profile.

For example, here are the first 12 seconds of the two facebook pagecycle profiles:
https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FRWG5SBYyR06RmRodcTNuZQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_raptor-tp6-facebook-firefox.zip/calltree/?file=profile_raptor-tp6-facebook-firefox%2Fraptor-tp6-facebook-firefox_pagecycle_1.profile&globalTrackOrder=0-1-2-3-4-5&hiddenGlobalTracks=1-2-4&localTrackOrderByPid=6392-1-0~5708-0~6428-0~6576-0~&range=0.0000_12.0106&thread=7&v=3
https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FRWG5SBYyR06RmRodcTNuZQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_raptor-tp6-facebook-firefox.zip/calltree/?file=profile_raptor-tp6-facebook-firefox%2Fraptor-tp6-facebook-firefox_pagecycle_2.profile&globalTrackOrder=0-1-2-3-4-5&hiddenGlobalTracks=1-2-3&localTrackOrderByPid=6392-1-0~5708-0~6428-0~6576-0~&range=0.0000_12.0045&thread=0&v=3

They're identical. The second profile additionally contains the second pagecycle at the end, but in this link I've restricted the visible range to the first 12 seconds.

Also, in the speedometer profile, it looks like the buffer isn't big enough to contain the entire benchmark run. It probably needs to be 4 or 5 times the current size.
Blocks: 1502873
(In reply to Markus Stange [:mstange] from comment #29)

Thanks Markus. I filed Bug 1502873 for this follow-up work.
Blocks: 1502922
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/74af7ef01752
Add gecko profiling support to raptor (part 1: cmd line args); r=jmaher
https://hg.mozilla.org/integration/autoland/rev/64f653c604bb
Add gecko profiling support to raptor (part 2: limit page cycles); r=jmaher
https://hg.mozilla.org/mozilla-central/rev/74af7ef01752
https://hg.mozilla.org/mozilla-central/rev/64f653c604bb
https://hg.mozilla.org/mozilla-central/rev/70378a21cbf5
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
Blocks: 1514354
Attachment #9010455 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.