Closed Bug 1425308 Opened 5 years ago Closed 5 years ago

Make it easier to view talos gecko profiles in perf-html.io when running locally

Categories

(Testing :: Talos, enhancement)

Product:
Testing
Component:
Talos
Version:
Version 3
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox62 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla62
Tracking Flags:
Tracking Status
firefox62 --- fixed

People

(Reporter: rwood, Assigned: rwood)

References

(Blocks 1 open bug)

Details

(Whiteboard: [PI:May][perf-tools])

Attachments

(1 file, 3 obsolete files)

Bug 1425308 - Automatically view local talos gecko profile in perf-html.io;
59 bytes, text/x-review-board-request
jmaher
: review+
Details 
When running talos tests locally with --geckoProfiling enabled (via ./mach talos-test), the resulting profile files are packaged up into the local /mozharness/build/blobber_upload_dir.

The next step is to add the ability to have the generated profile automatically opened locally in https://perf-html.io/ so that the profile can be analyzed.
Whiteboard: [PI:January]
Assignee: nobody → rwood
Status: NEW → ASSIGNED
The plan:

After talos test run is done with --geckoProfile,

1. start web server
2. serve out the cycle_0.profile file (profile from the last pageloader cycle, in blobber_uploads)
3. convert that url to perf-html.io format [1]
4. start Firefox and browse to the built perf-html url
5. profile will be loaded and displayed

[1] https://github.com/devtools-html/perf.html/blob/b759d22966406d69383a647f400120290815d675/docs/loading-in-profiles.md#url
I'm having troubles formatting the URL for perf-html.io.

I have a local webserver running, and this URL works fine (browser asks to open/save the profile):

http://127.0.0.1:58474/profile_perf_reftest_singletons/bloom-basic.html_pagecycle_1/cycle_0.profile

Then I try to encode the URL for perf-html.io as noted in the perf-html docs [1]. I'm using 'requests.utils.quote' in python. This is the URL that I am using for perf-html.io:

https://perf-html.io/from-url/$http%3A%2F%2F127.0.0.1%3A58474%2Fprofile_perf_reftest_singletons%2Fbloom-basic.html_pagecycle_1%2Fcycle_0.profile

The perf-html.io site can't upload it, and displays this error:

The requested URL /from-url/$http://127.0.0.1:58474/profile_perf_reftest_singletons/bloom-basic.html_pagecycle_1/cycle_0.profile was not found on this server.

:gregtatum, hi :) I must be formatting the URL incorrectly, can you please give me a pointer here on what I'm doing wrong?

[1] https://github.com/devtools-html/perf.html/blob/b759d22966406d69383a647f400120290815d675/docs/loading-in-profiles.md#url
Flags: needinfo?(gtatum)
Sorry, I think the example may be a little confusing since it's using JavaScript template tags, there is no $ in the URL. I should have written the example like this:

> function getPerfHtmlUrl (profileUrl) {
>   return "https://perf-html.io/from-url/" + encodeURIComponent(profileUrl);
> }

Does that fix the issue?
Flags: needinfo?(gtatum)
(In reply to Greg Tatum [:gregtatum] [@gregtatum] from comment #9)
> 
> Does that fix the issue?

Thanks Greg!

Unfortunately not, I removed the '$' and still receiving this error in the browser when try to go to the full perf-html.io URL:

The requested URL /from-url/http://127.0.0.1:53118/profile_perf_reftest_singletons/bloom-basic.html_pagecycle_1/cycle_0.profile was not found on this server.
Flags: needinfo?(gtatum)
It looks like you are not encoding the URI component. It should look like.

https://perf-html.io/from-url/http%3A%2F%2F127.0.0.1%3A8000%2Fprofile_perf_reftest_singletons%2Fbloom-basic.html_pagecycle_1%2Fcycle_0.profile

I think this will do it correctly:

> import urllib
> url = "http://127.0.0.1:8000/profile_perf_reftest_singletons/bloom-basic.html_pagecycle_1/cycle_0.profile"
> print urllib.quote(url, safe="")

I did some more investigation and it turns out because of perf.html's CSP, you need to serve over https, and add the proper header. I just sent in a PR for updated docs on how to correctly serve from a URL, they're not reviewed or merged in, but I went through the process of serving my own profile from node.js.

https://github.com/gregtatum/perf.html/blob/73ec4c79be9fb17ecd688c0e72e892501e354ab5/docs/loading-in-profiles.md#url
Flags: needinfo?(gtatum)
(In reply to Greg Tatum [:gregtatum] [@gregtatum] from comment #11)
...
Thanks Greg, much appreciated - I'll give that a try!
After doing some preliminary work on this, instead of starting up perf-html.io and automatically loading one of the gecko profiles, I'd like to go a slightly different route.

What I would like to do instead, when running talos locally with --geckoProfile is enabled, is:

- copy all the resulting raw/unzipped resulting gecko profile files into /testing/talos/talos/gecko-proiles-out
- just startup Firefox with the home page of https://perf-html.io/ loaded
- all the user has to do do is click the "browse" button manually, and in the file open dialog, point it to the local gecko-profiles-out folder and select a profile

The benefits of just doing this are:
- gives the developer the flexibility of which gecko profile to open, instead of talos choosing one
- no need to run a local https server and deal with CA certificates, platform differences etc.
- can finish the local talos run and just leave the browser / perf-html.io displayed on it's own; talos not having to wait until viewing is finished and then kill the local https server afterwards
- if the developer wants to view another gecko profile they'd have to use the same method to select a different one and load it anyway

:jmaher, I'd like to go this route instead, what do you think?
Flags: needinfo?(jmaher)
can you make it easier to find that directory?  My concern is that it will be frustrating to find the directory- maybe passing in a hint if possibly: https://perf-html.io/?local_profile_dir=/home/mozilla/testing/talos/talos/profiles, then it would be easier to find.  Possibly if that is too difficult we could have a OBS live hacking session where you generate some profiles and load them in perf-html.io :)
Flags: needinfo?(jmaher)
Summary: Add ability to open local talos test gecko profiles automatically in the perf.html viewer → Make it easier to view talos gecko profiles in perf-html.io
Summary: Make it easier to view talos gecko profiles in perf-html.io → Make it easier to view talos gecko profiles in perf-html.io when running locally
I filed a bug to remove the https-only restriction on perf-html.io:

https://github.com/devtools-html/perf.html/issues/754

I think it's important to make this as easy as possible to load a profile. I filed a bug to provide a URL route where you can display multiple profile options so that the user could select the profile.

https://github.com/devtools-html/perf.html/issues/755

e.g. Something like:

http://perf-html.io/from-ci/?profile[]=https://.../profile1.json&profile[]=https://.../profile2.json

Then you would be presented a screen with multiple profile options.

Would this be enough to make it feasible to view the profiles from a local webserver? I'm starting work on opening gzipped profiles today.
Whiteboard: [PI:January] → [PI:February]
(In reply to Greg Tatum [:gregtatum] [@gregtatum] from comment #22)
...
Thanks Greg. The only issue I see is that some talos suites have multiple tests, and each test has multiple page cycles, with a gecko profile generated for each cycle. So that could be a really really long URL. If there's a screen so that providing multiple URL options that would be great but what happens when the user wants to load another profile from the same test run i.e. the next cycle? Do they press the back button to get the selection screen again? (Same as is they had to use the back button to load again via the Browse button).

Would it be easier to use /from-file/ - but be able to pass in the name of a local folder that could be used as the default top-level folder in the file open / browse dialog? Then when perf-html.io is opened the first time, using a special /from-file/ url, it could maybe automatically display the file open (browse) dialog pointing to the local top level gecko profiles dir? And maintain that root folder for the Browse button dialog? Or is that insecure.
Flags: needinfo?(gtatum)
(In reply to Robert Wood [:rwood] from comment #23)

> ... it could maybe automatically display the file
> open (browse) dialog pointing to the local top level gecko profiles dir? And
> maintain that root folder for the Browse button dialog? Or is that insecure.

We're an unprivileged page, so we only have access to basic HTML elements, like <input type=file />. I don't believe there is anyway to have that file dialog start at a specific place. I just tried on my machine and couldn't get anything working in that regards.

> Thanks Greg. The only issue I see is that some talos suites have multiple
> tests, and each test has multiple page cycles, with a gecko profile
> generated for each cycle.

I'm trying to decide what worst case looks like here. My local build is broken from my OS upgrade right now, but I need to try and generate some profiles locally using talos so I can see what exactly we are dealing with.

If the files are structured at all we could probably shorten the representation in the URL. Or the other option may be to provide some kind of simple REST interface where we could post questions to your local server, and get a JSON representation of the file structure underneath. That sounds a bit more complicated to support and implement though.

perf.html's architecture is such that we encode all of our state in the URL, so the back button should always work if we want to support going between multiple profiles.

I feel like :ochameau is a good person to rope into this conversation, as he has experience with both talos and perf.html.
Flags: needinfo?(gtatum) → needinfo?(poirot.alex)
You surely can make it so that a webpage get access to a local file via chrome privileges,
but as perf-html relies on WebExtension add-on, you most likely can't. I don't think there is any filesystem API in WebExtension.
So I don't see how to sanely expose a local file to perf-html. Unless perf-html interacts indirectly with a privileged profiler API throught a WebExtension API. In such case, you may pass an absolute file path to WebExtension API which is interpreted as-is by the profiler privileged code.

Otherwise, when I want a profile of a local talos run (it is always DAMP for me), I ensure I only run the test once, as:
 * I don't want to wait for the 25 runs to finish before getting a profile,
 * the profile is easier to read, faster to open and navigate,
 * I reduced a lot the noise/average stddev of DAMP and so there isn't a lot of difference between two runs.

For DAMP, it forces me to use a quite complex command line:
  ./mach talos-test --activeTests damp --cycles 1 --tppagecycles 1 --geckoProfile --geckoProfileEntries 100000000 --subtest mytest
I need to tweak both cycles to run the tests only once.
I need to tweak the profiler buffer size, especially if I record all DAMP tests and not only one subtests. (may be the new buffer size allows me to drop this argument?)

So at the end, I always try to make it so that there is only one test run and one profile,
may be we should ensure doing that and that would simplify your work here?

If you are concerned about the URL length, you could do:
  http://perf-html.io/from-ci/?base-url=https://.../?profile[]=profile1.json&profile[]=profile2.json
Or:
  http://perf-html.io/from-ci/?manifest=https://.../manifest.json
  with manifest descripting at length the list of all JSON files with some additional description to help build a profile selector UI.
Flags: needinfo?(poirot.alex)
So it turns out loading a single ZIP file is quite nice for navigating a file system. I still haven't seen exactly what talos is dumping out locally, but from the try build the view is nice:

https://github.com/devtools-html/perf.html/issues/255
https://user-images.githubusercontent.com/1588648/36002652-8e87a3b4-0cf0-11e8-8eed-f0af293b6779.png
(In reply to Greg Tatum [:gregtatum] [@gregtatum] from comment #26)
> So it turns out loading a single ZIP file is quite nice for navigating a
> file system. I still haven't seen exactly what talos is dumping out locally,
> but from the try build the view is nice:
> 
> https://github.com/devtools-html/perf.html/issues/255
> https://user-images.githubusercontent.com/1588648/36002652-8e87a3b4-0cf0-
> 11e8-8eed-f0af293b6779.png

That is AWESOME! Well actually when running talos locally it also creates the same .zip file with the profiles - so I can just host that zip and provide it to perf-html.io the same way.
Whiteboard: [PI:February] → [PI:February][perf-tools]
Whiteboard: [PI:February][perf-tools] → [PI:March][perf-tools]
I think the only other thing blocking this work is allowing http requests, which I just sent in a PR for:
https://github.com/devtools-html/perf.html/issues/727
Awesome thanks Greg!
Attachment #8941617 - Attachment is obsolete: true
Attachment #8942275 - Attachment is obsolete: true
Now that perf-html.io accepts full talos profile zip files, there's no need to have talos/mozharness unzip the talos profile zip file and make it available elsewhere locally.

Now when running talos locally we can leave the talos profile zips at their default / current location (build/blobber_upload_dir) but serve them out - and automatically open the latest one in perf-html.io at the end of the talos run.
Comment on attachment 8964699 [details]
Bug 1425308 - Make it easier to view talos gecko profiles in perf-html.io when running locally (WIP)

https://reviewboard.mozilla.org/r/233410/#review239012

::: testing/mozharness/mozharness/mozilla/testing/talos.py:835
(Diff revision 3)
> +        # (we don't want the local talos run blocked while user is viewing the gecko profile)
> +        perfhtml_proc.wait(timeout=3)
> +
> +        # TODO: How can we have the python localhost server stop when Firefox is shutdown?
> +        # OR do we want to block talos while the profile is being displayed in perf-html.io?
> +        # then when user closes Firefox then kill the server and finish up?

I would expect to hit ctrl-c from the command line on my Mac, with a friendly message explaining that it's serving the profiles.
Whiteboard: [PI:March][perf-tools] → [PI:April][perf-tools]
(In reply to Greg Tatum [:gregtatum] [@gregtatum] from comment #35)
> Comment on attachment 8964699 [details]
> Bug 1425308 - Make it easier to view talos gecko profiles in perf-html.io
> when running locally (WIP)
> 
> https://reviewboard.mozilla.org/r/233410/#review239012
> 
> ::: testing/mozharness/mozharness/mozilla/testing/talos.py:835
> (Diff revision 3)
> > +        # (we don't want the local talos run blocked while user is viewing the gecko profile)
> > +        perfhtml_proc.wait(timeout=3)
> > +
> > +        # TODO: How can we have the python localhost server stop when Firefox is shutdown?
> > +        # OR do we want to block talos while the profile is being displayed in perf-html.io?
> > +        # then when user closes Firefox then kill the server and finish up?
> 
> I would expect to hit ctrl-c from the command line on my Mac, with a
> friendly message explaining that it's serving the profiles.

Thanks Greg!
I'm thinking it would be better to make a separate python tool that receives on the command line:

- path to firefox binary
- path to folder that contains gecko profile zip files

And then:

- starts local python http server to serve out the profile zip folder
- finds the newest gecko profile zip file in the specified folder
- encodes a perf-html.io /from-url/ URL for the gecko profiles zip
- then starts Firefox and loads that URL

Then talos would call that python command in a separate process and after that talos will just shutdown normally.

If it's a separate tool then it could also be used by other frameworks like Raptor.
Note: Keep the talos profiling documentation up to date after this eventually lands

https://wiki.mozilla.org/Performance_sheriffing/Talos/Profiling
Good progress on a tool to view a specified gecko profile zip in perf-htnl.io. i.e. command line will be:

python view_gecko_profile.py -b "/Users/rwood/mozilla-unified/obj-x86_64-apple-darwin17.4.0/dist/Nightly.app/Contents/MacOS/firefox" -p /Users/rwood/mozilla-unified/testing/mozharness/build/blobber_upload_dir/profile_damp.zip
Attachment #8964699 - Attachment is obsolete: true
Depends on: 1456629
Whiteboard: [PI:April][perf-tools] → [PI:May][perf-tools]
:rwood, didn't bug 1456629 solve this?
Flags: needinfo?(rwood)
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #40)
> :rwood, didn't bug 1456629 solve this?

I was going to integrate the view_gecko_profile tool into talos now, so that when talos is run with --geckoProfile have it automatically call view_gecko_profile and open the profile zip.
Flags: needinfo?(rwood)
is there more work to do here now?
Flags: needinfo?(rwood)
Yes thanks for the ping, will get on this ASAP
Flags: needinfo?(rwood)
Comment on attachment 8981201 [details]
Bug 1425308 - Automatically view local talos gecko profile in perf-html.io;

https://reviewboard.mozilla.org/r/247282/#review253658

pretty close to r+, a few small things

::: testing/talos/talos/run_tests.py:335
(Diff revision 3)
>                    % (results_urls['output_urls']))
>  
> +    # when running talos locally with gecko profiling on, use the view-gecko-profile
> +    # tool to automatically load the latest gecko profile in perf-html.io
> +    if config['gecko_profile'] and browser_config['develop']:
> +        LOG.info("*** RW *** config and then browser_config:")

these log statements seem like debugging statements

::: testing/talos/talos/run_tests.py:349
(Diff revision 3)
>  
> +def view_gecko_profile(ffox_bin):
> +    # automatically load the latest talos gecko-profile archive in perf-html.io
> +    # unless TALOS_DISABLE_PERFHTMLIO_LAUNCH is set
> +    if os.environ.get('TALOS_DISABLE_PROFILE_LAUNCH', '0') == '1':
> +        LOG.info("Not launching perf-html.io because TALOS_DISABLE_PROFILE_LAUNCH=1")

could this check be done outside of this call?  If not, I would prefer that we don't if/else, but instead:
if:
  return
  
<show profile>

::: testing/talos/talos/run_tests.py:355
(Diff revision 3)
> +    else:
> +        if not os.path.exists(ffox_bin):
> +            LOG.info("unable to find Firefox bin, cannot launch view-gecko-profile")
> +            return
> +        profile_zip = os.environ.get('TALOS_LATEST_GECKO_PROFILE_ARCHIVE', None)
> +        if profile_zip is not None:

I would rather see:
if profile_zip is none:
  return
Attachment #8981201 - Flags: review?(jmaher) → review-
Comment on attachment 8981201 [details]
Bug 1425308 - Automatically view local talos gecko profile in perf-html.io;

https://reviewboard.mozilla.org/r/247282/#review253658

Thanks! Updated accordingly.
Comment on attachment 8981201 [details]
Bug 1425308 - Automatically view local talos gecko profile in perf-html.io;

https://reviewboard.mozilla.org/r/247282/#review253688

thanks, this looks cleaner and is easy to read
Attachment #8981201 - Flags: review?(jmaher) → review+
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7f3ed39073e6
Automatically view local talos gecko profile in perf-html.io; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/7f3ed39073e6
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox62: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
You need to log in before you can comment on or make changes to this bug.