835589 - Profiler script needs more parallelism

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Right now all the profiles are pulled off device serially, and then symbolicated serially.

My machine with 16 HW threads is sitting twiddling its thumbs during this process.  And this overhead is pretty significant in my work cycle.

Comment 1

[Justin Lebar (not reading bugmail)]

I made the symbolicate step very fast for DMD by caching the results across invocations.  Doing so resolved any need for parallelism for me.

Comment 2

[Dave Hylands [:dhylands]]

One of the big delays is waiting for the files to stabilize. Paralleling that would speed things up dramatically.

(as well as adding justin's caching)

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Yeah, I should have added that my common case is |./profile.sh capture|, that is, pulling all profiles off device.  This obviously wouldn't make a difference for |./profile.sh capture Foo|, unless we got really smart about streaming samples through symbolification.

Comment 4

[Gabriele Svelto [:gsvelto]]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/B2G/pull/204

Pointer to Github pull-request

Comment 5

[Gabriele Svelto [:gsvelto]]

Comment on attachment 710225 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/B2G/pull/204

This patch adds a crude mechanism to launch the pull and symbolicate commands in parallel. On my laptop it drops the time required to gather profiles for 7 open applications from ~35s to ~27s.

Comment 6

[Dave Hylands [:dhylands]]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/B2G/pull/205

I would probably do it this way instead...

Comment 7

[Dave Hylands [:dhylands]]

The next optimization would be to do this:

For each b2g sub-process:
  launch a subshell which does the following
    wait for 1 file to stabilize
    pull that 1 file
    symbolicate that 1 file

Comment 8

[Dave Hylands [:dhylands]]

I went ahead and modified pull request 205 to be the fully parallelized version

Comment 9

[Gabriele Svelto [:gsvelto]]

Comment on attachment 710336 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/B2G/pull/205

(In reply to Dave Hylands [:dhylands] from comment #8)
> I went ahead and modified pull request 205 to be the fully parallelized
> version

This is so much better than what I had done; my bash-fu still needs a lot of practice. There's only a small issue to fix before we merge it: the single-pid functionality doesn't work anymore because it doesn't wait for the file to stabilize anymore, I've left the comment on the GitHub diff.

Comment 10

[Dave Hylands [:dhylands]]

- Added some more comments.
- fixed the missing call to cmd_stabilize (as per review)
- did a little more cleanup of the echos

Merged
https://github.com/mozilla-b2g/B2G/commit/739c93331486700eb5cab3c8dfbc77599be3303e