1596182 - Consider adding raptor-webaudio to PGO training

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, enhancement)

Description

[(Away)]

As a bonus sidequest during bug 1596172, I also looked into adding various other perf tests to our PGO training set. The webaudio test had the most dramatic wins, because it's so heavy on C++ code. PGO can give >20% speedups to this code.

However, being unfamiliar with the code area, I can't say whether this improvement would be useful for real-world users of these codepaths, or if this is just improvement for the sake of test numbers and is useless (or even harmful) for real-world scenarios.

On IRC padenot mentioned wanting to test out some builds locally before we proceed. This try comparison links to a pair of linux64-shippable builds that ought to be suitable: https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&newProject=try&newRevision=fa26a0ae0f49142fa7c42d7e893512bd78e101a8&originalSignature=1904137&newSignature=1904137&framework=10&originalRevision=028f3d54f9277acb9c6a38b2f59270cd94bbd748

Comment 1

[Paul Adenot (:padenot)]

Andreas, this is what I was talking to you about, if you can try those with your benchmark.

Comment 2

[Andreas Pehrson [:pehrsons]]

My full benchmark I run on a mac, because my linux desktop can load up with remote peers. I cannot run that here since there's no mac PGO build, so I had to something simpler. Then my fiber connection happens to be down, so I couldn't do it anyway (the benchmark would use a service with an SFU, basically a server component so each peer only has to encode audio/video once).

I used a simpler benchmark with 3 remote peers (on the same machine as it happens) in a mesh setup (3 audio streams and 3 video streams encoded). It shows some differences. CPU was not nearly saturated by the look of things.

This was with audioipc enabled. Screenshots of the analyses coming up.

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: bug1596182_linux64_mesh-3remotes_pre.png (without raptor-webaudio)

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: bug1596182_linux64_mesh-3remotes_post.png (with raptor-webaudio)

Comment 5

[Andreas Pehrson [:pehrsons]]

And FWIW these are both 10s of traces taken once the call was stable, normalized to 0.

Result of adding raptor-webaudio to PGO training:
Mean: +0.00923766088
Median: +0.00738383829
Variance: -0.00016806234
Standard deviation: -0.00179392191

So stddev shows a slight improvement, but we're taking almost one percent point more of the audio budget.

Paul, thoughts?

Comment 6

[Paul Adenot (:padenot)]

This is probably not statistically significant, I'll try to reproduce those numbers, and try other benchmarks (with more web audio, for example, AudioParam usage), thanks!

Comment 7

[(Away)]

FWIW, don't spend too too much time on this, if there's nothing obviously great, I don't mind abandoning the patch (it'll keep our PGO builds faster anyway). I'm also happy to let this wait until bug 1528374 enables PGO on mac.

Comment 8

[Paul Adenot (:padenot)]

If we can get improved performances on Web Audio without regressing something else by adding a couple lines to a JSON file, it's worth it imo.

Comment 9

[Chris Manchester (limited bugmail, email directly)]

Hi :padenot, I was just taking a look at this as we turned on PGO for macOS and there's a noticeable regression in our raptor webaudio benchmark. If I add the benchmark into the profile run it turns into an improvement. Is this benchmark trustworthy? Should we land this change based on this?

Comment 10

[Paul Adenot (:padenot)]

I think it will be fine, yes.

Comment 11

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1596182 - Add raptor-webaudio to the pgo profile run.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/476527cfd083
Add raptor-webaudio to the pgo profile run. r=firefox-build-system-reviewers,rstewart

Comment 13

[Narcis Beleuzu [:NarcisB]]

Backed out for Bpgo crashes on linux-gate.so

Backout link: https://hg.mozilla.org/integration/autoland/rev/43f26061743cb8f64c3a853b7770892615b5dfb5
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=282138149&repo=autoland&lineNumber=316

Comment 14

[Chris Manchester (limited bugmail, email directly)]

The failure is:
Sandbox: seccomp sandbox violation: pid 1045, tid 1045, syscall 172, args 2 4290949876 12383 4016714057 1450277296 4290950008. Killing process.

These images don't seem to have the setup required for audio things.

Comment 15

[Chris Manchester (limited bugmail, email directly)]

Hmm. Perhaps we should add these only on osx for the moment to avoid the regression.

:egao, do you have a sense of what is usually required to run media tests on linux workers? These jobs are using the ubuntu1804-test and debian9-amd64-build docker images currently: https://searchfox.org/mozilla-central/rev/ac43d7e69a435ede789827f20ab309da6f04c09b/taskcluster/ci/generate-profile/kind.yml#26

Comment 16

[Edwin Takahashi (:egao | infrequent contributor)]

:chmanchester - I switched over the image used from desktop1604-test to ubuntu1804-test a couple of weeks ago, and I thought there were no fallout from it - clearly that wasn't the case!

Typically mochitest-media tests and media-related web-platform-tests run just fine on the ubuntu1804-test image, though since yesterday I have noticed a single failure cropping up in mda3 on the reasonably new mozilla-central:

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=361906068f588a4441ffc7c21dc32dce4500cf55

I'm not sure if the above was helpful. Perhaps we can revert to using desktop1604-test image for profile generation and see if this patch fails then?

Comment 17

[Edwin Takahashi (:egao | infrequent contributor)]

Just to add, for what it's worth I am not seeing this bustage when I push to try and schedule the same task:

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=ea0cc7380662e88ee07ee1107e51293ef8b7146d

Comment 18

[Chris Manchester (limited bugmail, email directly)]

(In reply to Edwin Takahashi (:egao, :etakahashi) from comment #16)

:chmanchester - I switched over the image used from desktop1604-test to ubuntu1804-test a couple of weeks ago, and I thought there were no fallout from it - clearly that wasn't the case!

Typically mochitest-media tests and media-related web-platform-tests run just fine on the ubuntu1804-test image, though since yesterday I have noticed a single failure cropping up in mda3 on the reasonably new mozilla-central:

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&tier=1%2C2%2C3&revision=361906068f588a4441ffc7c21dc32dce4500cf55

I'm not sure if the above was helpful. Perhaps we can revert to using desktop1604-test image for profile generation and see if this patch fails then?

Ok, I tried this here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9c1a8bad80c07042634c645505cbab5efbc90407

... but we have the same crash. We're running these on linux as a part of the raptor suite, there must be some difference I'm missing.

Comment 19

[Edwin Takahashi (:egao | infrequent contributor)]

That's strange. My linux32 profile generation seems to complete without any issues using ubuntu1804 in the push I linked.

I'm sure you've tried, but is your base m-c revision up to date? I can't think of anything else that may be at fault.

I can also try applying your patch to my environment locally then doing a push to see if it fails for me.

Comment 20

[Chris Manchester (limited bugmail, email directly)]

Oh, sorry, yeah, I wouldn't expect this to fail without the patch applied. I didn't think anything here was your fault, I was just hoping you might have some hints for me as to what to try next :-)

Comment 21

[Edwin Takahashi (:egao | infrequent contributor)]

Even with the current m-c revision (on 12/20 at least) the push failed, so I wonder why. Considering that it fails on both ubuntu1604 and ubuntu1804, I suspect it has to do with a 32bit version of the binary that is not installed in the setup.

The currently installed audio-related packages on ubuntu1804 are:

• pulseaudio
• gstreamer1.0 plugins
• libasound
  libcanberra
  libgstreamer
• libpulse
• pulseaudio-module-blueooth and pulseaudio-module-gconf

None of these have the i386 version explicitly installed.

The full list of the installed packages: https://searchfox.org/mozilla-central/source/taskcluster/docker/recipes/ubuntu1804-test-system-setup.sh

I will try a push with i386 versions of these installed where available and see if that solves anything.

EDIT: also, SIGSYS is bad system call, which is a bit confusing...

Comment 22

[Chris Manchester (limited bugmail, email directly)]

Aha, it turns out this is bug 1553850 but for the rdd process sandbox. We can probably just turn off the decoder process sandbox during the profile run for now. Thanks for taking a look, though!

Comment 23

[(Away)]

(Catching up on holiday bugmail)
Thanks for taking this on, Chris!

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f01d0cb8a695
Add raptor-webaudio to the pgo profile run. r=firefox-build-system-reviewers,rstewart

Comment 25

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/f01d0cb8a695

Comment 26

[Florin Strugariu [:Bebe]]

== Change summary for alert #24609 (as of Fri, 03 Jan 2020 22:47:04 GMT) ==

Improvements:

11% perf_reftest_singletons external-string-pass.html macosx1014-64-shippable opt e10s stylo 901.05 -> 805.43

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24609

Comment 27

[Dave Hunt [:davehunt] [he/him] ⌚BST]

== Change summary for alert #24608 (as of Fri, 03 Jan 2020 21:40:56 GMT) ==

Improvements:

25% raptor-webaudio-firefox macosx1014-64-shippable opt 196.00 -> 146.83
21% raptor-webaudio-firefox linux64-shippable opt 150.83 -> 119.25
21% raptor-webaudio-firefox linux64-shippable-qr opt 157.50 -> 124.67
10% raptor-webaudio-firefox windows10-64-shippable opt 160.38 -> 145.00
9% raptor-webaudio-firefox windows10-64-shippable-qr opt 159.42 -> 144.42
8% raptor-webaudio-firefox windows7-32-shippable opt 197.04 -> 180.58

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24608