1650363 - Performance testing of HTTP3 in Firefox using only xpcshell tests

Status: RESOLVED FIXED

(Core :: Networking: HTTP, task, P2)

Description

[Dragana Damjanovic [:dragana]]

Implement a simple xpcshell tests that:

• download a large amount of data over one stream
• upload a large amount of data over a single stream
• download a large number of small responses that should fill the flow-control window (number of streams and size TBA)
• upload a large number of small responses that should fill the flow-control window (number of streams and size TBA)

Set up different network conditions:

• bandwidth 10Mbps, 20Mbps, 50Mbps and 100Mbps
• delay by 10ms, 50ms, 100ms and 250ms
• loss 1%, 2%, 5%, 10% and 20%

Compare the performance to the performance of TCP sending/receiving large amount of data. Use NewReno, Cubic and BBR as the congestion control algorithm.

Comment 1

[Dragana Damjanovic [:dragana]]

This tests will estimate performance of of neqo and its integration into Firefox networking stack, but it will not be influence by DOM, layout and all upper code. The complete Firefox performance will be tested in bug 1650358.

Comment 2

[Andy Grover [:grover]]

Michal, can you own this?

Comment 3

[Dragana Damjanovic [:dragana]]

Michal , for now we only need xpcshell test and Tarek will integrate it into test automation. I will e-mail you some detail.

Comment 4

[Michal Novotny [:michal]]

Attachment: Bug 1650363 - Performance testing of HTTP3 in Firefox using only xpcshell tests

Comment 5

[Tarek Ziadé (:tarek)]

Try run https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=3110d0831d5832ed7158d414d1da36bdae993d2f

Comment 6

[Tarek Ziadé (:tarek)]

new try run https://treeherder.mozilla.org/#/jobs?repo=try&revision=220eaf1de158838b4c78e1ba68c9c44a33694969

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1ccda1d0c3ea
Performance testing of HTTP3 in Firefox using only xpcshell tests r=sparky

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/1ccda1d0c3ea

Comment 9

[Mike Hommey [:glandium]]

This caused a serious regression in packaging time: https://treeherder.mozilla.org/perf.html?#/graphs?highlightAlerts=1&series=autoland,1582207,1,2&series=autoland,1618755,1,2&series=autoland,1691791,1,2&timerange=1209600&zoom=1595891395333,1596540774732,-48.8679120730571,415.58575892303327

Comment 10

[Mike Hommey [:glandium]]

+        {
+            'source': buildconfig.topsrcdir,
+            'pattern': 'browser/config/**'
+        },
+        {
+            'source': buildconfig.topsrcdir,
+            'pattern':  'build/moz.configure/**'
+        },
+        {
+            'source': buildconfig.topobjdir,
+            'pattern': 'dist/bin/**',
+        },
+        {
+            'source': buildconfig.topobjdir,
+            'base': '_tests/modules',
+            'pattern': '**',
+            'dest': 'dist/bin/modules'
+        },
+        {
+            'source': buildconfig.topobjdir,
+            'pattern': 'dist/plugins/**'
+        },
+        {
+            'source': buildconfig.topsrcdir,
+            'base': 'netwerk/test/http3serverDB',
+            'pattern': '**',
+            'dest': 'netwerk/test/http3serverDB',
         }

No wonder why... this is very broad, and turned the perftest archives from a 19MB tarball into a 1.5GB one. Which is also a lot of storage use. I think this should be backed out.

Comment 11

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 1ccda1d0c3ea (bug 1650363) for causing a serious regression in packaging time

Backout link: https://hg.mozilla.org/integration/autoland/rev/283caf8887b3b34ed80921105b41a6dd0e828418

Comment 12

[Tarek Ziadé (:tarek)]

(In reply to Mike Hommey [:glandium] from comment #10)

No wonder why... this is very broad, and turned the perftest archives from a 19MB tarball into a 1.5GB one. Which is also a lot of storage use. I think this should be backed out.

Sorry I have not noticed it was so big on some platforms. I need the ability to run xpcshell, which is not present in target.tar.bz2
In my attempts to use a subset of dist/bin I had several issues, I could not find the right subset, so I included it fully.

Mike, is there an existing way to have the xpcshell binary without having to get all of dist/bin ?

Comment 13

[Mike Hommey [:glandium]]

xpcshell is in target.common.tests.tar.gz. Check how the xpcshell tests use xpcshell...

Why do you package browser/config/** and build/moz.configure/**? They're not big, but they shouldn't be necessary.

Comment 14

[Tarek Ziadé (:tarek)]

(In reply to Mike Hommey [:glandium] from comment #13)

xpcshell is in target.common.tests.tar.gz. Check how the xpcshell tests use xpcshell...

ok

Why do you package browser/config/** and build/moz.configure/**? They're not big, but they shouldn't be necessary.

They are required by the MozbuildObject object in my (non mach) runner.

Comment 15

[Mike Hommey [:glandium]]

They are required by the MozbuildObject object in my (non mach) runner.

None of the other test runners need that.

Comment 16

[Mike Hommey [:glandium]]

Come to think of it, why is this not using the xpcshell test harness?

Comment 17

[Tarek Ziadé (:tarek)]

Come to think of it, why is this not using the xpcshell test harness?

because we provide an environment to run performance tests, and performance tests can be xpcshell tests or browsertime tests.
and I don't want it to rely on mozharness, which forces an architecture and design we don't want.

We wrap runxpcshelltests.py and that works fine. The only remaining issue is to have the xpcshell binary
working in the CI without pulling the whole dist/bin dir.

So, target.common.tests.tar.gz does contain the xpshell binary but it's dynamically linked to libs that are not
there. You can't run it without all the *dylib files

dyld: Library not loaded: @executable_path/libnss3.dylib
  Referenced from: /Users/tarek/Downloads/tarball/bin/xpcshell
  Reason: image not found

That brings me back to the reason why I grabbed everyhing under dist/bin in my test package.
But that would grow again the package. I guess those libs are available in target.tar.bz2, target.dmg and that's what mozharness does.

I'll try to do something with that.

Maybe a simple solution would be to have an artifact that provides everything required to run xpcshell tests directly (the binary and all its libs)

Comment 18

[Tarek Ziadé (:tarek)]

(that makes me wonder why we don't ship the xpcshell binary in the main target artifact)

Comment 19

[Mike Hommey [:glandium]]

runxpcshelltests should be dealing with those details if you pass the right arguments (--xre-path).
I don't get why you need build system stuff if you're using runxpcshelltests.

(that makes me wonder why we don't ship the xpcshell binary in the main target artifact)

Because it's not something we ship with Firefox.

Comment 20

[Tarek Ziadé (:tarek)]

Try run with reduced test package size

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=78914f18c97ff08695bb05585ac82de57bd74c4e

Comment 21

[Tarek Ziadé (:tarek)]

try with latets trunk https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=1119a05ee97c9c1817749c2cd82ecd617f7f67cd

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dfa35557a3aa
Performance testing of HTTP3 in Firefox using only xpcshell tests r=michal,sparky

Comment 23

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/dfa35557a3aa