938644 - Add web latency benchmark to talos

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[William Lachance (:wlach)]

Google's web latency benchmark looks really useful, measuring jank while performing a number of common browser operations:

http://google.github.io/latency-benchmark/
https://github.com/google/latency-benchmark

(the latter link describes how it works)

IMO we should consider making Talos run these tests. It appears that the test requires a small amount of platform-specific compiled code to work, so this wouldn't be 100% trivial to deploy, but it does seem to provide a headless (i.e. scalable) way of measuring latency and jank on the desktop -- which we really haven't had success in doing up to this point (we had written peptest in the past, but it didn't seem to be successful in providing reproducible or actionable data).

Comment 1

[Vladan Djeric (:vladan)]

I filed bug 935680 with a very similar idea

Comment 2

[Olivier Yiptong [:oyiptong]]

Looks like this benchmark may fit the bill.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: initial prototype of how we could run this in talos (0.5)

this requires a forked version of latency-benchmark:
https://github.com/jmaher/latency-benchmark

obviously some more work to be done, but we are getting closer.  I will probably just keep a collection of binaries in talos and we can reference them.  Open to other suggestions.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: initial prototype of how we could run this in talos (0.6)

Updates to allow for posting from the latency-benchmark tool as updated in my recent push:
https://github.com/jmaher/latency-benchmark

Once those changes are upstreamed, we can work on landing talos changes as needed to make this work.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add latency benchmark to talos (1.0)

will update this for review once I have tested this in various configurations and ensure stability in the numbers.

Comment 6

[Avi Halachmi (:avih)]

Tried the patch from comment 5 locally on OS X 10.9.1 on a 2010 MBA. Some issues I noticed:

1. On first run, the OS pops a firewall dialog, asking if "latency-benchmark" is allowed to accept incoming connections. Until this dialog is released, the test is hanged. Confirming the dialog once prevented it from popping up the next time I tried. Not sure what will happen if the binary changes.

2. Running from console, the Firefox window didn't appear on top, but rather behind the console window. The test failed.

3. Even if I minimized the console right after issueing the talos cli, long time before the Firefox window spawned, the Firefox window still wasn't focused when it showed, so it couldn't accept KB events which are part of the test, and the test failed.

4. I clicked the Firefox window as soon as it showed, so it got focus and the test ran and completed successfully (the browser window displayed all the 6 tests results)
. However, the console (talos) didn't display any results, and did display this message: "ERROR : need more than 2 values to unpack".

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

The toolchain for latency-benchmark is setup to build linux64, would we want this on linux32 and linux64?  I ask as it will take a bit of work to get this compiled to 32 bit (some library dependencies) and we would need an additional binary.  I am just trying to prioritize work items.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: updated patch with 32/64 bit linux binary (1.1)

updated patch to apply to latest talos along with updated linux binary.  The linux binary works on 32 bit and doesn't have oculus.  Unfortunately this doesn't work on try server for linux, I suspect one of the tests are failing.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

closing out old bugs that haven't been a priority