1543247 - Add a `mach browsertime` command to make it easy to install and invoke browsertime

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

In the Performance Team we're having a lot of success running browsertime for local performance testing. In particular, we're driving GeckoView-based vehicles with an experimental toolchain developed by nalexander.

This ticket tracks making it easier for our engineers to install and invoke the custom version of browsertime that we're building out.

Comment 1

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1543247 - Part 1: Add `mach browsertime` command that installs and invokes browsertime. r?Standard8,ahal

Comment 2

[Nick Alexander :nalexander [he/him]]

Next steps for this are to make the command install the Visual Metrics dependencies, especially on Windows. I'll get to this ASAP, 'cuz it'll make the whole thing very valuable.

Comment 3

[Andrew Halberstadt [:ahal]]

This would have been a good use case for bug 1526021. I'm not saying we should block or anything, just noting the direction I eventually think we should head for things like this.

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1543247 - Part 2: Install `visualmetrics.py` prerequisites. r?ahal

Under the hood, browsertime invokes a certain visualmetrics.py
script. That script depends on ffmpeg and ImageMagick's convert,
compare, and mogrify commands. It also depends on certain Python
packages.

So this installs those dependencies, and then wires up the evaluation
environment such that ./mach browsertime can find the dependencies.
It also adds a ./mach visualmetrics command for processing a
captured MP4 file in the same way that browsertime processes such a
file.

In order to avoid downloading dependencies multiple time, the existing
artifact cache is extracted. This is a small first step towards Bug
1526021, which
might want to use this artifact cache as well.

At this time, hashes and filesizes are not verified. During
development, the upstream files changed multiple times, and it's not
worth being completely locked down while experimenting with this
functionality. If we start running this code in automation or in more
sensitive environments, we can build fetch tasks and TC indexes to
streamline the artifact gathering process.

It is expected that a future mach command will want to invoke
browsertime without suffering the overhead of invoking Python (and
mach, which is itself bulky) so a nod is given to exposing the
relevant environment pieces.

During testing, it was discovered that MozillaBuild doesn't ship
git, so that
git repositories can't be used out-of-the-box on Windows. So instead
we use a tarball link from github.com/$USER/$REPO/tarball/$COMMIT-LIKE.

Depends on D26820

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4b3619d89abd
Part 1: Add `mach browsertime` command that installs and invokes browsertime. r=Standard8,ahal
https://hg.mozilla.org/integration/autoland/rev/feb726e4f15d
Part 2: Install `visualmetrics.py` prerequisites. r=ahal

Comment 6

[Razvan Maries]

Backed out for build bustages.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=feb726e4f15d761c5e6a03d4cb9ce732f368b257&selectedJob=244614553

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=244614553&repo=autoland&lineNumber=38987

[task 2019-05-04T00:05:38.733Z] 00:05:38 WARNING - ../python/mozbuild/mozbuild/test/test_artifacts.py::TestArtifactCache::test_artifact_cache_persistence TEST-UNEXPECTED-FAIL
[task 2019-05-04T00:05:38.734Z] 00:05:38 INFO - =================================== FAILURES ===================================
[task 2019-05-04T00:05:38.734Z] 00:05:38 INFO - ______________ TestArtifactCache.test_artifact_cache_persistence _______________
[task 2019-05-04T00:05:38.735Z] 00:05:38 INFO - self = <mozbuild.test.test_artifacts.TestArtifactCache testMethod=test_artifact_cache_persistence>
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - def setUp(self):
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - > self.min_cached_artifacts = artifacts.MIN_CACHED_ARTIFACTS
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - E AttributeError: 'module' object has no attribute 'MIN_CACHED_ARTIFACTS'
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_artifacts.py:58: AttributeError
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - =========================== 1 failed in 0.07 seconds ===========================
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - Setting retcode to 1 from /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/test_artifacts.py
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/test_expression.py
[task 2019-05-04T00:05:38.738Z] 00:05:38 INFO - ============================= test session starts ==============================
[task 2019-05-04T00:05:38.738Z] 00:05:38 INFO - platform linux2 -- Python 2.7.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/obj-firefox-8yIyzR8r-2.7/bin/python
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - rootdir: /builds/worker/workspace/build/src, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - collecting ... collected 12 items
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_in PASSED
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_string_literal PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_variable PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_defined PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_equals PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_and PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_ops PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_or PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_not PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_notequals PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_string_literal PASS

Backout: https://hg.mozilla.org/integration/autoland/rev/ce409badd9d99ebef2b56471d88631a9da5400c1

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d407071dd793
Part 1: Add `mach browsertime` command that installs and invokes browsertime. r=Standard8,ahal
https://hg.mozilla.org/integration/autoland/rev/0f7f3f92c585
Part 2: Install `visualmetrics.py` prerequisites. r=ahal

Comment 8

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/d407071dd793
https://hg.mozilla.org/mozilla-central/rev/0f7f3f92c585

Comment 9

[Nick Alexander :nalexander [he/him]]

This has landed, so clearing NI.