Add a `mach browsertime` command to make it easy to install and invoke browsertime

RESOLVED FIXED in Firefox 68

Status

enhancement
RESOLVED FIXED
2 months ago
6 hours ago

People

(Reporter: nalexander, Assigned: nalexander)

Tracking

(Depends on 1 bug, Blocks 2 bugs, Regressed 1 bug)

unspecified
mozilla68
Dependency tree / graph

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(2 attachments)

In the Performance Team we're having a lot of success running browsertime for local performance testing. In particular, we're driving GeckoView-based vehicles with an experimental toolchain developed by nalexander.

This ticket tracks making it easier for our engineers to install and invoke the custom version of browsertime that we're building out.

Type: defect → enhancement

Next steps for this are to make the command install the Visual Metrics dependencies, especially on Windows. I'll get to this ASAP, 'cuz it'll make the whole thing very valuable.

Attachment #9057103 - Attachment description: Bug 1543247 - Add `mach browsertime` that installs and invokes browsertime. → Bug 1543247 - Part 1: Add `mach browsertime` command that installs and invokes browsertime. r?Standard8,ahal

This would have been a good use case for bug 1526021. I'm not saying we should block or anything, just noting the direction I eventually think we should head for things like this.

Blocks: 1545627

Under the hood, browsertime invokes a certain visualmetrics.py
script. That script depends on ffmpeg and ImageMagick's convert,
compare, and mogrify commands. It also depends on certain Python
packages.

So this installs those dependencies, and then wires up the evaluation
environment such that ./mach browsertime can find the dependencies.
It also adds a ./mach visualmetrics command for processing a
captured MP4 file in the same way that browsertime processes such a
file.

In order to avoid downloading dependencies multiple time, the existing
artifact cache is extracted. This is a small first step towards Bug
1526021
, which
might want to use this artifact cache as well.

At this time, hashes and filesizes are not verified. During
development, the upstream files changed multiple times, and it's not
worth being completely locked down while experimenting with this
functionality. If we start running this code in automation or in more
sensitive environments, we can build fetch tasks and TC indexes to
streamline the artifact gathering process.

It is expected that a future mach command will want to invoke
browsertime without suffering the overhead of invoking Python (and
mach, which is itself bulky) so a nod is given to exposing the
relevant environment pieces.

During testing, it was discovered that MozillaBuild doesn't ship
git
, so that
git repositories can't be used out-of-the-box on Windows. So instead
we use a tarball link from github.com/$USER/$REPO/tarball/$COMMIT-LIKE.

Depends on D26820

Comment 5

Last month
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4b3619d89abd
Part 1: Add `mach browsertime` command that installs and invokes browsertime. r=Standard8,ahal
https://hg.mozilla.org/integration/autoland/rev/feb726e4f15d
Part 2: Install `visualmetrics.py` prerequisites. r=ahal

Comment 6

Last month

Backed out for build bustages.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=feb726e4f15d761c5e6a03d4cb9ce732f368b257&selectedJob=244614553

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=244614553&repo=autoland&lineNumber=38987

[task 2019-05-04T00:05:38.733Z] 00:05:38 WARNING - ../python/mozbuild/mozbuild/test/test_artifacts.py::TestArtifactCache::test_artifact_cache_persistence TEST-UNEXPECTED-FAIL
[task 2019-05-04T00:05:38.734Z] 00:05:38 INFO - =================================== FAILURES ===================================
[task 2019-05-04T00:05:38.734Z] 00:05:38 INFO - ______________ TestArtifactCache.test_artifact_cache_persistence _______________
[task 2019-05-04T00:05:38.735Z] 00:05:38 INFO - self = <mozbuild.test.test_artifacts.TestArtifactCache testMethod=test_artifact_cache_persistence>
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - def setUp(self):
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - > self.min_cached_artifacts = artifacts.MIN_CACHED_ARTIFACTS
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - E AttributeError: 'module' object has no attribute 'MIN_CACHED_ARTIFACTS'
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_artifacts.py:58: AttributeError
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - =========================== 1 failed in 0.07 seconds ===========================
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - Setting retcode to 1 from /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/test_artifacts.py
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/test_expression.py
[task 2019-05-04T00:05:38.738Z] 00:05:38 INFO - ============================= test session starts ==============================
[task 2019-05-04T00:05:38.738Z] 00:05:38 INFO - platform linux2 -- Python 2.7.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/obj-firefox-8yIyzR8r-2.7/bin/python
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - rootdir: /builds/worker/workspace/build/src, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - collecting ... collected 12 items
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_in PASSED
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_string_literal PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_variable PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_defined PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_equals PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_and PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_ops PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_or PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_not PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_notequals PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_string_literal PASS

Backout: https://hg.mozilla.org/integration/autoland/rev/ce409badd9d99ebef2b56471d88631a9da5400c1

Flags: needinfo?(nalexander)

Comment 7

Last month
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d407071dd793
Part 1: Add `mach browsertime` command that installs and invokes browsertime. r=Standard8,ahal
https://hg.mozilla.org/integration/autoland/rev/0f7f3f92c585
Part 2: Install `visualmetrics.py` prerequisites. r=ahal

Comment 8

Last month
bugherder
Status: ASSIGNED → RESOLVED
Closed: Last month
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Regressions: 1549781

This has landed, so clearing NI.

Flags: needinfo?(nalexander)
Blocks: 1556567
Regressions: 1559168
Depends on: 1559727
You need to log in before you can comment on or make changes to this bug.