Add a `mach browsertime` command to make it easy to install and invoke browsertime
Categories
(Firefox Build System :: General, enhancement)
Tracking
(firefox68 fixed)
Tracking | Status | |
---|---|---|
firefox68 | --- | fixed |
People
(Reporter: nalexander, Assigned: nalexander)
References
(Blocks 2 open bugs)
Details
Attachments
(2 files)
In the Performance Team we're having a lot of success running browsertime for local performance testing. In particular, we're driving GeckoView-based vehicles with an experimental toolchain developed by nalexander.
This ticket tracks making it easier for our engineers to install and invoke the custom version of browsertime that we're building out.
Assignee | ||
Updated•6 years ago
|
Assignee | ||
Comment 1•6 years ago
|
||
Assignee | ||
Comment 2•6 years ago
|
||
Next steps for this are to make the command install the Visual Metrics dependencies, especially on Windows. I'll get to this ASAP, 'cuz it'll make the whole thing very valuable.
Updated•6 years ago
|
Comment 3•6 years ago
|
||
This would have been a good use case for bug 1526021. I'm not saying we should block or anything, just noting the direction I eventually think we should head for things like this.
Assignee | ||
Comment 4•6 years ago
|
||
Under the hood, browsertime invokes a certain visualmetrics.py
script. That script depends on ffmpeg
and ImageMagick's convert
,
compare
, and mogrify
commands. It also depends on certain Python
packages.
So this installs those dependencies, and then wires up the evaluation
environment such that ./mach browsertime
can find the dependencies.
It also adds a ./mach visualmetrics
command for processing a
captured MP4 file in the same way that browsertime processes such a
file.
In order to avoid downloading dependencies multiple time, the existing
artifact cache is extracted. This is a small first step towards Bug
1526021, which
might want to use this artifact cache as well.
At this time, hashes and filesizes are not verified. During
development, the upstream files changed multiple times, and it's not
worth being completely locked down while experimenting with this
functionality. If we start running this code in automation or in more
sensitive environments, we can build fetch tasks and TC indexes to
streamline the artifact gathering process.
It is expected that a future mach command will want to invoke
browsertime without suffering the overhead of invoking Python (and
mach, which is itself bulky) so a nod is given to exposing the
relevant environment pieces.
During testing, it was discovered that MozillaBuild doesn't ship
git, so that
git repositories can't be used out-of-the-box on Windows. So instead
we use a tarball link from github.com/$USER/$REPO/tarball/$COMMIT-LIKE.
Depends on D26820
Comment 6•6 years ago
|
||
Backed out for build bustages.
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=244614553&repo=autoland&lineNumber=38987
[task 2019-05-04T00:05:38.733Z] 00:05:38 WARNING - ../python/mozbuild/mozbuild/test/test_artifacts.py::TestArtifactCache::test_artifact_cache_persistence TEST-UNEXPECTED-FAIL
[task 2019-05-04T00:05:38.734Z] 00:05:38 INFO - =================================== FAILURES ===================================
[task 2019-05-04T00:05:38.734Z] 00:05:38 INFO - ______________ TestArtifactCache.test_artifact_cache_persistence _______________
[task 2019-05-04T00:05:38.735Z] 00:05:38 INFO - self = <mozbuild.test.test_artifacts.TestArtifactCache testMethod=test_artifact_cache_persistence>
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - def setUp(self):
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - > self.min_cached_artifacts = artifacts.MIN_CACHED_ARTIFACTS
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - E AttributeError: 'module' object has no attribute 'MIN_CACHED_ARTIFACTS'
[task 2019-05-04T00:05:38.736Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_artifacts.py:58: AttributeError
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - =========================== 1 failed in 0.07 seconds ===========================
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - Setting retcode to 1 from /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/test_artifacts.py
[task 2019-05-04T00:05:38.737Z] 00:05:38 INFO - /builds/worker/workspace/build/src/python/mozbuild/mozbuild/test/test_expression.py
[task 2019-05-04T00:05:38.738Z] 00:05:38 INFO - ============================= test session starts ==============================
[task 2019-05-04T00:05:38.738Z] 00:05:38 INFO - platform linux2 -- Python 2.7.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/obj-firefox-8yIyzR8r-2.7/bin/python
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - rootdir: /builds/worker/workspace/build/src, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - collecting ... collected 12 items
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_in PASSED
[task 2019-05-04T00:05:38.739Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_string_literal PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestContext::test_variable PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_defined PASSED
[task 2019-05-04T00:05:38.740Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_equals PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_and PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_ops PASSED
[task 2019-05-04T00:05:38.741Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_logical_or PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_not PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_notequals PASSED
[task 2019-05-04T00:05:38.742Z] 00:05:38 INFO - ../python/mozbuild/mozbuild/test/test_expression.py::TestExpression::test_string_literal PASS
Backout: https://hg.mozilla.org/integration/autoland/rev/ce409badd9d99ebef2b56471d88631a9da5400c1
Comment 8•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/d407071dd793
https://hg.mozilla.org/mozilla-central/rev/0f7f3f92c585
Assignee | ||
Updated•5 years ago
|
Description
•