1563090 - Add a visualmetrics task type for computing visualmetrics from captured video

Status: RESOLVED FIXED

(Testing :: Performance, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

For Bug 1563082, we're going to capture lots of videos. Much of that capture will happen on hardware (say, our limited supply of Android devices) or even on special desktop hardware (say reference laptops with capture cards). We want to do the expensive processing with elastic compute, i.e., in AWS; and we want it to be cheap for developers to do this as well without having them run image analysis on their own devices.

This ticket tracks building a task type that moves video files through that pipeline, and potentially a ./mach try ... front-end for processing locally captured videos.

Comment 1

[Beth Rennie [:beth] (she/her)]

:nalexander and I discussed this and came up with the following next steps forward:

1. We need a proper docker image for running visualmetrics.py. The POC that Nick used to demo this was a full image meant for driving a browser vehicle. We only really need Python, the Python libraries that visualmetrics.py requires, and imagemagick.

2. We need infrastructure to make visualmetrics.py available to the docker image. The POC did a full m-c checkout and ran ./mach browsertime -setup to download it, but that is a lot of time and bandwidth to grab the script. Ideally we could grab it from https://github.com/mozilla/browsertime at tagged releases, but for now hard-coding a known good commit SHA into the script will be sufficient.

3. visualmetrics.py needs to be parallelized to support processing multiple videos at once. The calculations on each frame are independent of each other and only need to be collated at the end. The multiprocessing module can be leveraged to implement this.

4. The actual taskcluster task to run this and report the results to perfherder.

Further work would include having a way to submit the captured video from e.g. a capture card and do the visual metrics processing of that video (e.g. a ./mach command to upload the video and create the task).

Did I miss anything, :nalexander?

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Barret Rennie [:brennie] (he/him) from comment #1)

:nalexander and I discussed this and came up with the following next steps forward:

1. We need a proper docker image for running visualmetrics.py. The POC that Nick used to demo this was a full image meant for driving a browser vehicle. We only really need Python, the Python libraries that visualmetrics.py requires, and imagemagick.

Beautiful. You can follow the wpr-go example from https://phabricator.services.mozilla.com/D20567 (or my other try pushes) to see how to make a new Docker image in tree.

2. We need infrastructure to make visualmetrics.py available to the docker image. The POC did a full m-c checkout and ran ./mach browsertime -setup to download it, but that is a lot of time and bandwidth to grab the script. Ideally we could grab it from https://github.com/browsertime at tagged releases, but for now hard-coding a known good commit SHA into the script will be sufficient.

https://github.com/mozilla/browsertime.

3. visualmetrics.py needs to be parallelized to support processing multiple videos at once. The calculations on each frame are independent of each other and only need to be collated at the end. The multiprocessing module can be leveraged to implement this.

Just for the record: I'm not sure we want to do to much work to visualmetrics.py itself; I think we want to run it multiple times and investigate better implementations independently. If that happens in the existing file, fine, but don't be bound to that.

4. The actual taskcluster task to run this and report the results to perfherder.

Further work would include having a way to submit the captured video from e.g. a capture card and do the visual metrics processing of that video (e.g. a ./mach command to upload the video and create the task).

Did I miss anything, :nalexander?

Looks great!

Comment 3

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1563090 - Add a visual metrics treeherder task r?nalexander,tomprince

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1563090 - Part 1: Add visual metrics TC kind.

Comment 5

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1563090 - Fetch visual metrics task definition from a template r?nalexander,tomprince

The ./mach try {fuzzy,chooser} commands now support a --visual-metrics-jobs
option which can be used to pass the job descriptions to the visual-metrics
task.

Depends on D41052

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9071c90b2e84
Add a visual metrics treeherder task r=nalexander,tomprince
https://hg.mozilla.org/integration/autoland/rev/1b2389663474
Fetch visual metrics task definition from a template r=nalexander,tomprince,ahal

Comment 7

[Narcis Beleuzu [:NarcisB]]

Backed out for Linting failure on run-visual-metrics.py

Backout link: https://hg.mozilla.org/integration/autoland/rev/80986a96f9e8f5fa7c3dcc05f3be0c6bc640389a
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=264963430&repo=autoland&lineNumber=220

Comment 8

[Beth Rennie [:beth] (she/her)]

Sorry about that. Fixed in latest revision.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e68340655e1
Add a visual metrics treeherder task r=nalexander,tomprince
https://hg.mozilla.org/integration/autoland/rev/45049852ea48
Fetch visual metrics task definition from a template r=nalexander,tomprince,ahal

Comment 10

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/5e68340655e1
https://hg.mozilla.org/mozilla-central/rev/45049852ea48