run-visual-metrics.py assumes that each video will have a different browsertime.json
Categories
(Testing :: Performance, defect, P1)
Tracking
(firefox75 fixed)
| Tracking | Status | |
|---|---|---|
| firefox75 | --- | fixed |
People
(Reporter: barret, Assigned: barret)
References
Details
Attachments
(1 file)
The script running the visual metrics was expecting the jobs.json to list jobs with different browsertime.json files. It presently extracts the first video path from the list of videos and uses that. However, when generating multiple videos with e.g., -n 15, this will result in processing the same video repeatedly.
The issue is this code.
However, since we are not downloading browsertime.json or video files anymore (as they are coming in a fetch task), we should switch to using just the browsertime.json files and do away with the jobs.json files.
|
Assignee | |
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 1•4 years ago
|
||
The run-visual-metrics.py script was intended to consume a jobs.json file
containing one browsertime.json per video. However it was not being used as
such and was continuously re-processing the first video specified in the
browsertime.json file. If a job were submitted with a browsertime.json
containing 15 videos and 15 different videos, only the first would be
processed. This leads to us having incorrect metrics because over all runs all
the metrics will be identical.
Now we only specify the browsertime.json in the jobs.json file and extract
the paths to videos from there. Also because we are never downloading inputs
this way, we get to remove some dead code and our dependency on requests.
Pushed by brennie@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/1e26514f1dc3 Do not specifiy individual videos in jobs.json r=tarek,perftest-reviewers,sparky
|
||
Comment 3•4 years ago
|
||
| bugherder | ||
Description
•