Open Bug 1955647 Opened 1 month ago Updated 1 month ago

Side-by-side videos should only be generated for regressing tests

Categories

(Tree Management :: Perfherder, enhancement, P2)

Product:
Tree Management
Component:
Perfherder
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: sparky, Unassigned)

References

Details

(Whiteboard: [fxp])

I've noticed that side-by-side doesn't seem to be properly generated at times. For example in this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1952486

There's a side-by-side generated for the yahoo test, but those are only improvements. We should have generated a side-by-side for the regression on ebay instead.

one ( maybe rare) reason I could think to keep it for improvements is to visually validate a true improvement. e.g. what if the "improvement" actually doesn't fully load all resources like thumbnails, thus resulting in a lower speedindex score and being a fake "improvement"?

I agree, but I think we can generate those when needed since they are rare. Most of the time we need these are for regressions, but in many cases, I only see them get generated for improvements.

You need to log in before you can comment on or make changes to this bug.