807471 - Document how third parties can run reftest packages, and ensure it's as easy as possible

Status: NEW

(Testing :: Reftest, defect)

Description

[Benoit Jacob [:bjacob] (mostly away)]

We are having conversations with GPU vendors about how they can ensure that their drivers don't have regressions that affect Firefox. From our perspective, the best thing that could happen is that they'd integrate our tests into their own testing.

Some tests don't require Chrome privileges (e.g. canvas and webgl tests). These are already very easy for them to run without assistance from us.

Reftest is more tricky because it requires Chrome privileges and therefore can't just be run by pointing a browser to a URL.

From a conversation with Jonathan it seemed that the best approach was for third parties to download pre-built Firefoxen and reftest packages and run them --- much like we do on our own test slaves.

To that effect we need:
  - Very easy and fool-proof documentation on how to do that.
  - Ensure that this can be completely automated. E.g. if they have to download a new file every day, the URL of that download should be predictable.
  - If anything in that procedure can be simplified, it's worth the effort.

If getting this to work on all platforms is tricky, then please prioritize Windows.

Tentatively assigning to Jonathan.