666064 - Need a way to run tests occasionally but not for every build

Status: NEW

(Testing :: Reftest, enhancement)

Description

[Mats Palmgren (inactive)]

In bug 654002 I have a reftest that takes 30 seconds to 1 minute in desktop
Opt builds (worst case Android Opt: 109s).
This is too long to run for every build, so I need a way to run this test
occasionally to make sure the bug doesn't regress.

For example, we could introduce a reftest keyword "slow" or "slow(1minute)"
so that the framework can run it only when there are available resources;
or we could let the test writer make the decision: "weekly".

Sorry if this has been discussed before; I couldn't find a bug about it.

Comment 1

[cmtalbert]

I don't really want the test runner script itself trying to ascertain "current resource load" of the overall infrastructure.  And almost all of the problems in our end-to-end time are not diagnosable from information available to a single slave in the automation pool.  

Instead of doing it within the harness, I'd prefer to add tests like this to another manifest.  Then we can change the high level test-automation infrastructure to run a reftest job with the "slow" manifest every 5 checkins (or on some regular schedule that makes sense).