417935 - Sisyphus/JavaScript Tests - use restart mode for browser tests

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Bob Clary [:bc] (inactive)]

Too many times since we have switched to non-restart mode, I have had the browser tests fail to complete due to regressions on one or the other branch. This means the entire browser test run will fail if a single test crashes or fails to complete. 

Since we are running these once per day, the browser tests therefore sometimes will only find the one regression whereas the restart mode browser tests would have run to completion and reported on an unlimited number of regressions. In addition, a timeout regression which prevents a single test from completing can result in the entire test run waiting for the full five hours before it times out.

I recommend adding a -R parameter to all invocations of runtests.sh for the browser tests on each branch. This will use restart mode and allow the tests to complete regardless of regressions.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: patch

rob, I just copied the method from class SisyphusJSTest(ShellCommand) into class SisyphusBrowserTest(SisyphusJSTest) thinking that it will inherit everything else and only override the start method.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: patch v2

sorry, I should have realized that since we are doing -R now, we can include the tests which crash. I've also added the crash tests to the shell since they don't care.

Comment 3

[Bob Clary [:bc] (inactive)]

Checking in sisyphus.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/sisyphus/sisyphus.py,v  <--  sisyphus.py
new revision: 1.6; previous revision: 1.5
done

and copied to qm-jstest01.

Thanks!

Comment 4

[Bob Clary [:bc] (inactive)]

I missed the windows parts of this... :-(

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: patch for windows

this adds crashes to the windows shell, and crashes and restart to the windows browser tests.

Comment 6

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 303868 [details] [diff] [review]
patch for windows

yep, that should be fine. I'll find a spot today to move that onto the master.

Comment 7

[Bob Clary [:bc] (inactive)]

/cvsroot/mozilla/tools/buildbot-configs/testing/sisyphus/sisyphus.py,v  <--  sisyphus.py
new revision: 1.7; previous revision: 1.6

I actually copied this to jstest01 and restarted everyone already. No need to do more.

Comment 8

[(not currently active) Ted Mielczarek]

Mass move of Sisyphus bugs to Testing : Sisyphus. Filter on SisyphusMassMove to ignore.