It used to be the case that at the end of the run I could click the run tests button and have it re-run the tests, even with any changes I made to the test files on OSX and Linux where they are just symlinked. Now clicking the run tests button a second time displays the error about no matching tests.
So this looks to be an issue whose underlying cause is bug 366941. server.js's arrayOfTestFiles() uses |for (var [link, value] in linkArray)|, where linkArray is actually a plain object. The reason it isn't failing consistently is that the JSContext where the relevant server.js script is being evaluated changes versions between the first invocation and the second. In the first invocation, it's 170, and in the second, it's 185. I'm not sure what would be triggering the version change (I did check that it's the same JSContext in both cases).
Created attachment 495358 [details] [diff] [review] patch This fixes the bug so that things work regardless of what version the code is run under.
Comment on attachment 495358 [details] [diff] [review] patch Changing versions on the context sounds sorta bad, do we have a bug on file for that?
Looks like bug 595691 will be taking care of that.