1330321 - Chunking of tests seem to be broken (some tests are not executed)

Status: RESOLVED INCOMPLETE

(Remote Protocol :: Marionette, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

While working on bug 1311350 I noticed that a couple of new test failures appear in areas which do not have gotten a change at all. In that case it's for test_execute_script.

When you check latest autoland it's all fine and no failures are reported. But it actually should because the chrome test class in that file should produce failures for Fennec each time. 

Now that I added two more tests, the total number of tests have been changed and as such tests are divided differently between the chunks. With that the failures magically appear.

Here the example try run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5a837ea4eeb6&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=67955621

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so all the test failures which have appeared since my above patch has been landed were about using close_chrome_window() in Fennec which produces the UnsupportedOperationError. But adding this assert was done on bug 1326174 and it landed a couple of days ago. So why was this failure not present until now?

Another point is that my patch for bug 1311350 changed the behavior of the close window methods including close_chrome_window(). But that should only be a timing issue for the tests. The assert as mentioned above we call as very first step in Marionette server. So it means due to some reason we are now trying to close chrome windows what we haven't done before.

When I did further checks now if all the tests were run I cannot see the behavior from above anymore. Test results are clearly listed in one of the chunks.

I don't think it makes sense to spend more time on this bug. Lets close it as incomplete, and if something strange happens again we can reopen it.