555347 - Completely remove the needs of sleep(0) in Mozmill to get more reliable tests

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Today we noticed a problem in a test for the downloads manager. An element we wanted to click on didn't show any reaction. A way to circumvent this was to add a sleep(0) right before it. After checking the DOM with DOMi I have seen that the nodes are created by JS dynamically. They are not always in the tree. In our test we run a waitForEval before that click, but as it looks like it also blocks some events or in this case the node creation.

Another scenario would be a loop which runs a couple of backend stuff and at some point an event is created which is catched by a listener or observer and causes an UI update. If we directly initiate an user action after the loop, we wouldn't be able to see the UI changes because the event cannot be processed.

Mikeal and Clint, I believe we should also add a sleep(0) call right before any user event. Otherwise we will not be able to fix the above problem. Do you agree?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

I talked with Gavin on IRC today and he was really surprised how we have to handle situations like that in our tests. As said we absolutely need a more robust way for our tests. CC'ing Gavin to that bug in the hope he can give us some more enlightening input.

Raising severity and adding it to our qae-p1 list.

Comment 2

[Mikeal Rogers [:mikeal] (mrogers@)]

I wouldn't try to fix this.

This is all going to severely change when everything goes out of process. I would punt on this and just wait to see how things shake out when you retrofit all of mozmill to work with electrolosis.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Mikeal, just to get an idea about the basics I have two questions. Which code executes the tests? Is it somewhere inside the frame module or do we send it out as a complete chunk to the interpreter in Firefox? Would we be able to add sleep calls directly into the interpreting code? I know that's hacky too but then we would have those sleep(0) calls only at one place.

Comment 4

[cmtalbert]

We're going to review this for 2.0, as we move to E10S, we'll need to refactor heavily in this area so let's do this then.

Comment 5

[Jeff Hammel]

Why is this critical?  Since this hasn't been acted on in awhile, should we put at a priority that actually reflects its likelihood to be attacked and interest in the problem?

Comment 6

[Mikeal Rogers [:mikeal] (mrogers@)]

This bug sounds more like a goal than an action item.

There isn't one reason we have sleep(0), it's a workaround for a couple different bugs and edge cases. It's always indicative of a larger problem but that's usually in the product not in the test framework.

Don't know how ya'll want to handle goals like this, whether they are bugs or not.

Comment 7

[u279076]

Not sure if this adds value to the discussion, but one such "edge-case" is in UtilsAPI.handleWindow():
http://hg.mozilla.org/qa/mozmill-tests/file/f4209042c07c/shared-modules/utils.js#l424

ie. bug 628576

Comment 8

[cmtalbert]

I think this is not something we're going to completely solve in 2.0.  With the changes Henrik and others made to the waitFor* APIs, I think we have a good story here now.  However, there are still sleeps, and in those places where there are sleeps, they are there because we're lacking critical pieces of instrumentation in the platform, as Anthony notes above. We should add that functionality to the platform so that we can work through the issues.  The platform is not immutable and it is not perfect.  If we can find ways to add instrumentation like this that doesn't affect overall performance but improves our testing, then it's a win for everyone.

Can we get an enumeration of where we're sleeping in the code?  Once we have that we will have a much better idea how to fix the platform or fix the tests in question.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.