Closed Bug 846351 Opened 10 years ago Closed 9 years ago

Mutt test appears to report a Pass when same test under Mozmill reports a Fail

Categories

(Testing Graveyard :: Mozmill, defect, P4)

x86
Windows XP

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: jfrench, Unassigned)

References

()

Details

(Whiteboard: [mozmill-2.1?][mentor=whimboo][lang=py][sprint2013-32])

Attachments

(1 file)

Attached file mozmill console debug
Begat from bug 846007:

"I'm observing that dnd_chrome.js hangs the mutt test run, unless some sort of mouse movement is issued to the system during that particular test. If, in a subsequent mutt run you jiggle the mouse during dnd_chrome, the Mutt test will actually continue and pass."

[1;34mTEST-START[0m | js\controller\dnd_chrome.js | setupModule
[1;34mTEST-START[0m | js\controller\dnd_chrome.js | test
[1;32mTEST-PASS[0m | js\controller\dnd_chrome.js | test *******************

However, when performing the same workflow under Mozmill 2.0rc1 (jiggling the mouse to get the test to continue), the test in Mozmill reports a fail (see attached mozmill console debug showing a failure). 

Henrik requested I enter this as a separate issue, from the above initial bug(that dnd_chrome.js itself is failing).

Hopefully others are able to reproduce this, and it's not just me. If others can, perhaps there is an issue with Mutt's reporting reliability.
Thanks Jon. I think we shouldn't release Mozmill 2 if the test suite we are using is broken regarding js test failures.
Whiteboard: [mozmill-2.0?]
Component: Mozmill Tests → Mozmill
Product: Mozilla QA → Testing
Whiteboard: [mozmill-2.0?] → [mozmill-2.0?][mentor=whimboo][lang=py]
Hi Henrik,

I'm banas from IRC. I've run the mozmill tests and now also written a couple of trivial tests. Can I try this bug? Just to remind you, I currently know Python/C++.

Thank you!
Sarup, you are more than welcome to help out here. Let me know which additional information you need to get started. Thanks!
Assignee: nobody → sbanskota08
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Sarup, please try to run this with the latest code from: https://github.com/mozilla/mozmill The test should probably fail if you have to manually move the mouse to get it to pass (i.e. we can't have people moving mice in automation) So, try running it and let us know what you see, and then we can decide what to do next.

Let us know if you need a hand by dropping into #automation on IRC
Whiteboard: [mozmill-2.0?][mentor=whimboo][lang=py] → [mozmill-2.1?][mentor=whimboo][lang=py]
Sarup, could you confirm that you're still working on this?
Flags: needinfo?(sbanskota08)
Josh, no, you can assign this to someone else.
Flags: needinfo?(sbanskota08)
Assignee: sbanskota08 → nobody
Whiteboard: [mozmill-2.1?][mentor=whimboo][lang=py] → [mozmill-2.1?][mentor=whimboo][lang=py][sprint2013-32]
I see this test Passing and Failing at the same rate while running it with mozmill and mutt.
The rate of failure on OSX is 1 out of 10.
On Linux I see 2-3 out of 10.

On windows I see the original reported problem in bug 846007, but I still have the same outcome with mozmill and mutt. Basically if I don't move the mouse during the test it will timeout, if I move the mouse the test continues to run.

Jonathan, can you please retry with the latest mozmill version (rc03) and see if this still occurs?
Flags: needinfo?(tojonmz)
Yup, will check and report back.
Flags: needinfo?(tojonmz)
The results I see when running dnd_chrome.js directly under mozmill rc3 in the form `mozmill -t mutt/mutt/tests/js/controller/dnd_chrome.js -b (binary)`, while issuing mouse input from the moment the test is invoked - the results are now, at best random. 

I observe all 3 test result types: Pass, Fail and even Skip, during different runs.

That is, if running the mutt test in this way via mozmill is a valid way of running it to compare behaviour.

So to me this implies this bug is no longer applicable (that Mozmill reports a different result to Mutt), and the reliability of this test itself dnd_chrome.js may be suspect (sort of covered by bug 846007). I would favor marking this bug as resolved worksforme, for lack of a better resolution status.
Sounds good. Thanks Jonathan. I agree in getting the test fixed here. We can reopen if we see this issue again after bug 846007 got fixed.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.