Closed Bug 445876 Opened 12 years ago Closed 1 year ago

Tracking bug for mochitest failures

Categories

(Firefox for Android Graveyard :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: harth, Unassigned)

References

Details

(Keywords: meta)

Attachments

(8 files)

Tracking bug for Mochitest failures on Fennec 

==Steps to reproduce==
http://wiki.mozilla.org/Mobile/Fennec_Mochitest


Log output of failures on Ubuntu desktop build is attached as well as a a list of just the file names of the tests that failed. Running these tests on N800 is buggy (running into JS engine assertion failures, seg faults, and hangs where there should be timeouts - but will look into these). 

Many of these failures (especially the timeouts) seem to be due to the disabling of new windows/tabs. 

==Overall stats==
Passed: 48268
Failed: 1748
Todo: 1132
Flags: blocking-fennec1.0+
Priority: -- → P2
Target Milestone: --- → Fennec A2
Attached file failed tests on N800
failed test files for N800. Had to run tests in small directories and sometimes a directory would hang or die, noted by DIRECTORY ABORTED in log file.
heather: are these recent lists all of the tests that are failing now? (possibly more, due to the directory aborting)
The N800 results are on a build from 1-2 weeks ago. I tried a desktop build about a week ago and the same tests failed. Don't have a N800 or Ubuntu anymore, but Clint might try these out. I will try them on a Windows desktop build. 
sorry, misread. Yes, they are all the tests that failed, but the directories that aborted could mean a lot more (like the widgets directory). 
Failures on a Windows XP desktop build:

Passed: 61288
Failed: 1741
Todo:   1632
failed test files on Windows desktop build
This is a build log with the failed tests from a build on the Ubuntu desktop (Ubuntu 7.1) and it once again timed out in docshell/test/navigation/ area.

Heather and I tried to pull a build on the same day. She's going to run it on windows desktop and report her results to see if they are different.

Once again, I made it through 22,000 tests, and had around 260 failures.
Anyone running these tests is probably going to have to change the limit on the number of timeouts in _tests/testing/mochitest/tests/SimpleTest/TestRunner.js#16 to something really high. I'll run these tonight from what I pulled yesterday.
Comparing these runs (between windows desktop and ubuntu desktop), we do have some differences, but they aren't as bad as I first thought they might be.

* content/html/content/test/test_bug300691-1.html fails two extra cases on windows than it does on Ubuntu
* content/html/document/test/test_bug199692.html fails *ONLY* on Windows
* content/html/document/test/test_bug369370.html fails *ONLY* on Windows
* content/html/document/test_test_bug391777.html fails *ONLY* on Ubuntu

And the comparison ends here because the windows run doesn't time out. The Ubuntu run times out in docshell/test/navigation/test_bug13871.html.  The Windows build also times out in this test, but that timeout doesn't kill the test run.

This is promising because it looks like the runs are pretty deterministic. A few slight differences between the platforms, but largely they are the same.

@Heather: what did you set your TestRunner.js number of timeouts to for this run?
Keywords: meta
Target Milestone: Fennec A2 → ---
tracking-fennec: --- → 1.0+
tracking-fennec: 1.0+ → ---
Flags: blocking-fennec1.0+ → wanted-fennec1.0+
Depends on: 585178
Closing all opened bug in a graveyard component
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.