Tracking bug for mochitest failures

NEW
Unassigned

Status

Fennec Graveyard
General
P2
normal
10 years ago
8 years ago

People

(Reporter: harth, Unassigned)

Tracking

(Depends on: 1 bug, {meta})

Bug Flags:
wanted-fennec1.0 +

Details

Attachments

(8 attachments)

(Reporter)

Description

10 years ago
Created attachment 330121 [details]
failed tests on ubuntu build

Tracking bug for Mochitest failures on Fennec 

==Steps to reproduce==
http://wiki.mozilla.org/Mobile/Fennec_Mochitest


Log output of failures on Ubuntu desktop build is attached as well as a a list of just the file names of the tests that failed. Running these tests on N800 is buggy (running into JS engine assertion failures, seg faults, and hangs where there should be timeouts - but will look into these). 

Many of these failures (especially the timeouts) seem to be due to the disabling of new windows/tabs. 

==Overall stats==
Passed: 48268
Failed: 1748
Todo: 1132
(Reporter)

Comment 1

10 years ago
Created attachment 330122 [details]
failed test files on ubuntu build

Updated

10 years ago
Flags: blocking-fennec1.0+
Priority: -- → P2
Target Milestone: --- → Fennec A2
(Reporter)

Comment 2

10 years ago
Created attachment 334040 [details]
failed tests on N800

failed test files for N800. Had to run tests in small directories and sometimes a directory would hang or die, noted by DIRECTORY ABORTED in log file.
(Reporter)

Comment 3

10 years ago
Created attachment 334041 [details]
failed test files on N800

Comment 4

10 years ago
heather: are these recent lists all of the tests that are failing now? (possibly more, due to the directory aborting)
(Reporter)

Comment 5

10 years ago
The N800 results are on a build from 1-2 weeks ago. I tried a desktop build about a week ago and the same tests failed. Don't have a N800 or Ubuntu anymore, but Clint might try these out. I will try them on a Windows desktop build. 
(Reporter)

Comment 6

10 years ago
sorry, misread. Yes, they are all the tests that failed, but the directories that aborted could mean a lot more (like the widgets directory). 
(Reporter)

Comment 7

10 years ago
Created attachment 337889 [details]
failed tests on Windows build

Failures on a Windows XP desktop build:

Passed: 61288
Failed: 1741
Todo:   1632
(Reporter)

Comment 8

10 years ago
Created attachment 337890 [details]
failed test files on Windows build

failed test files on Windows desktop build

Comment 9

10 years ago
Created attachment 338394 [details]
This is a run from a build pulled on Thurs, sept 11.

This is a build log with the failed tests from a build on the Ubuntu desktop (Ubuntu 7.1) and it once again timed out in docshell/test/navigation/ area.

Heather and I tried to pull a build on the same day. She's going to run it on windows desktop and report her results to see if they are different.

Once again, I made it through 22,000 tests, and had around 260 failures.
(Reporter)

Comment 10

10 years ago
Anyone running these tests is probably going to have to change the limit on the number of timeouts in _tests/testing/mochitest/tests/SimpleTest/TestRunner.js#16 to something really high. I'll run these tonight from what I pulled yesterday.
(Reporter)

Comment 11

10 years ago
Created attachment 339074 [details]
failures from 09/11/08 - Windows

Comment 12

10 years ago
Comparing these runs (between windows desktop and ubuntu desktop), we do have some differences, but they aren't as bad as I first thought they might be.

* content/html/content/test/test_bug300691-1.html fails two extra cases on windows than it does on Ubuntu
* content/html/document/test/test_bug199692.html fails *ONLY* on Windows
* content/html/document/test/test_bug369370.html fails *ONLY* on Windows
* content/html/document/test_test_bug391777.html fails *ONLY* on Ubuntu

And the comparison ends here because the windows run doesn't time out. The Ubuntu run times out in docshell/test/navigation/test_bug13871.html.  The Windows build also times out in this test, but that timeout doesn't kill the test run.

This is promising because it looks like the runs are pretty deterministic. A few slight differences between the platforms, but largely they are the same.

@Heather: what did you set your TestRunner.js number of timeouts to for this run?

Updated

9 years ago
Keywords: meta
Target Milestone: Fennec A2 → ---

Updated

9 years ago
tracking-fennec: --- → 1.0+

Updated

9 years ago
tracking-fennec: 1.0+ → ---
Flags: blocking-fennec1.0+ → wanted-fennec1.0+
Depends on: 585178
You need to log in before you can comment on or make changes to this bug.