Closed Bug 435064 Opened 12 years ago Closed 12 years ago

qm-centos5-moz2-01 and qm-win2k3-moz2-01 failing reftests, crashtests, mochitests

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: rcampbell, Assigned: lsblakk)

Details

Attachments

(4 files, 5 obsolete files)

Do you suspect misconfiguration or code error, or some gremlin inbetween?
FWIW, it appears that qm-centos5-moz2-01 was green for a while, on 2008-05-14
(In reply to comment #1)
> Do you suspect misconfiguration or code error, or some gremlin inbetween?

I think these could be code errors or if the failing tests are timing-related, it could be because we're running these on VMs which could be running slower than dedicated hardware. the only consistent machine seems to be the Mac which is running on a mini.
one of the errors on qm-centos5-moz2-01 is:

REFTEST UNEXPECTED FAIL (LOADING): file:///builds/slave/linux/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png

A loading error suggests the file might be missing - does this have something to do with the changes to the way we're doing binary checkins on hg?
Assignee: nobody → rcampbell
Priority: -- → P1
hm, yeah. I haven't tried running these tests manually to see what's actually happening, but it seems they should work. The current failure on qm-win2k3-moz2-01 is similar:

REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/layout/reftests/bugs/352980-3c.html

Odd they they're both loading failures and both different.
and another different one from qm-win2k3-moz2-01 from earlier in the day:

REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s34i3p04.png

matches the above failure on qm-centos5-moz2-01.

I'm going to try clobbering these to see if that clears the problems.
scratch that. bsmedberg's bumping the reftest timeout which was recently decreased in:

http://hg.mozilla.org/mozilla-central/index.cgi/rev/79fc7037d91e
Is this perhaps a dup/related to bug 431494 and bug 425987? ... need to clobber the profile before each run?
Attached patch profile clobber patch (obsolete) — Splinter Review
a quick and dirty patch to add the --clobber option to the createProfile step in the master.cfg. Ultimately, these command parameters should be moved out of these steps and run properly from the CreateProfile classes in buildbotcustom.
Attachment #322280 - Flags: review?(bhearsum)
args in the right order. This seems to have fixed the problem on moz2.
Attachment #322280 - Attachment is obsolete: true
Attachment #322325 - Flags: review?(bhearsum)
Attachment #322280 - Flags: review?(bhearsum)
Attachment #322325 - Flags: review?(bhearsum) → review+
Comment on attachment 322325 [details] [diff] [review]
[checked in] profile clobber patch redux

changeset:   89:386fa565c863
tag:         tip
user:        Rob Campbell <rcampbell@mozilla.com>
date:        Mon May 26 17:04:13 2008 -0300
summary:     bug 435064 - add clobber to createProfile arguments, p=me, r=bhearsum
Attachment #322325 - Attachment description: profile clobber patch redux → [checked in] profile clobber patch redux
currently we're seeing in Tunit on Linux:

../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: 
command timed out: 2400 seconds without output, killing pid 25293

(see bug 431745 for another example of this)

and in browser chrome,

chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js
	FAIL - Ignore warning button should be present for malware - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js
	FAIL - Timed out - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js

The previous run had completely different failures (in Tunit),

../../../../_tests/xpcshell-simple/test_dm/unit/test_download_manager.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2779 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

and in Mochitest:

*** 13416 ERROR FAIL | Should not be able to navigate popup's popup by calling window.open. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
*** 13417 ERROR FAIL | Should not be able to navigate popup's popup by submitting form. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
*** 13418 ERROR FAIL | Should not be able to navigate popup's popup by targeted hyperlink. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
most-recent run on centos5, browser chrome:

FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js

from qm-win2k3-moz2-01 TUnit after a full clobber:

make[3]: Entering directory `/d/builds/slave/win2k3/build/objdir/toolkit/components/downloads/test'
...
../../../../_tests/xpcshell-simple/test_dm/unit/test_bug_401430.js: FAIL

...
*** Throwing trying to get plugin.scan.4xPluginFolder
*** exiting
*** CHECK FAILED: false == true

...
JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/head.js :: anonymous :: line 62
*** FAIL ***


from a clobbered build on qm-centos5-moz2-01, TUnit:

../../../../_tests/xpcshell-simple/test_dm/unit/test_resume.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2181 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

a whole bunch similar to:

(process:2181): Gtk-CRITICAL **: _gtk_style_peek_property_value: assertion `GTK_IS_STYLE (style)' failed

…
../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2205 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL
from qm-win2k3-moz2-01 TUnit:

../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js: FAIL

../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js.log:
>>>>>>>
*** test pending
*** exiting
*** CHECK FAILED: new title 1 == new title 2

JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/execute_test.js :: <TOP_LEVEL> :: line 38
2147500036
*** FAIL ***
Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.remove]"  nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame :: ../../../../_tests/xpcshell-simple/test_places/unit/head_bookmarks.js :: clearDB :: line 99"  data: no]
<<<<<<<
from qm-win2k3-moz2-01 Mochichrome:

*** 420 ERROR FAIL | Test timed out. |  | chrome://mochikit/content/chrome/dom/tests/mochitest/chrome/test_domstorage.xul
from qm-centos5-03:

REFTEST UNEXPECTED FAIL (LOADING): file:///builds/moz2slave/linux-3/build/layout/reftests/pixel-rounding/background-color-left-5.html
from qm-centos5-03 (browser chrome):

chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
	PASS - Pref window opened
	PASS - Specified pane was opened
	FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
	FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
This is the data mined from qm-centos5-moz2-01  sorted by build ID

I had to stop at Build 80 because it appears that there was a disruption because of the ssh keys, and there was a lot of cvs errors at that point.  Hopefully this is enough to look at for now, if you need more, let me know.
and the windows errors.
Attached file Error parsing script (obsolete) —
Put this together to run against a slave dir to look through the logs for errors.  Improvement suggestions are welcome - right now it searched the logs for certain patterns and organizes them by log file name (which happens to start with the build id).
A more concise version - thanks to Dave Humphrey's tips.
Attachment #323951 - Attachment is obsolete: true
No longer blocks: 422754
nice!

I have one recommendation/request - could you add 2 extra trailing lines of context to the reftest "UNEXPECTED FAIL" section? That way if it's an image comparison failure, we'll get the data URIs with the screen grabs.
This one adds two trailing lines of context to UNEXPECTED FAIL for reftests so that the data URI is present.
Attachment #323952 - Attachment is obsolete: true
Newest version of script - this time you can pass in several directories:

./parseErrors.sh dir1 dir2 dir3

and it will put the output for each in a directory called parsed_errors where each output is the name of the directory.output.sorted
Attachment #324072 - Attachment is obsolete: true
nice improvement. a+
This is more like it - so apparently sorting was the worst thing I could do.  Now the script greps each pattern match separately and the output is much easier to read over.  Still going to work on turning the error output into sql insert statements so that we can have a simple, light database of errors.
Attachment #324378 - Attachment is obsolete: true
Assignee: rcampbell → lukasblakk
Blocks: 438871
Priority: P1 → P2
Status: NEW → ASSIGNED
Closing this bug since the intermittent failures not about specific machines, more the flakey tests in general which are being covered by other bugs under bug 438871.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
No longer blocks: 433384, 438871
Resolution: FIXED → INCOMPLETE
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.