435064 - qm-centos5-moz2-01 and qm-win2k3-moz2-01 failing reftests, crashtests, mochitests

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P2)

Description

[Rob Campbell [:rc] (:robcee)]

These boxes have been consistently orange since landing them.

see,
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla2/1211383609.1211390504.5331.gz&fulltext=1

and
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla2/1211383609.1211389025.362.gz&fulltext=1

for examples.

Comment 1

[Benjamin Smedberg]

Do you suspect misconfiguration or code error, or some gremlin inbetween?

Comment 2

[Benjamin Smedberg]

FWIW, it appears that qm-centos5-moz2-01 was green for a while, on 2008-05-14

Comment 3

[Rob Campbell [:rc] (:robcee)]

(In reply to comment #1)
> Do you suspect misconfiguration or code error, or some gremlin inbetween?

I think these could be code errors or if the failing tests are timing-related, it could be because we're running these on VMs which could be running slower than dedicated hardware. the only consistent machine seems to be the Mac which is running on a mini.

Comment 4

[Rob Campbell [:rc] (:robcee)]

one of the errors on qm-centos5-moz2-01 is:

REFTEST UNEXPECTED FAIL (LOADING): file:///builds/slave/linux/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png

A loading error suggests the file might be missing - does this have something to do with the changes to the way we're doing binary checkins on hg?

Comment 5

[Benjamin Smedberg]

It appears to be there and correct: http://hg.mozilla.org/mozilla-central/index.cgi/file/79924d3b5bba/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png

Comment 6

[Rob Campbell [:rc] (:robcee)]

hm, yeah. I haven't tried running these tests manually to see what's actually happening, but it seems they should work. The current failure on qm-win2k3-moz2-01 is similar:

REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/layout/reftests/bugs/352980-3c.html

Odd they they're both loading failures and both different.

Comment 7

[Rob Campbell [:rc] (:robcee)]

and another different one from qm-win2k3-moz2-01 from earlier in the day:

REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s34i3p04.png

matches the above failure on qm-centos5-moz2-01.

I'm going to try clobbering these to see if that clears the problems.

Comment 8

[Rob Campbell [:rc] (:robcee)]

scratch that. bsmedberg's bumping the reftest timeout which was recently decreased in:

http://hg.mozilla.org/mozilla-central/index.cgi/rev/79fc7037d91e

Comment 9

[Benjamin Smedberg]

Is this perhaps a dup/related to bug 431494 and bug 425987? ... need to clobber the profile before each run?

Comment 10

[Rob Campbell [:rc] (:robcee)]

Attachment: profile clobber patch

a quick and dirty patch to add the --clobber option to the createProfile step in the master.cfg. Ultimately, these command parameters should be moved out of these steps and run properly from the CreateProfile classes in buildbotcustom.

Comment 11

[Rob Campbell [:rc] (:robcee)]

Attachment: [checked in] profile clobber patch redux

args in the right order. This seems to have fixed the problem on moz2.

Comment 12

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 322325 [details] [diff] [review]
[checked in] profile clobber patch redux

changeset:   89:386fa565c863
tag:         tip
user:        Rob Campbell <[e-mail]>
date:        Mon May 26 17:04:13 2008 -0300
summary:     bug 435064 - add clobber to createProfile arguments, p=me, r=bhearsum

Comment 13

[Rob Campbell [:rc] (:robcee)]

currently we're seeing in Tunit on Linux:

../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: 
command timed out: 2400 seconds without output, killing pid 25293

(see bug 431745 for another example of this)

and in browser chrome,

chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js
	FAIL - Ignore warning button should be present for malware - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js
	FAIL - Timed out - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js

The previous run had completely different failures (in Tunit),

../../../../_tests/xpcshell-simple/test_dm/unit/test_download_manager.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2779 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

and in Mochitest:

*** 13416 ERROR FAIL | Should not be able to navigate popup's popup by calling window.open. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
*** 13417 ERROR FAIL | Should not be able to navigate popup's popup by submitting form. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
*** 13418 ERROR FAIL | Should not be able to navigate popup's popup by targeted hyperlink. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html

Comment 14

[Rob Campbell [:rc] (:robcee)]

most-recent run on centos5, browser chrome:

FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js

Comment 15

[Rob Campbell [:rc] (:robcee)]

from qm-win2k3-moz2-01 TUnit after a full clobber:

make[3]: Entering directory `/d/builds/slave/win2k3/build/objdir/toolkit/components/downloads/test'
...
../../../../_tests/xpcshell-simple/test_dm/unit/test_bug_401430.js: FAIL

...
*** Throwing trying to get plugin.scan.4xPluginFolder
*** exiting
*** CHECK FAILED: false == true

...
JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/head.js :: anonymous :: line 62
*** FAIL ***

Comment 16

[Rob Campbell [:rc] (:robcee)]

from a clobbered build on qm-centos5-moz2-01, TUnit:

../../../../_tests/xpcshell-simple/test_dm/unit/test_resume.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2181 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

a whole bunch similar to:

(process:2181): Gtk-CRITICAL **: _gtk_style_peek_property_value: assertion `GTK_IS_STYLE (style)' failed

…
../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2205 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

Comment 17

[Rob Campbell [:rc] (:robcee)]

from qm-win2k3-moz2-01 TUnit:

../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js: FAIL

../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js.log:
>>>>>>>
*** test pending
*** exiting
*** CHECK FAILED: new title 1 == new title 2

JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/execute_test.js :: <TOP_LEVEL> :: line 38
2147500036
*** FAIL ***
Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.remove]"  nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame :: ../../../../_tests/xpcshell-simple/test_places/unit/head_bookmarks.js :: clearDB :: line 99"  data: no]
<<<<<<<

Comment 18

[Rob Campbell [:rc] (:robcee)]

from qm-win2k3-moz2-01 Mochichrome:

*** 420 ERROR FAIL | Test timed out. |  | chrome://mochikit/content/chrome/dom/tests/mochitest/chrome/test_domstorage.xul

Comment 19

[Rob Campbell [:rc] (:robcee)]

from qm-centos5-03:

REFTEST UNEXPECTED FAIL (LOADING): file:///builds/moz2slave/linux-3/build/layout/reftests/pixel-rounding/background-color-left-5.html

Comment 20

[Rob Campbell [:rc] (:robcee)]

from qm-centos5-03 (browser chrome):

chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
	PASS - Pref window opened
	PASS - Specified pane was opened
	FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
	FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: qm-centos5-moz2-01 errors

This is the data mined from qm-centos5-moz2-01  sorted by build ID

I had to stop at Build 80 because it appears that there was a disruption because of the ssh keys, and there was a lot of cvs errors at that point.  Hopefully this is enough to look at for now, if you need more, let me know.

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: qm-win2k3-moz2-01 errors

and the windows errors.

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Error parsing script

Put this together to run against a slave dir to look through the logs for errors.  Improvement suggestions are welcome - right now it searched the logs for certain patterns and organizes them by log file name (which happens to start with the build id).

Comment 24

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Error parsing script - a better version

A more concise version - thanks to Dave Humphrey's tips.

Comment 25

[Rob Campbell [:rc] (:robcee)]

nice!

I have one recommendation/request - could you add 2 extra trailing lines of context to the reftest "UNEXPECTED FAIL" section? That way if it's an image comparison failure, we'll get the data URIs with the screen grabs.

Comment 26

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Error parsing script - now with more context

This one adds two trailing lines of context to UNEXPECTED FAIL for reftests so that the data URI is present.

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Parse Error script for multiple directories

Newest version of script - this time you can pass in several directories:

./parseErrors.sh dir1 dir2 dir3

and it will put the output for each in a directory called parsed_errors where each output is the name of the directory.output.sorted

Comment 28

[Rob Campbell [:rc] (:robcee)]

nice improvement. a+

Comment 29

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: Script to gather up errors from buildbot master log files

This is more like it - so apparently sorting was the worst thing I could do.  Now the script greps each pattern match separately and the output is much easier to read over.  Still going to work on turning the error output into sql insert statements so that we can have a simple, light database of errors.

Comment 30

[Lukas Blakk [:lsblakk] use ?needinfo]

Closing this bug since the intermittent failures not about specific machines, more the flakey tests in general which are being covered by other bugs under bug 438871.