qm-centos5-moz2-01 and qm-win2k3-moz2-01 failing reftests, crashtests, mochitests

RESOLVED INCOMPLETE

Status

Release Engineering
General
P2
normal
RESOLVED INCOMPLETE
10 years ago
4 years ago

People

(Reporter: rc, Assigned: lsblakk)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 5 obsolete attachments)

(Reporter)

Description

10 years ago
These boxes have been consistently orange since landing them.

see,
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla2/1211383609.1211390504.5331.gz&fulltext=1

and
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla2/1211383609.1211389025.362.gz&fulltext=1

for examples.

Comment 1

10 years ago
Do you suspect misconfiguration or code error, or some gremlin inbetween?

Comment 2

10 years ago
FWIW, it appears that qm-centos5-moz2-01 was green for a while, on 2008-05-14
(Reporter)

Comment 3

10 years ago
(In reply to comment #1)
> Do you suspect misconfiguration or code error, or some gremlin inbetween?

I think these could be code errors or if the failing tests are timing-related, it could be because we're running these on VMs which could be running slower than dedicated hardware. the only consistent machine seems to be the Mac which is running on a mini.
(Reporter)

Comment 4

10 years ago
one of the errors on qm-centos5-moz2-01 is:

REFTEST UNEXPECTED FAIL (LOADING): file:///builds/slave/linux/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png

A loading error suggests the file might be missing - does this have something to do with the changes to the way we're doing binary checkins on hg?
(Reporter)

Updated

10 years ago
Assignee: nobody → rcampbell
Priority: -- → P1

Comment 5

10 years ago
It appears to be there and correct: http://hg.mozilla.org/mozilla-central/index.cgi/file/79924d3b5bba/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png
(Reporter)

Comment 6

10 years ago
hm, yeah. I haven't tried running these tests manually to see what's actually happening, but it seems they should work. The current failure on qm-win2k3-moz2-01 is similar:

REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/layout/reftests/bugs/352980-3c.html

Odd they they're both loading failures and both different.
(Reporter)

Comment 7

10 years ago
and another different one from qm-win2k3-moz2-01 from earlier in the day:

REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s34i3p04.png

matches the above failure on qm-centos5-moz2-01.

I'm going to try clobbering these to see if that clears the problems.
(Reporter)

Comment 8

10 years ago
scratch that. bsmedberg's bumping the reftest timeout which was recently decreased in:

http://hg.mozilla.org/mozilla-central/index.cgi/rev/79fc7037d91e

Comment 9

10 years ago
Is this perhaps a dup/related to bug 431494 and bug 425987? ... need to clobber the profile before each run?
(Reporter)

Comment 10

10 years ago
Created attachment 322280 [details] [diff] [review]
profile clobber patch

a quick and dirty patch to add the --clobber option to the createProfile step in the master.cfg. Ultimately, these command parameters should be moved out of these steps and run properly from the CreateProfile classes in buildbotcustom.
Attachment #322280 - Flags: review?(bhearsum)
(Reporter)

Comment 11

10 years ago
Created attachment 322325 [details] [diff] [review]
[checked in] profile clobber patch redux

args in the right order. This seems to have fixed the problem on moz2.
Attachment #322280 - Attachment is obsolete: true
Attachment #322325 - Flags: review?(bhearsum)
Attachment #322280 - Flags: review?(bhearsum)

Updated

10 years ago
Attachment #322325 - Flags: review?(bhearsum) → review+
(Reporter)

Comment 12

10 years ago
Comment on attachment 322325 [details] [diff] [review]
[checked in] profile clobber patch redux

changeset:   89:386fa565c863
tag:         tip
user:        Rob Campbell <rcampbell@mozilla.com>
date:        Mon May 26 17:04:13 2008 -0300
summary:     bug 435064 - add clobber to createProfile arguments, p=me, r=bhearsum
Attachment #322325 - Attachment description: profile clobber patch redux → [checked in] profile clobber patch redux
(Reporter)

Comment 13

10 years ago
currently we're seeing in Tunit on Linux:

../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: 
command timed out: 2400 seconds without output, killing pid 25293

(see bug 431745 for another example of this)

and in browser chrome,

chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js
	FAIL - Ignore warning button should be present for malware - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js
	FAIL - Timed out - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js

The previous run had completely different failures (in Tunit),

../../../../_tests/xpcshell-simple/test_dm/unit/test_download_manager.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2779 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

and in Mochitest:

*** 13416 ERROR FAIL | Should not be able to navigate popup's popup by calling window.open. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
*** 13417 ERROR FAIL | Should not be able to navigate popup's popup by submitting form. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
*** 13418 ERROR FAIL | Should not be able to navigate popup's popup by targeted hyperlink. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
(Reporter)

Comment 14

10 years ago
most-recent run on centos5, browser chrome:

FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js

(Reporter)

Comment 15

10 years ago
from qm-win2k3-moz2-01 TUnit after a full clobber:

make[3]: Entering directory `/d/builds/slave/win2k3/build/objdir/toolkit/components/downloads/test'
...
../../../../_tests/xpcshell-simple/test_dm/unit/test_bug_401430.js: FAIL

...
*** Throwing trying to get plugin.scan.4xPluginFolder
*** exiting
*** CHECK FAILED: false == true

...
JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/head.js :: anonymous :: line 62
*** FAIL ***


(Reporter)

Comment 16

10 years ago
from a clobbered build on qm-centos5-moz2-01, TUnit:

../../../../_tests/xpcshell-simple/test_dm/unit/test_resume.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2181 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL

a whole bunch similar to:

(process:2181): Gtk-CRITICAL **: _gtk_style_peek_property_value: assertion `GTK_IS_STYLE (style)' failed

…
../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111:  2205 Segmentation fault      (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2
FAIL
(Reporter)

Comment 17

10 years ago
from qm-win2k3-moz2-01 TUnit:

../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js: FAIL

../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js.log:
>>>>>>>
*** test pending
*** exiting
*** CHECK FAILED: new title 1 == new title 2

JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/execute_test.js :: <TOP_LEVEL> :: line 38
2147500036
*** FAIL ***
Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.remove]"  nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame :: ../../../../_tests/xpcshell-simple/test_places/unit/head_bookmarks.js :: clearDB :: line 99"  data: no]
<<<<<<<
(Reporter)

Comment 18

10 years ago
from qm-win2k3-moz2-01 Mochichrome:

*** 420 ERROR FAIL | Test timed out. |  | chrome://mochikit/content/chrome/dom/tests/mochitest/chrome/test_domstorage.xul
(Reporter)

Comment 19

10 years ago
from qm-centos5-03:

REFTEST UNEXPECTED FAIL (LOADING): file:///builds/moz2slave/linux-3/build/layout/reftests/pixel-rounding/background-color-left-5.html
(Reporter)

Comment 20

10 years ago
from qm-centos5-03 (browser chrome):

chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
	PASS - Pref window opened
	PASS - Specified pane was opened
	FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
	FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
(Assignee)

Comment 21

10 years ago
Created attachment 323455 [details]
qm-centos5-moz2-01 errors

This is the data mined from qm-centos5-moz2-01  sorted by build ID

I had to stop at Build 80 because it appears that there was a disruption because of the ssh keys, and there was a lot of cvs errors at that point.  Hopefully this is enough to look at for now, if you need more, let me know.
(Assignee)

Comment 22

10 years ago
Created attachment 323462 [details]
qm-win2k3-moz2-01 errors

and the windows errors.
(Assignee)

Comment 23

10 years ago
Created attachment 323951 [details]
Error parsing script

Put this together to run against a slave dir to look through the logs for errors.  Improvement suggestions are welcome - right now it searched the logs for certain patterns and organizes them by log file name (which happens to start with the build id).
(Assignee)

Comment 24

10 years ago
Created attachment 323952 [details]
Error parsing script - a better version

A more concise version - thanks to Dave Humphrey's tips.
Attachment #323951 - Attachment is obsolete: true

Updated

10 years ago
Blocks: 433384

Updated

10 years ago
No longer blocks: 422754
(Reporter)

Comment 25

10 years ago
nice!

I have one recommendation/request - could you add 2 extra trailing lines of context to the reftest "UNEXPECTED FAIL" section? That way if it's an image comparison failure, we'll get the data URIs with the screen grabs.
(Assignee)

Comment 26

10 years ago
Created attachment 324072 [details]
Error parsing script - now with more context

This one adds two trailing lines of context to UNEXPECTED FAIL for reftests so that the data URI is present.
Attachment #323952 - Attachment is obsolete: true
(Assignee)

Comment 27

10 years ago
Created attachment 324378 [details]
Parse Error script for multiple directories

Newest version of script - this time you can pass in several directories:

./parseErrors.sh dir1 dir2 dir3

and it will put the output for each in a directory called parsed_errors where each output is the name of the directory.output.sorted
Attachment #324072 - Attachment is obsolete: true
(Reporter)

Comment 28

10 years ago
nice improvement. a+
(Assignee)

Comment 29

10 years ago
Created attachment 324493 [details]
Script to gather up errors from buildbot master log files

This is more like it - so apparently sorting was the worst thing I could do.  Now the script greps each pattern match separately and the output is much easier to read over.  Still going to work on turning the error output into sql insert statements so that we can have a simple, light database of errors.
Attachment #324378 - Attachment is obsolete: true
(Assignee)

Updated

10 years ago
Assignee: rcampbell → lukasblakk
Blocks: 438871
Priority: P1 → P2
(Assignee)

Updated

10 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 30

10 years ago
Closing this bug since the intermittent failures not about specific machines, more the flakey tests in general which are being covered by other bugs under bug 438871.
Status: ASSIGNED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
No longer blocks: 433384, 438871
Resolution: FIXED → INCOMPLETE
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.