1.65 KB, patch
|Details | Diff | Splinter Review|
27.88 KB, text/plain
18.35 KB, text/plain
1.48 KB, text/plain
These boxes have been consistently orange since landing them. see, http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla2/1211383609.1211390504.5331.gz&fulltext=1 and http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla2/1211383609.1211389025.362.gz&fulltext=1 for examples.
Do you suspect misconfiguration or code error, or some gremlin inbetween?
FWIW, it appears that qm-centos5-moz2-01 was green for a while, on 2008-05-14
(In reply to comment #1) > Do you suspect misconfiguration or code error, or some gremlin inbetween? I think these could be code errors or if the failing tests are timing-related, it could be because we're running these on VMs which could be running slower than dedicated hardware. the only consistent machine seems to be the Mac which is running on a mini.
one of the errors on qm-centos5-moz2-01 is: REFTEST UNEXPECTED FAIL (LOADING): file:///builds/slave/linux/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png A loading error suggests the file might be missing - does this have something to do with the changes to the way we're doing binary checkins on hg?
It appears to be there and correct: http://hg.mozilla.org/mozilla-central/index.cgi/file/79924d3b5bba/modules/libpr0n/test/reftest/pngsuite-oddsizes/s04i3p01.png
hm, yeah. I haven't tried running these tests manually to see what's actually happening, but it seems they should work. The current failure on qm-win2k3-moz2-01 is similar: REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/layout/reftests/bugs/352980-3c.html Odd they they're both loading failures and both different.
and another different one from qm-win2k3-moz2-01 from earlier in the day: REFTEST UNEXPECTED FAIL (LOADING): file:///D:/builds/slave/win2k3/build/modules/libpr0n/test/reftest/pngsuite-oddsizes/s34i3p04.png matches the above failure on qm-centos5-moz2-01. I'm going to try clobbering these to see if that clears the problems.
scratch that. bsmedberg's bumping the reftest timeout which was recently decreased in: http://hg.mozilla.org/mozilla-central/index.cgi/rev/79fc7037d91e
Is this perhaps a dup/related to bug 431494 and bug 425987? ... need to clobber the profile before each run?
Created attachment 322280 [details] [diff] [review] profile clobber patch a quick and dirty patch to add the --clobber option to the createProfile step in the master.cfg. Ultimately, these command parameters should be moved out of these steps and run properly from the CreateProfile classes in buildbotcustom.
Created attachment 322325 [details] [diff] [review] [checked in] profile clobber patch redux args in the right order. This seems to have fixed the problem on moz2.
Comment on attachment 322325 [details] [diff] [review] [checked in] profile clobber patch redux changeset: 89:386fa565c863 tag: tip user: Rob Campbell <email@example.com> date: Mon May 26 17:04:13 2008 -0300 summary: bug 435064 - add clobber to createProfile arguments, p=me, r=bhearsum
currently we're seeing in Tunit on Linux: ../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: command timed out: 2400 seconds without output, killing pid 25293 (see bug 431745 for another example of this) and in browser chrome, chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js FAIL - Ignore warning button should be present for malware - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js FAIL - Timed out - chrome://mochikit/content/browser/browser/components/safebrowsing/content/test/browser_bug400731.js The previous run had completely different failures (in Tunit), ../../../../_tests/xpcshell-simple/test_dm/unit/test_download_manager.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111: 2779 Segmentation fault (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2 FAIL and in Mochitest: *** 13416 ERROR FAIL | Should not be able to navigate popup's popup by calling window.open. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html *** 13417 ERROR FAIL | Should not be able to navigate popup's popup by submitting form. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html *** 13418 ERROR FAIL | Should not be able to navigate popup's popup by targeted hyperlink. | got 1, expected 2 | /tests/docshell/test/navigation/test_not-opener.html
most-recent run on centos5, browser chrome: FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
from qm-win2k3-moz2-01 TUnit after a full clobber: make: Entering directory `/d/builds/slave/win2k3/build/objdir/toolkit/components/downloads/test' ... ../../../../_tests/xpcshell-simple/test_dm/unit/test_bug_401430.js: FAIL ... *** Throwing trying to get plugin.scan.4xPluginFolder *** exiting *** CHECK FAILED: false == true ... JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/head.js :: anonymous :: line 62 *** FAIL ***
from a clobbered build on qm-centos5-moz2-01, TUnit: ../../../../_tests/xpcshell-simple/test_dm/unit/test_resume.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111: 2181 Segmentation fault (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2 FAIL a whole bunch similar to: (process:2181): Gtk-CRITICAL **: _gtk_style_peek_property_value: assertion `GTK_IS_STYLE (style)' failed … ../../../../_tests/xpcshell-simple/test_dm/unit/test_sleep_wake.js: /builds/slave/linux/build/tools/test-harness/xpcshell-simple/test_all.sh: line 111: 2205 Segmentation fault (core dumped) NATIVE_TOPSRCDIR="$native_topsrcdir" TOPSRCDIR="$topsrcdir" $xpcshell -s $headfiles -f $t $tailfiles 2>$t.log 1>&2 FAIL
from qm-win2k3-moz2-01 TUnit: ../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js: FAIL ../../../../_tests/xpcshell-simple/test_places/unit/test_419731.js.log: >>>>>>> *** test pending *** exiting *** CHECK FAILED: new title 1 == new title 2 JS frame :: d:/builds/slave/win2k3/build/tools/test-harness/xpcshell-simple/execute_test.js :: <TOP_LEVEL> :: line 38 2147500036 *** FAIL *** Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.remove]" nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)" location: "JS frame :: ../../../../_tests/xpcshell-simple/test_places/unit/head_bookmarks.js :: clearDB :: line 99" data: no] <<<<<<<
from qm-win2k3-moz2-01 Mochichrome: *** 420 ERROR FAIL | Test timed out. | | chrome://mochikit/content/chrome/dom/tests/mochitest/chrome/test_domstorage.xul
from qm-centos5-03: REFTEST UNEXPECTED FAIL (LOADING): file:///builds/moz2slave/linux-3/build/layout/reftests/pixel-rounding/background-color-left-5.html
from qm-centos5-03 (browser chrome): chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js PASS - Pref window opened PASS - Specified pane was opened FAIL - handlersView is present - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js FAIL - App handler list populated - chrome://mochikit/content/browser/browser/components/preferences/tests/browser_bug410900.js
Created attachment 323455 [details] qm-centos5-moz2-01 errors This is the data mined from qm-centos5-moz2-01 sorted by build ID I had to stop at Build 80 because it appears that there was a disruption because of the ssh keys, and there was a lot of cvs errors at that point. Hopefully this is enough to look at for now, if you need more, let me know.
Created attachment 323951 [details] Error parsing script Put this together to run against a slave dir to look through the logs for errors. Improvement suggestions are welcome - right now it searched the logs for certain patterns and organizes them by log file name (which happens to start with the build id).
Created attachment 323952 [details] Error parsing script - a better version A more concise version - thanks to Dave Humphrey's tips.
nice! I have one recommendation/request - could you add 2 extra trailing lines of context to the reftest "UNEXPECTED FAIL" section? That way if it's an image comparison failure, we'll get the data URIs with the screen grabs.
Created attachment 324072 [details] Error parsing script - now with more context This one adds two trailing lines of context to UNEXPECTED FAIL for reftests so that the data URI is present.
Created attachment 324378 [details] Parse Error script for multiple directories Newest version of script - this time you can pass in several directories: ./parseErrors.sh dir1 dir2 dir3 and it will put the output for each in a directory called parsed_errors where each output is the name of the directory.output.sorted
nice improvement. a+
Created attachment 324493 [details] Script to gather up errors from buildbot master log files This is more like it - so apparently sorting was the worst thing I could do. Now the script greps each pattern match separately and the output is much easier to read over. Still going to work on turning the error output into sql insert statements so that we can have a simple, light database of errors.
Closing this bug since the intermittent failures not about specific machines, more the flakey tests in general which are being covered by other bugs under bug 438871.