619672 - Perma-Orange on Thunderbird Linux | TEST-UNEXPECTED-FAIL | test_redirectsMode.js followed by head_queries.js | 5 == 3

Status: RESOLVED FIXED

(Toolkit :: Places, defect)

Description

[Mark Banner (:standard8)]

Since the latest places landing, test_redirectsMode.js has been orange on Thunderbird Linux (both 32 and 64 bit builds):

TESTING: includeHidden(false), redirectsMode(0), maxResults(5), sortingMode(0).
*** PLACES TESTS: Comparing Array to Results
TEST-UNEXPECTED-FAIL | /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/places/tests/queries/head_queries.js | 5 == 3 - See following stack:
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/head.js :: do_throw :: line 439
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/head.js :: do_check_eq :: line 491
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/places/tests/queries/head_queries.js :: compareArrayToResult :: line 345
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/places/tests/queries/test_redirectsMode.js :: check_results_callback :: line 125
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/places/tests/queries/test_redirectsMode.js :: cartProd :: line 179
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/tests/toolkit/components/places/tests/queries/test_redirectsMode.js :: run_test :: line 307
JS frame :: /buildbot/comm-central-trunk-linux-opt-unittest-xpcshell/build/xpcshell/head.js :: _execute_test :: line 322
JS frame :: -e :: <TOP_LEVEL> :: line 1

I'm currently investigating, but ideas would be appreciated.

From what I can tell, SeaMonkey doesn't have this issue. So if its something to do with the build-configuration set ups, then it would be most likely an IPC versus non-IPC issue.

Comment 1

[Mark Banner (:standard8)]

Local non-libxul build couldn't reproduce. Therefore trying a clobber on the boxes and a libxul build locally.

Comment 2

[Marco Bonardo [:mak]]

hm, we had almost permaorange on windows, thus the test is disabled on Windows, but we were unable to reproduce the problem. If you find a way to make it reproduceable, I'd be pretty happy.

Comment 3

[Marco Bonardo [:mak]]

and if the failure sticks, we can temporarily disable it completely. but would be really awesome for us to have a working way to reproduce on a local build.

Comment 4

[Mark Banner (:standard8)]

This passes locally with a debug build of libxul Thunderbird. I guess I'm going to have to try opt next, though I might see if I can grab a builder later and take a closer look.

Comment 5

[Marco Bonardo [:mak]]

I can just tell you that I ran this test in a loop 6000 times on Win, and was never able to reproduce a failure :(

Comment 6

[Mark Banner (:standard8)]

I've just reproduced this by downloading the packaged test that's been built on the builders and running locally in my linux vm.

On my VM it seems random with about 50% failure rate. Additionally the tree has just had its first green run (out of about 10).

So very frequent orange, probably timing related at a guess.

FTR the steps are:

# wget http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-linux/1292518869/thunderbird-3.3a2pre.en-US.linux-i686.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tinderbox-builds/comm-central-linux/1292518869/thunderbird-3.3a2pre.en-US.linux-i686.tests.zip
# tar xfj thunderbird-3.3a2pre.en-US.linux-i686.tar.bz2 
# unzip -o thunderbird-3.3a2pre.en-US.linux-i686.tests.zip bin* certs* xpcshell*
# mkdir thunderbird/plugins
# cp bin/xpcshell thunderbird
# cp -R bin/components/* thunderbird/components/
# cp -R bin/plugins/* thunderbird/plugins/

# python -u xpcshell/runxpcshelltests.py thunderbird/xpcshell --test-path=test_redirectsMode.js xpcshell/tests/toolkit/components/places/tests/queries/

Comment 7

[Mark Banner (:standard8)]

From what I've gleaned so far. This is the section of the test we're having trouble with, but when it actually passes:

TESTING: includeHidden(false), redirectsMode(0), maxResults(5), sortingMode(0).
*** PLACES TESTS: Comparing Array to Results
TEST-PASS | /home/cltbld/tests/xpcshell/tests/toolkit/components/places/tests/queries/head_queries.js | [compareArrayToResult : 387] 5 == 5
*** PLACES TESTS: testing testData[http://1.example.com/] vs result[http://1.example.com/]
*** PLACES TESTS: testing testData[http://2.example.com/] vs result[http://2.example.com/]
*** PLACES TESTS: testing testData[http://3.example.com/] vs result[http://3.example.com/]
*** PLACES TESTS: testing testData[http://1.redirect.temp.example.com/] vs result[http://1.redirect.temp.example.com/]
*** PLACES TESTS: testing testData[http://2.redirect.temp.example.com/] vs result[http://2.redirect.temp.example.com/]
*** PLACES TESTS: Comparing Array to Results passes

When the tests fails,  those two redirects aren't there.

Comment 8

[Marco Bonardo [:mak]]

yes, so I tried with David Dahl to reproduce this on a real linux box, and we were unable to. So sounds like this could be some sort of VM issue, maybe with timers.

Comment 9

[Mark Banner (:standard8)]

Attachment: More detailed failing log

This is a log of the failing mode. The interesting bits to note are:

- This is the first of the 'includeHidden(false)' tests.
- When it fails, the http://*redirect*/ lines show that hidden is set to 1, as per the attached log.

Adding dump_table("moz_places") appears to improve the failure rate, but it still isn't perfect (probably 1 in 5 rather than 1 in 2).

Comment 10

[Marco Bonardo [:mak]]

Attachment: patch v1.0

please try this one.

Comment 11

[Mark Banner (:standard8)]

I've done 200 runs with some with busy CPU and some without and not had a failure - it looks good to me.

Comment 12

[Shawn Wilsher :sdwilsh]

Comment on attachment 498196 [details] [diff] [review]
patch v1.0

r=sdwilsh

Comment 13

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/mozilla-central/rev/4ca2a5e830ce