Last Comment Bug 652294 - intermittent failure in url-classifier/tests/unit/test_hashcompleter.js, failure after "Now on completion set index 4"
: intermittent failure in url-classifier/tests/unit/test_hashcompleter.js, fail...
Status: RESOLVED FIXED
: intermittent-failure
Product: Toolkit
Classification: Components
Component: Safe Browsing (show other bugs)
: unspecified
: x86 Linux
: -- normal (vote)
: Firefox 7
Assigned To: Mehdi Mulani [:mmm] (I don't check this)
:
Mentors:
Depends on:
Blocks: 438871
  Show dependency treegraph
 
Reported: 2011-04-22 18:38 PDT by Marco Bonardo [::mak]
Modified: 2014-05-27 12:25 PDT (History)
3 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
A couple dumps to provide more information in this case. (checked in) (1.48 KB, patch)
2011-04-22 19:10 PDT, Mehdi Mulani [:mmm] (I don't check this)
no flags Details | Diff | Review
Patch to fix typo. (combined check-in) (1.48 KB, patch)
2011-04-22 19:34 PDT, Mehdi Mulani [:mmm] (I don't check this)
no flags Details | Diff | Review
Add random number generator, switch to that and check for duplicates. (4.61 KB, patch)
2011-06-04 20:12 PDT, Mehdi Mulani [:mmm] (I don't check this)
dcamp: review+
Details | Diff | Review
Take two: check for duplicate prefixes, not hashes. (4.69 KB, patch)
2011-06-13 14:31 PDT, Mehdi Mulani [:mmm] (I don't check this)
dcamp: review+
Details | Diff | Review

Description Marco Bonardo [::mak] 2011-04-22 18:38:31 PDT
http://tinderbox.mozilla.org/showlog.cgi?log=Places/1303519746.1303520765.14912.gz
Rev3 Fedora 12 places opt test xpcshell on 2011/04/22 17:49:06

Now on completion set index 4
TEST-PASS | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | [completionFinished : 296] false == false
TEST-PASS | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | [completionFinished : 296] false == false
TEST-PASS | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | [completionFinished : 296] false == false
TEST-PASS | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | [completionFinished : 296] false == false
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
JS frame :: /home/cltbld/talos-slave/test/build/xpcshell/head.js :: do_throw :: line 444
JS frame :: /home/cltbld/talos-slave/test/build/xpcshell/head.js :: _do_check_eq :: line 538
JS frame :: /home/cltbld/talos-slave/test/build/xpcshell/head.js :: do_check_eq :: line 559
JS frame :: /home/cltbld/talos-slave/test/build/xpcshell/head.js :: do_check_true :: line 573
JS frame :: /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js :: completion :: line 267
Comment 1 Marco Bonardo [::mak] 2011-04-22 18:39:20 PDT
Also, philor pointed out this happened on 20 Apr in central

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1303304729.1303305530.18043.gz
Comment 2 Mehdi Mulani [:mmm] (I don't check this) 2011-04-22 19:10:40 PDT
Created attachment 527903 [details] [diff] [review]
A couple dumps to provide more information in this case. (checked in)

It seems we are running into this intermittent issue in the randomly generated portion of the test. So far, the only case I have thought of is that two generated hashes are the same and one expects a completion while the other doesn't.

It does seem odd that these are popping up recently.

Pushed this patch to m-c:
http://hg.mozilla.org/mozilla-central/rev/05825e4dc77a
Comment 3 Mehdi Mulani [:mmm] (I don't check this) 2011-04-22 19:34:17 PDT
Created attachment 527906 [details] [diff] [review]
Patch to fix typo. (combined check-in)

Previous attachment had a typo. If the intermittent issue isn't hit, it shouldn't cause a problem.

This patch is a combination of the past and current.

http://hg.mozilla.org/mozilla-central/rev/4f40b01a2f47
Comment 4 Treeherder Robot 2011-05-06 20:06:47 PDT
philor
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1304735235.1304736508.30193.gz
Rev3 Fedora 12 mozilla-central opt test xpcshell on 2011/05/06 19:27:15

s: talos-r3-fed-018
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | ticmmi == mdvsre - See following stack:
Comment 5 Mehdi Mulani [:mmm] (I don't check this) 2011-05-09 16:21:36 PDT
Don't have internet right now so I can't actively track/work on this but my plan of action is to change the random number generator and output a seed on start to help make this more reproducible.

So far, I still have the same theory: we create data which creates contradictions in what we expect from the completion (e.g. a data set with two same prefixes that have different tables and different |expectCompletion| values)
Comment 6 Treeherder Robot 2011-05-11 01:04:19 PDT
m_kato%ga2.so-net.ne.jp
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1305099298.1305100163.17366.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test xpcshell on 2011/05/11 00:34:58

s: talos-r3-leopard-038
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 7 Mehdi Mulani [:mmm] (I don't check this) 2011-05-11 06:54:54 PDT
(In reply to comment #6)
> m_kato%ga2.so-net.ne.jp
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1305099298.1305100163.
> 17366.gz
> Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test xpcshell on 2011/05/11
> 00:34:58
> 
> s: talos-r3-leopard-038
> TEST-UNEXPECTED-FAIL |
> /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-
> classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell
> return code: 0), see following log:
> TEST-UNEXPECTED-FAIL |
> /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-
> classifier/tests/unit/test_hashcompleter.js | false == true - See following
> stack:

Yaus. As expected, it looks like we created duplicate entries with the same prefix. Relevant debug info:

Did not expect a completion for this result. Provided values:
hash: "<\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
table: e8hmjb
chunkId: 29884
Actual values:
hash: "<\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
table: undefined
chunkId: undefined

my plan of action is to check for duplicate prefixes and handle them explicitly. (i.e. create a new prefix)
Comment 8 Treeherder Robot 2011-05-28 11:28:15 PDT
philor
http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1306590030.1306591736.24144.gz
Rev3 WINNT 5.1 tracemonkey opt test xpcshell on 2011/05/28 06:40:30

s: talos-r3-xp-004
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\url-classifier\tests\unit\test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 9 Treeherder Robot 2011-05-31 14:04:35 PDT
bz%mit.edu
http://tinderbox.mozilla.org/showlog.cgi?log=Cedar/1306859935.1306860976.1440.gz
Rev3 Fedora 12x64 cedar opt test xpcshell on 2011/05/31 09:38:55

s: talos-r3-fed64-026
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | j8ubwd == rtvl1n - See following stack:
Comment 10 Mehdi Mulani [:mmm] (I don't check this) 2011-06-04 20:12:34 PDT
Created attachment 537414 [details] [diff] [review]
Add random number generator, switch to that and check for duplicates.

I added in a tiny random number generator so that the random numbers are created by a seed that is trackable. (This should make debugging easier.)

I also added a hook to ensure that all prefixes are unique which handles this intermittent and lastly, I removed the debug lines I added in.

Dave, I flagged you as you reviewed the original test though I can move it to someone else.
Comment 11 Treeherder Robot 2011-06-07 16:10:30 PDT
philor
http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1307482771.1307486152.22024.gz
Rev3 WINNT 6.1 tracemonkey debug test xpcshell on 2011/06/07 14:39:31

s: talos-r3-w7-024
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\url-classifier\tests\unit\test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | myr82k == 3rsugv - See following stack:
Comment 12 Mehdi Mulani [:mmm] (I don't check this) 2011-06-08 15:57:39 PDT
http://hg.mozilla.org/integration/mozilla-inbound/rev/96fac50b57ed
Comment 13 Treeherder Robot 2011-06-08 20:54:02 PDT
ehsan%mozilla.com
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1307585471.1307587458.29616.gz
Rev3 WINNT 5.1 mozilla-inbound opt test xpcshell on 2011/06/08 19:11:11

s: talos-r3-xp-017
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\url-classifier\tests\unit\test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 14 Treeherder Robot 2011-06-08 21:42:33 PDT
rnewman%mozilla.com
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1307590838.1307593956.22849.gz
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2011/06/08 20:40:38

s: talos-r3-w7-053
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\url-classifier\tests\unit\test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 15 Boris Zbarsky [:bz] 2011-06-08 22:00:22 PDT
I backed out the attempt to fix this, since it caused Windows opt unit tests to be orange on every single run.
Comment 16 Treeherder Robot 2011-06-09 02:02:31 PDT
philipp%weitershausen.de
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1307607293.1307608372.9491.gz
Rev3 Fedora 12 mozilla-inbound opt test xpcshell on 2011/06/09 01:14:53

s: talos-r3-fed-033
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 17 Mehdi Mulani [:mmm] (I don't check this) 2011-06-13 14:31:49 PDT
Created attachment 539017 [details] [diff] [review]
Take two: check for duplicate prefixes, not hashes.

In the first patch I ensured that we generated unique hashes and thought this implied a unique |completion| call per hash. However, if two hashes share the same prefix (first 32 bits) then it is possible to get multiple completions per hash. I changed the logic to check for duplicate prefixes.

This ensures that each hash only goes through one code path while limiting the number of possible hashes generated by the random set to 256^4. I reran the failed tests (by using the same seed) and they now pass. And lastly, it looks like this failure was hit on WinXP opt 2/3 times and by chance (that is, it does not appear to be a problem with WinXP opt test runs).
Comment 18 Treeherder Robot 2011-06-14 07:22:19 PDT
philor
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1308026760.1308029929.14406.gz
Rev3 WINNT 6.1 mozilla-inbound debug test xpcshell on 2011/06/13 21:46:00

s: talos-r3-w7-013
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\toolkit\components\url-classifier\tests\unit\test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | c:/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 19 Treeherder Robot 2011-06-14 08:24:05 PDT
mbrubeck%mozilla.com
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1308062839.1308063602.4189.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-inbound opt test xpcshell on 2011/06/14 07:47:19

s: talos-r3-leopard-041
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | false == true - See following stack:
Comment 20 Treeherder Robot 2011-06-20 11:51:07 PDT
mbrubeck%mozilla.com
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Aurora/1308594730.1308595376.13624.gz
Rev3 MacOSX Leopard 10.5.8 mozilla-aurora opt test xpcshell on 2011/06/20 11:32:10

s: talos-r3-leopard-003
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/toolkit/components/url-classifier/tests/unit/test_hashcompleter.js | ut8sc7 == q49846 - See following stack:
Comment 21 Mehdi Mulani [:mmm] (I don't check this) 2011-06-20 12:52:25 PDT
Second try, pushed to mozilla-inbound.
http://hg.mozilla.org/integration/mozilla-inbound/rev/303ddaeb736f
Comment 22 :Ehsan Akhgari (busy, don't ask for review please) 2011-06-21 06:42:31 PDT
http://hg.mozilla.org/mozilla-central/rev/303ddaeb736f

Note You need to log in before you can comment on or make changes to this bug.