Closed Bug 904227 Opened 11 years ago Closed 11 years ago

Intermittent TEST-UNEXPECTED-FAIL | ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds)

Categories

(Core :: Graphics: Layers, defect)

Product:
Core
Component:
Graphics: Layers
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla28
Tracking Flags:
Tracking Status
firefox26 --- wontfix
firefox27 --- fixed
firefox28 --- fixed
firefox-esr24 --- unaffected
b2g-v1.2 --- fixed

People

(Reporter: RyanVM, Assigned: BenWa)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(4 files, 2 obsolete files)

ignore gmock warnings from MockContentController. r=kats
10.48 KB, patch
BenWa
: review+
Details | Diff | Splinter Review
Disable gfxWordCacheTest.cpp
801 bytes, patch
bjacob
: review+
Details | Diff | Splinter Review
Print ? if we don't have a test object
1.48 KB, patch
bjacob
: review+
Details | Diff | Splinter Review
patch
5.58 KB, patch
kats
: review+
Details | Diff | Splinter Review
https://tbpl.mozilla.org/php/getParsedLog.php?id=26428795&tree=Mozilla-Central Linux x86-64 mozilla-central leak test build on 2013-08-12 02:29:46 PDT for push f057fca09627 slave: bld-linux64-ec2-416 TEST-START | AsyncPanZoomController.Constructor TEST-PASS | AsyncPanZoomController.Constructor | test completed (time: 0ms) TEST-START | AsyncPanZoomController.SimpleTransform TEST-PASS | AsyncPanZoomController.SimpleTransform | test completed (time: 1ms) TEST-START | AsyncPanZoomController.ComplexTransform TEST-PASS | AsyncPanZoomController.ComplexTransform | test completed (time: 4ms) TEST-START | AsyncPanZoomController.Pan TEST-PASS | AsyncPanZoomController.Pan | test completed (time: 48ms) TEST-START | AsyncPanZoomController.Fling TEST-PASS | AsyncPanZoomController.Fling | test completed (time: 4ms) TEST-START | AsyncPanZoomController.OverScrollPanning TEST-PASS | AsyncPanZoomController.OverScrollPanning | test completed (time: 3ms) TEST-START | APZCTreeManager.GetAPZCAtPoint TEST-PASS | APZCTreeManager.GetAPZCAtPoint | test completed (time: 6ms) TEST-START | MozillaGTestSanity.Runs TEST-PASS | MozillaGTestSanity.Runs | test completed (time: 0ms) TEST-START | MozillaGMockSanity.Runs TEST-PASS | MozillaGMockSanity.Runs | test completed (time: 0ms) TEST-PASS | GTest unit test: passed GMOCK WARNING: Uninteresting mock function call - returning directly. Function call: SendAsyncScrollDOMEvent(1, @0x7fff73d8e300 16-byte object <00-00 70-41 00-00 20-41 00-00 00-40 00-00 00-40>, @0x7fff73d8e2f8 8-byte object <00-00 48-42 00-00 48-42>) Stack trace: GMOCK WARNING: Uninteresting mock function call - returning directly. Function call: SendAsyncScrollDOMEvent(2, @0x7fff73d8e300 16-byte object <00-00 70-41 00-00 20-41 00-00 00-40 00-00 00-40>, @0x7fff73d8e2f8 8-byte object <00-00 48-42 00-00 48-42>) Stack trace: PROCESS-CRASH | firefox | application crashed [@ libxul.so + 0x179334e] Crash dump filename: /builds/slave/m-cen-l64-d-000000000000000000/build/obj-firefox/testing/gtest/5550a038-0790-d9e1-4a04ad9a-010453d2.dmp Operating system: Linux 0.0.0 Linux 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 CPU: amd64 family 6 model 45 stepping 7 4 CPUs Crash reason: SIGSEGV Crash address: 0x8 Thread 0 (crashed) 0 libxul.so + 0x179334e rbx = 0x0000000000000001 r12 = 0x00000000010c0500 r13 = 0x00007fff73d8dc90 r14 = 0x00007fff73d8dd20 r15 = 0x0000000000000001 rip = 0x00007f2b6f51234e rsp = 0x00007fff73d8dc10 rbp = 0x00007fff73d8dc10 Found by: given as instruction pointer in context 1 libxul.so + 0x1797b18 rip = 0x00007f2b6f516b19 rsp = 0x00007fff73d8dc20 rbp = 0x00007fff73d8dc40 Found by: stack scanning 2 libxul.so + 0x17a03f7 rip = 0x00007f2b6f51f3f8 rsp = 0x00007fff73d8dc50 rbp = 0x00007fff73d8dd00 Found by: stack scanning 3 libmozalloc.so!moz_xmalloc [mozalloc.cpp : 54 + 0x7] rip = 0x00007f2b72237069 rsp = 0x00007fff73d8dc60 rbp = 0x00007fff73d8dd00 Found by: stack scanning 4 libxul.so + 0x17a04e4 rbx = 0x00007f2b6f52d060 rip = 0x00007f2b6f51f4e5 rsp = 0x00007fff73d8dd10 rbp = 0x00007fff73d8dd50 Found by: call frame info 5 libxul.so + 0x179563d rip = 0x00007f2b6f51463e rsp = 0x00007fff73d8dd60 rbp = 0x00007fff73d8dda0 Found by: stack scanning
Since the stack doesn't actually contain any gtest code, I'm not convinced that the crash is related to the gtest warning above it. I believe those warnings should be ignored by the test framework anyway (I've seen similar warnings before when working on tests, and they don't usually cause the test to fail). BenWa, can you confirm?
Flags: needinfo?(bgirard)
Crash Signature: [@ libxul.so@0x179334e]
Depends on: 904574
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #2) > Since the stack doesn't actually contain any gtest code, I'm not convinced > that the crash is related to the gtest warning above it. I believe those > warnings should be ignored by the test framework anyway (I've seen similar > warnings before when working on tests, and they don't usually cause the test > to fail). BenWa, can you confirm? I think gmock might cause a failure at some point even if it doesn't fail that test. I'll take a look at this one.
Assignee: nobody → bgirard
Flags: needinfo?(bgirard)
Severity: normal → critical
Summary: Intermittent gtests TEST-UNEXPECTED-FAIL | firefox | test crashed | application crashed [@ libxul.so + 0x179334e] after GMOCK WARNING for SendAsyncScrollDOMEvent → Intermittent gtest | application crashed [@ libxul.so + 0x179334e] after GMOCK WARNING for SendAsyncScrollDOMEvent
Summary: Intermittent gtest | application crashed [@ libxul.so + 0x179334e] after GMOCK WARNING for SendAsyncScrollDOMEvent → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] after GMOCK WARNING for SendAsyncScrollDOMEvent
Attachment #8334270 - Flags: review?(bugmail.mozilla)
Comment on attachment 8334270 [details] [diff] [review] ignore gmock warnings from MockContentController Review of attachment 8334270 [details] [diff] [review]: ----------------------------------------------------------------- Oh cool I didn't know you could do that.
Attachment #8334270 - Flags: review?(bugmail.mozilla) → review+
benwa, kats, do this needs to be checkedin right? (since it already has review+)
Flags: needinfo?(bugmail.mozilla)
Flags: needinfo?(bgirard)
Yup
Flags: needinfo?(bugmail.mozilla)
Flags: needinfo?(bgirard)
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/0d2c9cafe2e8
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Target Milestone: --- → mozilla28
(In reply to TBPL Robot from comment #275) Still happening.
Status: RESOLVED → REOPENED
Flags: needinfo?(bgirard)
Resolution: FIXED → ---
Target Milestone: mozilla28 → ---
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #276) > (In reply to TBPL Robot from comment #275) > > Still happening. I don't see the gmock warning there.
Flags: needinfo?(bgirard)
So the gmock warning isn't causing the crash. However, we're still crashing, which means we need someone who understands these tests to look into this further.
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] after GMOCK WARNING for SendAsyncScrollDOMEvent → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan builds
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan builds → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan debug builds
Attached patch Disable gfxWordCacheTest.cpp — — Splinter Review
Lets see if this helps
Attachment #8342101 - Flags: review?(bjacob)
Maybe 98% debug, but not 100%, see comment 272 for a fairly recent opt build.
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan debug builds → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds
FWIW I built an ASan debug build on my Linux machine using the instructions at [1] but was unable to reproduce the gtest crash. [1] https://developer.mozilla.org/en-US/docs/Building_Firefox_with_Address_Sanitizer
Attachment #8342101 - Flags: review?(bjacob) → review+
https://hg.mozilla.org/mozilla-central/rev/71e8967053dc
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28
Thanks for landing.
Some recent ones were misstarred as bug 889073, but this is still happening.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: mozilla28 → ---
Per request from BenWa, I took the log by RyanVM to find out what slave, and updated the libxul.so temporarily to my people account: [cltbld@bld-linux64-ec2-094.build.releng.use1.mozilla.com ~]$ ls /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so [cltbld@bld-linux64-ec2-094.build.releng.use1.mozilla.com ~]$ ls -la /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so lrwxrwxrwx 1 cltbld mock_mozilla 43 Dec 6 11:58 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so -> ../../../toolkit/library/libgtest/libxul.so [cltbld@bld-linux64-ec2-094.build.releng.use1.mozilla.com ~]$ ls -la /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/toolkit/library/libgtest/../../../toolkit/library/libgtest/libxul.so -rwxrwxr-x 1 cltbld mock_mozilla 663928728 Dec 6 11:58 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/toolkit/library/libgtest/../../../toolkit/library/libgtest/libxul.so then scrapped it and uploaded it to people: [jwood@cruncher.srv.releng.scl3 ~]$ ssh jwood@people.mozilla.com "md5sum public_html/tmp/*.so" 6736c5d7569ce4b8f313ed317f02dd54 public_html/tmp/libxul.so [jwood@cruncher.srv.releng.scl3 ~]$ md5sum libxul.so 6736c5d7569ce4b8f313ed317f02dd54 libxul.so So find it at: http://people.mozilla.org/~jwood/tmp/libxul.so
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/mozilla/GTestRunner.cpp:47 /builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gtest/src/gtest.cc:2978 /builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gtest/src/gtest.cc:3778 /builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gtest/src/gtest.cc:356 /builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gmock/src/gmock-internal-utils.cc:85 /builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gmock/src/gmock-spec-builders.cc:426 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/gfx/tests/gtest/../../../dist/include/gmock/gmock-spec-builders.h:1490 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/gfx/tests/gtest/../../../dist/include/gmock/gmock-generated-function-mockers.h:140 /builds/slave/fx-team-l64-asan-d-00000000000/build/gfx/layers/ipc/AsyncPanZoomController.cpp:1702 /builds/slave/fx-team-l64-asan-d-00000000000/build/gfx/layers/ipc/AsyncPanZoomController.cpp:1290 /builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/chromium/src/base/message_loop.cc:340 /builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/chromium/src/base/message_loop.cc:348 /builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/chromium/src/base/message_loop.cc:475 /builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/glue/MessagePump.cpp:197 /builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/glue/MessagePump.cpp:57 /builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/threads/nsTimerImpl.cpp:554 /builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/threads/nsTimerImpl.cpp:637 /builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/threads/nsThread.cpp:612 /builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/glue/nsThreadUtils.cpp:210 /builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/build/nsXPComInit.cpp:684 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/testing/gtest/../../dist/include/testing/TestHarness.h:143 /builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/mozilla/GTestRunner.cpp:111 /builds/slave/fx-team-l64-asan-d-00000000000/build/toolkit/xre/nsAppRunner.cpp:3213 /builds/slave/fx-team-l64-asan-d-00000000000/build/toolkit/xre/nsAppRunner.cpp:4026
AsyncPanZoomController.cpp:1702 is a call to GeckoContentController::SendAsyncScrollDOMEvent. But it's crashing inside gtest code.. maybe we're doing something wrong with setting the mock call expectations?
Attached patch Print ? if we don't have a test object (obsolete) — — Splinter Review
Attachment #8343960 - Flags: review?(bjacob)
Attachment #8343960 - Attachment is obsolete: true
Attachment #8343960 - Flags: review?(bjacob)
Attachment #8343975 - Flags: review?(bjacob)
Comment on attachment 8343975 [details] [diff] [review] Print ? if we don't have a test object Review of attachment 8343975 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/gtest/mozilla/GTestRunner.cpp @@ +43,5 @@ > } > virtual void OnTestPartResult(const TestPartResult& aTestPartResult) MOZ_OVERRIDE { > printf("TEST-%s | %s.%s | %s @ %s:%i\n", > !aTestPartResult.failed() ? "PASS" : "UNEXPECTED-FAIL", > + mTestInfo ? mTestInfo->test_case_name() : "?", mTestInfo ? mTestInfo->name() : "?", Please add a newline after the comma.
Attachment #8343975 - Flags: review?(bjacob) → review+
https://tbpl.mozilla.org/php/getParsedLog.php?id=31593303&tree=Mozilla-Inbound
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds → Intermittent ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds)
https://hg.mozilla.org/mozilla-central/rev/9d2035e2155b
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28
Pretty sure someone said that didn't actually fix it ;)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → NEW
Stupid whole-line searches.
Summary: Intermittent ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds) → Intermittent TEST-UNEXPECTED-FAIL | ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds)
Attached patch patch — — Splinter Review
Alright we're making progress here. This is fairly trivial. Almost feel like landing without a review.
Attachment #8344123 - Flags: review?(bugmail.mozilla)
Comment on attachment 8344123 [details] [diff] [review] patch Review of attachment 8344123 [details] [diff] [review]: ----------------------------------------------------------------- I'd change them all to be AtLeast(1) since we don't really care about the exact number that will happen.
Attachment #8344123 - Flags: review?(bugmail.mozilla) → review+
https://hg.mozilla.org/mozilla-central/rev/b6c7d4aa8b72
Status: NEW → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: