Closed Bug 904227 Opened 11 years ago Closed 11 years ago

Intermittent TEST-UNEXPECTED-FAIL | ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds)

Categories

(Core :: Graphics: Layers, defect)

Product:
Core
Component:
Graphics: Layers
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla28
Tracking Flags:
Tracking Status
firefox26 --- wontfix
firefox27 --- fixed
firefox28 --- fixed
firefox-esr24 --- unaffected
b2g-v1.2 --- fixed

People

(Reporter: RyanVM, Assigned: BenWa)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(4 files, 2 obsolete files)

ignore gmock warnings from MockContentController. r=kats
10.48 KB, patch
BenWa
: review+
Details | Diff | Splinter Review
Disable gfxWordCacheTest.cpp
801 bytes, patch
bjacob
: review+
Details | Diff | Splinter Review
Print ? if we don't have a test object
1.48 KB, patch
bjacob
: review+
Details | Diff | Splinter Review
patch
5.58 KB, patch
kats
: review+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=26428795&tree=Mozilla-Central

Linux x86-64 mozilla-central leak test build on 2013-08-12 02:29:46 PDT for push f057fca09627
slave: bld-linux64-ec2-416

TEST-START | AsyncPanZoomController.Constructor
TEST-PASS | AsyncPanZoomController.Constructor | test completed (time: 0ms)
TEST-START | AsyncPanZoomController.SimpleTransform
TEST-PASS | AsyncPanZoomController.SimpleTransform | test completed (time: 1ms)
TEST-START | AsyncPanZoomController.ComplexTransform
TEST-PASS | AsyncPanZoomController.ComplexTransform | test completed (time: 4ms)
TEST-START | AsyncPanZoomController.Pan
TEST-PASS | AsyncPanZoomController.Pan | test completed (time: 48ms)
TEST-START | AsyncPanZoomController.Fling
TEST-PASS | AsyncPanZoomController.Fling | test completed (time: 4ms)
TEST-START | AsyncPanZoomController.OverScrollPanning
TEST-PASS | AsyncPanZoomController.OverScrollPanning | test completed (time: 3ms)
TEST-START | APZCTreeManager.GetAPZCAtPoint
TEST-PASS | APZCTreeManager.GetAPZCAtPoint | test completed (time: 6ms)
TEST-START | MozillaGTestSanity.Runs
TEST-PASS | MozillaGTestSanity.Runs | test completed (time: 0ms)
TEST-START | MozillaGMockSanity.Runs
TEST-PASS | MozillaGMockSanity.Runs | test completed (time: 0ms)
TEST-PASS | GTest unit test: passed

GMOCK WARNING:
Uninteresting mock function call - returning directly.
    Function call: SendAsyncScrollDOMEvent(1, @0x7fff73d8e300 16-byte object <00-00 70-41 00-00 20-41 00-00 00-40 00-00 00-40>, @0x7fff73d8e2f8 8-byte object <00-00 48-42 00-00 48-42>)
Stack trace:

GMOCK WARNING:
Uninteresting mock function call - returning directly.
    Function call: SendAsyncScrollDOMEvent(2, @0x7fff73d8e300 16-byte object <00-00 70-41 00-00 20-41 00-00 00-40 00-00 00-40>, @0x7fff73d8e2f8 8-byte object <00-00 48-42 00-00 48-42>)
Stack trace:
PROCESS-CRASH | firefox | application crashed [@ libxul.so + 0x179334e]
Crash dump filename: /builds/slave/m-cen-l64-d-000000000000000000/build/obj-firefox/testing/gtest/5550a038-0790-d9e1-4a04ad9a-010453d2.dmp
Operating system: Linux
                  0.0.0 Linux 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64
CPU: amd64
     family 6 model 45 stepping 7
     4 CPUs

Crash reason:  SIGSEGV
Crash address: 0x8

Thread 0 (crashed)
 0  libxul.so + 0x179334e
    rbx = 0x0000000000000001   r12 = 0x00000000010c0500
    r13 = 0x00007fff73d8dc90   r14 = 0x00007fff73d8dd20
    r15 = 0x0000000000000001   rip = 0x00007f2b6f51234e
    rsp = 0x00007fff73d8dc10   rbp = 0x00007fff73d8dc10
    Found by: given as instruction pointer in context
 1  libxul.so + 0x1797b18
    rip = 0x00007f2b6f516b19   rsp = 0x00007fff73d8dc20
    rbp = 0x00007fff73d8dc40
    Found by: stack scanning
 2  libxul.so + 0x17a03f7
    rip = 0x00007f2b6f51f3f8   rsp = 0x00007fff73d8dc50
    rbp = 0x00007fff73d8dd00
    Found by: stack scanning
 3  libmozalloc.so!moz_xmalloc [mozalloc.cpp : 54 + 0x7]
    rip = 0x00007f2b72237069   rsp = 0x00007fff73d8dc60
    rbp = 0x00007fff73d8dd00
    Found by: stack scanning
 4  libxul.so + 0x17a04e4
    rbx = 0x00007f2b6f52d060   rip = 0x00007f2b6f51f4e5
    rsp = 0x00007fff73d8dd10   rbp = 0x00007fff73d8dd50
    Found by: call frame info
 5  libxul.so + 0x179563d
    rip = 0x00007f2b6f51463e   rsp = 0x00007fff73d8dd60
    rbp = 0x00007fff73d8dda0
    Found by: stack scanning
Since the stack doesn't actually contain any gtest code, I'm not convinced that the crash is related to the gtest warning above it. I believe those warnings should be ignored by the test framework anyway (I've seen similar warnings before when working on tests, and they don't usually cause the test to fail). BenWa, can you confirm?
Flags: needinfo?(bgirard)
Crash Signature: [@ libxul.so@0x179334e]
Depends on: 904574
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #2)
> Since the stack doesn't actually contain any gtest code, I'm not convinced
> that the crash is related to the gtest warning above it. I believe those
> warnings should be ignored by the test framework anyway (I've seen similar
> warnings before when working on tests, and they don't usually cause the test
> to fail). BenWa, can you confirm?

I think gmock might cause a failure at some point even if it doesn't fail that test. I'll take a look at this one.
Assignee: nobody → bgirard
Flags: needinfo?(bgirard)
Severity: normal → critical
Summary: Intermittent gtests TEST-UNEXPECTED-FAIL | firefox | test crashed | application crashed [@ libxul.so + 0x179334e] after GMOCK WARNING for SendAsyncScrollDOMEvent → Intermittent gtest | application crashed [@ libxul.so + 0x179334e] after GMOCK WARNING for SendAsyncScrollDOMEvent
Summary: Intermittent gtest | application crashed [@ libxul.so + 0x179334e] after GMOCK WARNING for SendAsyncScrollDOMEvent → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] after GMOCK WARNING for SendAsyncScrollDOMEvent

        Attachment #8334270 -
        Flags: review?(bugmail.mozilla)

    
    

    
  
Comment on attachment 8334270 [details] [diff] [review]
ignore gmock warnings from MockContentController

Review of attachment 8334270 [details] [diff] [review]:
-----------------------------------------------------------------

Oh cool I didn't know you could do that.

        Attachment #8334270 -
        Flags: review?(bugmail.mozilla) → review+

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
benwa, kats, do this needs to be checkedin right? (since it already has review+)
Flags: needinfo?(bugmail.mozilla)
Flags: needinfo?(bgirard)

    
    

    
  
Yup
Flags: needinfo?(bugmail.mozilla)
Flags: needinfo?(bgirard)
Keywords: checkin-needed

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/0d2c9cafe2e8
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Target Milestone: --- → mozilla28

    
    

    
      
  

    
    

    
  
(In reply to TBPL Robot from comment #275)

Still happening.
Status: RESOLVED → REOPENED
Flags: needinfo?(bgirard)
Resolution: FIXED → ---
Target Milestone: mozilla28 → ---

    
    

    
  
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #276)
> (In reply to TBPL Robot from comment #275)
> 
> Still happening.

I don't see the gmock warning there.
Flags: needinfo?(bgirard)

    
    

    
  
So the gmock warning isn't causing the crash. However, we're still crashing, which means we need someone who understands these tests to look into this further.

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] after GMOCK WARNING for SendAsyncScrollDOMEvent → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan builds

    
    

    
      
  
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan builds → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan debug builds

    
    

    
  

      
      
      
      

        Attached patch
          
          
        Disable gfxWordCacheTest.cpp
        — 
          — Splinter Review
      

    


  
Lets see if this helps

        Attachment #8342101 -
        Flags: review?(bjacob)

    
    

    
  
Maybe 98% debug, but not 100%, see comment 272 for a fairly recent opt build.
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on leak test and asan debug builds → Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds

    
    

    
  
FWIW I built an ASan debug build on my Linux machine using the instructions at [1] but was unable to reproduce the gtest crash.

[1] https://developer.mozilla.org/en-US/docs/Building_Firefox_with_Address_Sanitizer

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

        Attachment #8342101 -
        Flags: review?(bjacob) → review+

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/71e8967053dc
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28

    
    

    
      
  

    
    

    
  
Thanks for landing.

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
Some recent ones were misstarred as bug 889073, but this is still happening.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: mozilla28 → ---

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
Per request from BenWa, I took the log by RyanVM to find out what slave, and updated the libxul.so temporarily to my people account:

[cltbld@bld-linux64-ec2-094.build.releng.use1.mozilla.com ~]$ ls /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so
/builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so

[cltbld@bld-linux64-ec2-094.build.releng.use1.mozilla.com ~]$ ls -la /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so
lrwxrwxrwx 1 cltbld mock_mozilla 43 Dec  6 11:58 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/dist/bin/gtest/libxul.so -> ../../../toolkit/library/libgtest/libxul.so

[cltbld@bld-linux64-ec2-094.build.releng.use1.mozilla.com ~]$ ls -la /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/toolkit/library/libgtest/../../../toolkit/library/libgtest/libxul.so
-rwxrwxr-x 1 cltbld mock_mozilla 663928728 Dec  6 11:58 /builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/toolkit/library/libgtest/../../../toolkit/library/libgtest/libxul.so

then scrapped it and uploaded it to people:

[jwood@cruncher.srv.releng.scl3 ~]$ ssh jwood@people.mozilla.com "md5sum public_html/tmp/*.so"
6736c5d7569ce4b8f313ed317f02dd54  public_html/tmp/libxul.so
[jwood@cruncher.srv.releng.scl3 ~]$ md5sum libxul.so
6736c5d7569ce4b8f313ed317f02dd54  libxul.so

So find it at: http://people.mozilla.org/~jwood/tmp/libxul.so

    
    

    
      
  

    
    

    
      
  

    
    

    
  
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/mozilla/GTestRunner.cpp:47
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gtest/src/gtest.cc:2978
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gtest/src/gtest.cc:3778
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gtest/src/gtest.cc:356
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gmock/src/gmock-internal-utils.cc:85
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/gmock/src/gmock-spec-builders.cc:426
/builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/gfx/tests/gtest/../../../dist/include/gmock/gmock-spec-builders.h:1490
/builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/gfx/tests/gtest/../../../dist/include/gmock/gmock-generated-function-mockers.h:140
/builds/slave/fx-team-l64-asan-d-00000000000/build/gfx/layers/ipc/AsyncPanZoomController.cpp:1702
/builds/slave/fx-team-l64-asan-d-00000000000/build/gfx/layers/ipc/AsyncPanZoomController.cpp:1290
/builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/chromium/src/base/message_loop.cc:340
/builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/chromium/src/base/message_loop.cc:348
/builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/chromium/src/base/message_loop.cc:475
/builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/glue/MessagePump.cpp:197
/builds/slave/fx-team-l64-asan-d-00000000000/build/ipc/glue/MessagePump.cpp:57
/builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/threads/nsTimerImpl.cpp:554
/builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/threads/nsTimerImpl.cpp:637
/builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/threads/nsThread.cpp:612
/builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/glue/nsThreadUtils.cpp:210
/builds/slave/fx-team-l64-asan-d-00000000000/build/xpcom/build/nsXPComInit.cpp:684
/builds/slave/fx-team-l64-asan-d-00000000000/build/obj-firefox/testing/gtest/../../dist/include/testing/TestHarness.h:143
/builds/slave/fx-team-l64-asan-d-00000000000/build/testing/gtest/mozilla/GTestRunner.cpp:111
/builds/slave/fx-team-l64-asan-d-00000000000/build/toolkit/xre/nsAppRunner.cpp:3213
/builds/slave/fx-team-l64-asan-d-00000000000/build/toolkit/xre/nsAppRunner.cpp:4026

    
    

    
  
AsyncPanZoomController.cpp:1702 is a call to GeckoContentController::SendAsyncScrollDOMEvent. But it's crashing inside gtest code.. maybe we're doing something wrong with setting the mock call expectations?

    
    

    
  

      
      
      
      

        Attached patch
          
          
        Print ? if we don't have a test object (obsolete)
        — 
          — Splinter Review
      

    


  

        Attachment #8343960 -
        Flags: review?(bjacob)

    
    

    
  


  

        Attachment #8343960 -
        Attachment is obsolete: true

        Attachment #8343960 -
        Flags: review?(bjacob)

        Attachment #8343975 -
        Flags: review?(bjacob)

    
    

    
  
Comment on attachment 8343975 [details] [diff] [review]
Print ? if we don't have a test object

Review of attachment 8343975 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/gtest/mozilla/GTestRunner.cpp
@@ +43,5 @@
>    }
>    virtual void OnTestPartResult(const TestPartResult& aTestPartResult) MOZ_OVERRIDE {
>      printf("TEST-%s | %s.%s | %s @ %s:%i\n",
>             !aTestPartResult.failed() ? "PASS" : "UNEXPECTED-FAIL",
> +           mTestInfo ? mTestInfo->test_case_name() : "?", mTestInfo ? mTestInfo->name() : "?",

Please add a newline after the comma.

        Attachment #8343975 -
        Flags: review?(bjacob) → review+

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
https://tbpl.mozilla.org/php/getParsedLog.php?id=31593303&tree=Mozilla-Inbound
Summary: Intermittent gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds → Intermittent ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds)

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/9d2035e2155b
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla28

    
    

    
      
  

    
    

    
  
Pretty sure someone said that didn't actually fix it ;)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → NEW

    
    

    
  
Stupid whole-line searches.
Summary: Intermittent ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds) → Intermittent TEST-UNEXPECTED-FAIL | ?.? | Mock function called more times than expected - returning directly. (formerly gtest | application crashed [@ libxul.so + 0xXXXXXXX] on Linux builds)

    
    

    
      
  

    
    

    
      
  

    
    

    
  

      
      
      
      

        Attached patch
          
          
        patch
        — 
          — Splinter Review
      

    


  
Alright we're making progress here.

This is fairly trivial. Almost feel like landing without a review.

        Attachment #8344123 -
        Flags: review?(bugmail.mozilla)

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
Comment on attachment 8344123 [details] [diff] [review]
patch

Review of attachment 8344123 [details] [diff] [review]:
-----------------------------------------------------------------

I'd change them all to be AtLeast(1) since we don't really care about the exact number that will happen.

        Attachment #8344123 -
        Flags: review?(bugmail.mozilla) → review+

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/b6c7d4aa8b72
Status: NEW → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED

    
    

    
      
  

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: