Closed Bug 573290 Opened 13 years ago Closed 11 years ago

Intermittent failure in test_crashreporter_crash.js | test failed (with xpcshell return code: 0) and false == true

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 2 obsolete files)

Corruption diagnosis, rev. 1
1.91 KB, patch
Details | Diff | Splinter Review
Use libc/syscall wrappers on Linux, but don't re-enable tests on x86
5.22 KB, patch
benjamin
: review+
Details | Diff | Splinter Review 
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1276991242.1276992271.20801.gz
Rev3 Fedora 12 mozilla-central opt test xpcshell on 2010/06/19 16:47:22
s: talos-r3-fed-048

TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js | test failed (with xpcshell return code: 0), see following log:
  >>>>>>>
  TEST-INFO | (xpcshell/head.js) | test 1 pending
arg[0] = -g
arg[1] = /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/firefox
arg[2] = -f
arg[3] = /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/crasher_subprocess_head.js
arg[4] = -f
arg[5] = /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/crasher_subprocess_tail.js
TEST-PASS | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/head_crashreporter.js | [do_crash : 65] 256 != 0
TEST-PASS | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/head_crashreporter.js | [do_crash : 84] true == true
TEST-PASS | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js | [anonymous : 10] true == true
TEST-PASS | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js | [anonymous : 11] true == true
TEST-PASS | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js | [anonymous : 12] true == true
TEST-UNEXPECTED-FAIL | /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js | false == true - See following stack:
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/head.js :: do_throw :: line 257
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/head.js :: do_check_eq :: line 287
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/head.js :: do_check_true :: line 299
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js :: anonymous :: line 13
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/head_crashreporter.js :: do_crash :: line 88
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/tests/crashreporter_test/unit/test_crashreporter_crash.js :: run_test :: line 9
JS frame :: /home/cltbld/talos-slave/mozilla-central-fedora-opt-u-xpcshell/build/xpcshell/head.js :: _execute_test :: line 151
JS frame :: -e :: <TOP_LEVEL> :: line 1
TEST-INFO | (xpcshell/head.js) | exiting test

  <<<<<<<
I'm getting this a lot with bug 568619 branch, and I can reproduce it very consistently on x86-linux if I don't debug it ;-)

AnnotateCrashReport(StartupTime, 1277824175)
  crashReporterAPIData=StartupTime=1277824175

ExceptionHandler::GenerateDump
API string: StartupTime=1277824175


MinidumpCallback, crashReporterAPIData<8397332>:
open

So sometime during generation of the minidump we trash crashReporterAPIData. Attaching a debugger, or adding a busy-sleep to the process initialization seems to fix it.
Holy Moses. We're lazily trying to resolve a symbol (open in this case) and the dynamic loader is trashing memory. I'm not sure *why* it would trash memory at this point, but I think the simple answer here is to force the symbols we need to be early-loaded.
Attached patch Resolve open/write early, rev. 1 (obsolete) — — Splinter Review 
Assignee: nobody → benjamin
Status: NEW → ASSIGNED

        Attachment #454878 -
        Flags: review?(ted.mielczarek)

        Attachment #454878 -
        Flags: review?(ted.mielczarek) → review+

    
    

    
  
Comment on attachment 454878 [details] [diff] [review]
Resolve open/write early, rev. 1

Yuck, but ok.

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
bsmedberg: did you land your other patch and not this, thus making it perma-orange?

    
    

    
  
No, I landed http://hg.mozilla.org/mozilla-central/rev/99ac7d16e904 with the other one and it didn't help. I have now disabled the crashreporter tests on Linux-x86 until I can experiment with using the libc-replacement stuff.

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
The plugin crash-submission tests were still failing after the band-aid; pushed http://hg.mozilla.org/mozilla-central/rev/567178446a35 to disable those too.
Assignee: benjamin → ted.mielczarek

    
    

    
  


  
This backs out the previous patch and the test disabling, and uses the Breakpad wrappers for Linux syscalls, as well as the trivial Breakpad libc helpers.

The tests pass on my x86-64 build, and I'm doing a quick OS X build to ensure that I didn't break that, and I'll also push-to-try to check that this fixes the test bustage.

        Attachment #454878 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 455674 [details] [diff] [review]
Use libc/syscall wrappers on Linux

Works on OS X, pushing to try to see if it fixes the test bustage.

        Attachment #455674 -
        Flags: review?(benjamin)

    
    

    
  
Comment on attachment 455674 [details] [diff] [review]
Use libc/syscall wrappers on Linux

Tests still fail on try.

        Attachment #455674 -
        Flags: review?(benjamin)

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  


  
Since this bug is now getting spammed with the x86-64 test failures, I think we should land this patch. This fixes the test failure on my x86-64 machine, but last I tried it it didn't fix x86.

        Attachment #455674 -
        Attachment is obsolete: true

        Attachment #462125 -
        Flags: review?(benjamin)

    
  

        Attachment #462125 -
        Flags: review?(benjamin) → review+

    
  
Depends on: 585394

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
      
  

    
    

    
  
May have found related bug: 691632
Assignee: ted.mielczarek → nobody

    
  
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME

    
    

    
  
Excitingly, we haven't been running these tests at all recently: bug 733501.
Whiteboard: [orange]

          You need to log in
          before you can comment on or make changes to this bug.