1495574 - browser_dbg_rr_breakpoints-01.js test failure due to EXC_BAD_ACCESS crash

Status: RESOLVED FIXED

(Core Graveyard :: Web Replay, defect)

Description

[Haik Aftandilian [:haik]]

When running the Web Replay test devtools/client/debugger/new/test/mochitest/browser_dbg_rr_breakpoints-01.js locally, the test always times out. This is on OS X Mojave with mozilla-central based on the following changeset. MacBook Pro (Retina, 15-inch, Mid 2015). 

On a non-debug build, the timeout seems to happen because the replay process crashes at startup due to Thread::Current() returning NULL.

The crash was hit here

  XUL was compiled with optimization - stepping may behave oddly; variables may not be available.
  Process 22336 stopped
  * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
      frame #0 XUL`::RecordReplayInterface_InternalRecordReplayValue(size_t)() at Thread.h:157
     154 	    mPassThroughEvents = aPassThrough;
     155 	  }
     156 	  bool PassThroughEvents() const {
  -> 157 	    return mPassThroughEvents;
     158 	  }
     159 	
     160 	  // Access the counter for whether events are disallowed in this thread.
  Target 0: (plugin-container) stopped.
  (lldb) bt
  * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    * frame #0 XUL`::RecordReplayInterface_InternalRecordReplayValue(size_t)() at Thread.h:157
      frame #1 XUL`::RecordReplayInterface_InternalRecordReplayValue(size_t)() at ProcessRecordReplay.cpp:163
      frame #2 XUL`XRE_InitChildProcess() at RecordReplay.h:421
      frame #3 XUL`XRE_InitChildProcess() at nsEmbedFunctions.cpp:627
      frame #4 plugin-container`main() at plugin-container.cpp:50
      frame #5 plugin-container`main() at MozillaRuntimeMain.cpp:25
      frame #6 libdyld.dylib`start() 
  (lldb) frame variable
  (const mozilla::recordreplay::Thread *) this = <variable not available>

From the debugger, I went upwards in the stack to RecordReplayInterface_InternalRecordReplayValue() and saw that the thread pointer returned from Thread::Current() at line 162 below was NULL.

  159 MOZ_EXPORT size_t
  160 RecordReplayInterface_InternalRecordReplayValue(size_t aValue)
  161 {
  162   Thread* thread = Thread::Current();
  163   if (thread->PassThroughEvents()) {
  164     return aValue;
  165   }
  166   EnsureNotDivergedFromRecording();
  ...
  ... }

Comment 1

[Brian Hackett [Laid off!]]

Hmm, it looks like Web Replay is broken on Mojave.  Thread::Current() fetches a value from TLS, which should have been initialized by the recordreplay::Initialize call in XRE_InitChildProcess.

What I think is happening is that record/replay initialization failed, and that we aren't able to deal with that afterwards and crash instead of reporting the failure properly.  I'll write a patch to fix that, but it won't deal with the original problem.

That original problem is likely an issue with initializing the function redirections: the redirection code has to scan and disassemble the code for hundreds of system library functions, and if it encounters an instruction it can't cope with then it fails.  With a new OS version the code for many of these library functions will have changed, so it's possible there's a new kind of instruction there that needs some special handling.  Once the failure reporting is working the problem might be obvious, but in any case I'll be able to download and try Mojave myself by late next week (I'm on a low bandwidth connection right now).

Comment 2

[Brian Hackett [Laid off!]]

Attachment: patch

The problem with running Web Replay on Mojave is (as predicted in comment 1) a new kind of instruction that needs special handling when redirecting functions (atomic byte exchange register with memory at an ip-relative offset).  After bug 1497027 we now print out / report what the actual problem is, and I just got a Mojave system and was able to add handling for these instructions, per the attached patch.  With this fixed, things seem to be working fine on Mojave.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a64b17b18dae
Support copying ip-relative xchg instructions, r=froydnj.

Comment 4

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/a64b17b18dae