Open Bug 1580999 Opened 2 years ago Updated 2 years ago

Allow to profile Gecko/GeckoView `--target=i686` under Android x86/x86_64 emulator

Categories

(Core :: Gecko Profiler, defect, P3)

x86
Android
defect

Tracking

()

People

(Reporter: nalexander, Unassigned)

Details

It "makes no sense" to profile Gecko and/or GeckoView under the Android x86 emulator in general, 'cuz that's not a configuration representative of anything. But it's extremely useful to be able to work on tooling for Android under the emulator.

STR:

  1. arrange a --target=i686 mozconfig with --enable-artifact-builds. I don't know if this is specific to artifact builds; I think it is not.
  2. ./mach android emulator --version x86-7.0 &
  3. ./mach build
  4. ./mach android install-geckoview_example
  5. On Desktop, use about:debugging to start profiling.
  6. Run adb logcat and observe:
      GeckoViewActivity  I  New Session state: {"scrolldata":{"zoom":{"resolution":1,"displaySize":{"height":1120,"width":800}}}}
            GeckoLinker  W  /system/bin/app_process32: dynamic header type #15 not handled
                         W  /system/bin/app_process32: dynamic header type #6ffffef5 not handled
                         W  /system/bin/app_process32: dynamic header type #20 not handled
                         W  /system/bin/app_process32: dynamic header type #21 not handled
                         W  /system/bin/app_process32: unhandled flags #8 not handled
                         E  /system/bin/app_process32: Missing or broken DT_HASH
                         W  /system/lib/libcutils.so: dynamic header type #6ffffef5 not handled
                         W  /system/lib/libcutils.so: unhandled flags #8 not handled
                         E  /system/lib/libcutils.so: Missing or broken DT_HASH
                 linker  E  library "/system/lib/libcutils.so" ("/system/lib/libcutils.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_librar
                            y_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]
            GeckoLinker  W  /system/lib/libutils.so: dynamic header type #6ffffef5 not handled
                         W  /system/lib/libutils.so: unhandled flags #8 not handled
                         E  /system/lib/libutils.so: Missing or broken DT_HASH
                 linker  E  library "/system/lib/libutils.so" ("/system/lib/libutils.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_
                            paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]
            GeckoLinker  W  /system/lib/liblog.so: dynamic header type #6ffffef5 not handled
                         W  /system/lib/liblog.so: unhandled flags #8 not handled
                         E  /system/lib/liblog.so: Missing or broken DT_HASH
                         E  /system/lib/libbinder.so: Unsupported minimal virtual address: 0x00006000
                 linker  E  library "/system/lib/libbinder.so" ("/system/lib/libbinder.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_librar
                            y_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]
            GeckoLinker  W  /system/lib/libnativeloader.so: dynamic header type #6ffffef5 not handled
                         W  /system/lib/libnativeloader.so: unhandled flags #8 not handled
                         E  /system/lib/libnativeloader.so: Missing or broken DT_HASH
                 linker  E  library "/system/lib/libnativeloader.so" ("/system/lib/libnativeloader.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace
                            ", ld_library_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_ex
                            ample"]
            GeckoLinker  E  /system/lib/libandroid_runtime.so: Unsupported minimal virtual address: 0x0000f000
                 linker  E  library "/system/lib/libandroid_runtime.so" ("/system/lib/libandroid_runtime.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-nam
                            espace", ld_library_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckov
                            iew_example"]
            GeckoLinker  E  /system/lib/libwilhelm.so: Unsupported minimal virtual address: 0x00003000
                 linker  E  library "/system/lib/libwilhelm.so" ("/system/lib/libwilhelm.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_libr
                            ary_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]
            GeckoLinker  E  /system/lib/libc++.so: Unsupported minimal virtual address: 0x00002000
                 linker  E  library "/system/lib/libc++.so" ("/system/lib/libc++.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_path
                            s="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]
            GeckoLinker  W  /system/lib/libc.so: dynamic header type #6ffffef5 not handled
                         W  /system/lib/libc.so: unhandled flags #8 not handled
        google-breakpad  W  ExceptionHandler::WaitForContinueSignal waiting for continue signal...
                         W  ExceptionHandler::GenerateDump cloned child
                         W  2933
                         W  ExceptionHandler::SendContinueSignalToChild sent continue signal to child
          Choreographer  I  Skipped 297 frames!  The application may be doing too much work on its main thread.

                            Process org.mozilla.geckoview_example:crash created for service org.mozilla.geckoview_example/.ExampleCrashHandler

I think something about stack unwinding is or symbol loading is chocking on libc.so. Many moons ago I tried to track this down but I can't recall exactly what I found -- something about the form of the .so itself seemed to be unexpected in our symbol reading code.

I have no Android experience (yet); Markus, any ideas, or know who to ask?

Flags: needinfo?(mstange)
OS: Unspecified → Android
Priority: -- → P3
Hardware: Unspecified → x86

So this is a crash? ExceptionHandler::GenerateDump looks like it is. Do we know for sure if the GeckoLinker warnings are related to the crash? I think we need to attach a debugger and get an actual crash stack here.

Flags: needinfo?(mstange)

(In reply to Markus Stange [:mstange] [on PTO until Sept 30] from comment #2)

So this is a crash? ExceptionHandler::GenerateDump looks like it is. Do we know for sure if the GeckoLinker warnings are related to the crash? I think we need to attach a debugger and get an actual crash stack here.

I believe they are; I think something in the Gecko Profiler's symbolication is running through our custom linker in unexpected ways, and that the warnings are our linker complaining about things. Those things probably aren't impactful, right up until they are, and we crash.

The following stack trace comes from lldb and a hand-built process. This is running an artifact build with full crashreporter symbols, which basically Just Works once you get the symbols fetched properly.

(lldb) c
Process 3639 resuming
Process 3639 stopped
* thread #12, name = 'Gecko', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0xb2b2539d
->  0xb2b2539d: movl   (%eax), %edx
    0xb2b2539f: movl   -0x68(%ebx), %eax
    0xb2b253a5: movl   %edx, (%eax)
    0xb2b253a7: leal   0x2930(%ebx), %eax
Target 0: (app_process32) stopped.
(lldb) bt
* thread #12, name = 'Gecko', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0xb2b2539d
    frame #1: 0xd5bb6337 libmozglue.so`CustomElf::Load(Mappable*, char const*, int) at CustomElf.cpp:676:52
    frame #2: 0xd5bb6305 libmozglue.so`CustomElf::Load(mappable=0xbd3f8f70, path="/system/lib/libc.so", flags=3) at CustomElf.cpp:238
    frame #3: 0xd5bb6fab libmozglue.so`ElfLoader::Load(this=0xd5d13dc4, path="/system/lib/libc.so", flags=3, parent=0x00000000) at ElfLoader.cpp:484:26
    frame #4: 0xd5bb6cd5 libmozglue.so`::__wrap_dlopen(path="/system/lib/libc.so", flags=3) at ElfLoader.cpp:64:51
    frame #5: 0xd23048ac libxul.so`AutoObjectMapperFaultyLib::Map(this=0xd6617bc0, start=0xd6617bbc, length=0xd6617bb8, fileName="{a) at AutoObjectMapper.cpp:161:17
    frame #6: 0xd230c296 libxul.so`read_procmaps(aLUL=0xbd78b580) at platform-linux-lul.cpp:47:22
    frame #7: 0xd22f1716 libxul.so`locked_profiler_start(mozilla::BaseAutoLock<PSMutex&> const&, mozilla::PowerOfTwo<unsigned int>, double, unsigned int, char const**, unsigned int, mozilla::Maybe<double> const&) at platform-linux-android.cpp:391:5
    frame #8: 0xd22f162e libxul.so`locked_profiler_start(mozilla::BaseAutoLock<PSMutex&> const&, mozilla::PowerOfTwo<unsigned int>, double, unsigned int, char const**, unsigned int, mozilla::Maybe<double> const&) [inlined] NewSamplerThread(aLock=<unavailable>, aGeneration=0, aInterval=1) at platform.cpp:2483
    frame #9: 0xd22f161d libxul.so`locked_profiler_start(mozilla::BaseAutoLock<PSMutex&> const&, mozilla::PowerOfTwo<unsigned int>, double, unsigned int, char const**, unsigned int, mozilla::Maybe<double> const&) [inlined] ActivePS::ActivePS(this=0xbd50f430, aLock=<unavailable>, aCapacity=<unavailable>, aInterval=<unavailable>, aFeatures=<unavailable>, aFilterCount=<unavailable>, aDuration=0xd6617df8) at platform.cpp:486
    frame #10: 0xd22f1566 libxul.so`locked_profiler_start(mozilla::BaseAutoLock<PSMutex&> const&, mozilla::PowerOfTwo<unsigned int>, double, unsigned int, char const**, unsigned int, mozilla::Maybe<double> const&) [inlined] ActivePS::Create(aLock=<unavailable>, aCapacity=<unavailable>, aInterval=<unavailable>, aFeatures=<unavailable>, aFilterCount=<unavailable>, aDuration=0xd6617df8) at platform.cpp:579
    frame #11: 0xd22f1557 libxul.so`locked_profiler_start(aLock=<unavailable>, aCapacity=(mValue = 16777216), aInterval=<unavailable>, aFeatures=<unavailable>, aFilters=0xc02e64d8, aFilterCount=2, aDuration=0xd6617d48) at platform.cpp:3403
    frame #12: 0xd22f32c4 libxul.so`profiler_start(aCapacity=(mValue = 16777216), aInterval=1, aFeatures=290, aFilters=0xc02e64d8, aFilterCount=2, aDuration=0xd6617d48) at platform.cpp:3513:5
    frame #13: 0xd22f75f1 libxul.so`nsProfiler::StartProfiler(this=0xbd50f350, aEntries=10000000, aInterval=1, aFeatures=0xd6617e98, aFilters=0xd6617eb0, aDuration=0) at nsProfiler.cpp:136:3
    frame #14: 0xcfedece0 libxul.so`NS_InvokeByIndex + 48
    frame #15: 0xd05526da libxul.so`XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [inlined] CallMethodHelper::Invoke(this=<unavailable>) at XPCWrappedNative.cpp:1650:10
    frame #16: 0xd05526bb libxul.so`XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) [inlined] CallMethodHelper::Call(this=<unavailable>) at XPCWrappedNative.cpp:1187
    frame #17: 0xd0551ba4 libxul.so`XPCWrappedNative::CallMethod(ccx=<unavailable>, mode=<unavailable>) at XPCWrappedNative.cpp:1149
    frame #18: 0xd0553454 libxul.so`XPC_WN_CallMethod(cx=<unavailable>, argc=5, vp=0xbc786068) at XPCWrappedNativeJSOps.cpp:943:10
    frame #19: 0xd256258d libxul.so`js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) [inlined] CallJSNative(cx=<unavailable>, native=<unavailable>, reason=<unavailable>, args=<unavailable>)(JSContext*, unsigned int, JS::Value*), js::CallReason, JS::CallArgs const&) at Interpreter.cpp:458:13
    frame #20: 0xd256245d libxul.so`js::InternalCallOrConstruct(cx=<unavailable>, args=0xd6618830, construct=NO_CONSTRUCT, reason=Call) at Interpreter.cpp:551
    frame #21: 0xd2562a38 libxul.so`InternalCall(cx=<unavailable>, args=<unavailable>, reason=Call) at Interpreter.cpp:620:10
    frame #22: 0xd255add6 libxul.so`Interpret(JSContext*, js::RunState&) [inlined] js::CallFromStack(cx=<unavailable>, args=0xbc786078) at Interpreter.cpp:624:10
    frame #23: 0xd255adc4 libxul.so`Interpret(cx=<unavailable>, state=<unavailable>) at Interpreter.cpp:3113
    frame #24: 0xd255131c libxul.so`js::RunScript(cx=<unavailable>, state=<unavailable>) at Interpreter.cpp:424:10
    frame #25: 0xd25621b8 libxul.so`js::InternalCallOrConstruct(cx=<unavailable>, args=0xd6618a80, construct=NO_CONSTRUCT, reason=Call) at Interpreter.cpp:592:13
    frame #26: 0xd2562a38 libxul.so`InternalCall(cx=<unavailable>, args=<unavailable>, reason=Call) at Interpreter.cpp:620:10
    frame #27: 0xd2aa6d3f libxul.so`js::jit::InvokeFunction(JSContext*, JS::Handle<JSObject*>, bool, bool, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) [inlined] js::Call(cx=<unavailable>, fval=<unavailable>, thisv=<unavailable>, args=0xd6618ac8, rval=<unavailable>, reason=Call) at Interpreter.cpp:637:8
    frame #28: 0xd2aa6d0c libxul.so`js::jit::InvokeFunction(cx=<unavailable>, obj=JS::HandleObject @ 0xd6618b14, constructing=<unavailable>, ignoresReturnValue=<unavailable>, argc=1, argv=0xd6618bb0, rval=JS::MutableHandleValue @ 0xd6618b28) at VMFunctions.cpp:260
    frame #29: 0xd2aa6feb libxul.so`js::jit::InvokeFromInterpreterStub(cx=<unavailable>, frame=0xd6618b9c) at VMFunctions.cpp:289:8
    frame #30: 0xc2d30b5a

The revision is wrong here but the code is the same: https://searchfox.org/mozilla-central/rev/1d0e84d2ec2843924bc6e5ed6658f93439e12351/mozglue/linker/CustomElf.cpp#676. (Searchfox hasn't indexed the m-c commit, https://hg.mozilla.org/mozilla-central/rev/9596d7f4a7457bccc78cadf9c39bcc9c4b5b97f8, yet.) Looks like we're trying to run initializers, and either we're not handling a NULL initializer or the initializer itself is crashing. Probably 'cuz we shouldn't be running it at all -- what on earth does initializing libc.so twice mean?

glandium: do you have any ideas on what's happening here?

Flags: needinfo?(mh+mozilla)

I'd worry more about the errors that look like the following than the crash:

linker  E  library "/system/lib/libcutils.so" ("/system/lib/libcutils.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]

Does android prevent dlopen now?

Flags: needinfo?(mh+mozilla)

(In reply to Mike Hommey [:glandium] from comment #4)

I'd worry more about the errors that look like the following than the crash:

linker  E  library "/system/lib/libcutils.so" ("/system/lib/libcutils.so") needed or dlopened by "/data/app/org.mozilla.geckoview_example-1/lib/x86/libmozglue.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_paths="", default_library_paths="/data/app/org.mozilla.geckoview_example-1/lib/x86:/data/app/org.mozilla.geckoview_example-1/base.apk!/lib/x86", permitted_paths="/data:/mnt/expand:/data/data/org.mozilla.geckoview_example"]

Does android prevent dlopen now?

It does for many libraries. https://developer.android.com/about/versions/nougat/android-7.0-changes

I dug into this a little bit, but my knowledge and ability quickly bottomed out. Here's what I conclude:

  • we have custom Android-aware symbol mapping code
  • that code assumes it can dlopen any libraries currently loaded by the active process
  • on Android, that assumption is false
  • but that code handles failure to dlopen (mostly) gracefully: we produce error messages and mark certain code sections as not having symbols, but don't otherwise crash
  • but we still fail because we try to use our custom linker for some system libraries, and either that fails outright or it fails because the system library init methods aren't idempotent

I have three questions:

  1. On Android, should we be trying to dlopen all libraries? This is going to fail for many libraries, but hopefully it will fail in a benign fashion.
  2. Can we avoid using our custom linker for this dlopen process entirely? Should we be avoiding our custom linker at least for libraries in /system?
  3. Do we have latent bugs in our library init process?

I can't really dig much further. I could try to add logging around the sequence of libraries that we try to dlopen, but it's not clear that would help.

Well, would you look at that. Following a somewhat oblique hint from gbrown over in https://bugzilla.mozilla.org/show_bug.cgi?id=1584976#c11, I tried with a --target=x86_64 artifact build, and I was able to profile! So it looks like something is wrong with unwinding specifically for --target=i686.

Summary: Allow to profile Gecko/GeckoView under Android x86 emulator → Allow to profile Gecko/GeckoView `--target=i686` under Android x86/x86_64 emulator
You need to log in before you can comment on or make changes to this bug.