879424 - Profiler Hangs with MOZ_PROFILER_STARTUP

Status: RESOLVED WORKSFORME

(Core :: Gecko Profiler, defect)

Description

[Benoit Girard (:BenWa)]

Attachment: gdb session

wlach sees that the profiler hangs fennec if started with MOZ_PROFILER_STARTUP. I took a look and it seems that the loader is stuck waiting on a lock:

AutoLock (mutex=0x5d7351b0, this=<synthetic pointer>) at mozglue/linker/Mappable.cpp:433

Comment 1

[Benoit Girard (:BenWa)]

Any idea what this lock is about glandium? I don't see any other threads waiting for that lock or in the profiler.

Comment 2

[William Lachance (:wlach)]

It looks like this regressed sometime in the last month. I'll try to get a tighter regression range when I get the chance. Only seen this hang on a galaxy nexus running Android 4.2.3; not sure about other platforms.

Comment 3

[Benoit Girard (:BenWa)]

(In reply to William Lachance (:wlach) from comment #2)
> Only seen this hang on a
> galaxy nexus running Android 4.2.3; not sure about other platforms.

We reproduce on my phone with your build which is 4.1.2.

Comment 4

[William Lachance (:wlach)]

Did some bisecting. Looks like this problem first started occuring on May 21st. So something in here (didn't see any obvious candidates):

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b94e996b05fd&tochange=4ac6c72b06c8

Comment 5

[Mike Hommey [:glandium] OoO until Dec 30]

bug 848764 is the likely culprit. You can work around by setting MOZ_LINKER_ONDEMAND=0 in the environment.

(In reply to Benoit Girard (:BenWa) from comment #0)
> Created attachment 758110 [details]
> gdb session
> 
> wlach sees that the profiler hangs fennec if started with
> MOZ_PROFILER_STARTUP. I took a look and it seems that the loader is stuck
> waiting on a lock:
> 
> AutoLock (mutex=0x5d7351b0, this=<synthetic pointer>) at
> mozglue/linker/Mappable.cpp:433

It would be interesting to know what other threads are doing.

Comment 6

[William Lachance (:wlach)]

Did this, did not fix the problem. Reverting the patch didn't help either. :(

(In reply to Mike Hommey (high latency until June 25) [:glandium] from comment #5)
> bug 848764 is the likely culprit. You can work around by setting
> MOZ_LINKER_ONDEMAND=0 in the environment.
> 
> (In reply to Benoit Girard (:BenWa) from comment #0)
> > Created attachment 758110 [details]
> > gdb session
> > 
> > wlach sees that the profiler hangs fennec if started with
> > MOZ_PROFILER_STARTUP. I took a look and it seems that the loader is stuck
> > waiting on a lock:
> > 
> > AutoLock (mutex=0x5d7351b0, this=<synthetic pointer>) at
> > mozglue/linker/Mappable.cpp:433
> 
> It would be interesting to know what other threads are doing.

Comment 7

[Benoit Girard (:BenWa)]

(In reply to Mike Hommey (high latency until June 25) [:glandium] from comment #5)
> It would be interesting to know what other threads are doing.

Did you see the GDB sessions? There's a dump of all the threads, it doesn't look like any other thread would be waiting on that look.

(In reply to William Lachance (:wlach) from comment #2)
> It looks like this regressed sometime in the last month.

The month time frame corresponds to many profiler changes. Regression window would certainly be lovely.

Comment 8

[Mike Hommey [:glandium] OoO until Dec 30]

(In reply to William Lachance (:wlach) from comment #6)
> Did this, did not fix the problem. Reverting the patch didn't help either. :(

There is no way you can hit that lock in Mappable.cpp if you revert bug 848764 or set MOZ_LINKER_ONDEMAND=0.

Comment 9

[William Lachance (:wlach)]

(In reply to Mike Hommey (high latency until June 25) [:glandium] from comment #8)
> (In reply to William Lachance (:wlach) from comment #6)
> > Did this, did not fix the problem. Reverting the patch didn't help either. :(
> 
> There is no way you can hit that lock in Mappable.cpp if you revert bug
> 848764 or set MOZ_LINKER_ONDEMAND=0.

I just verified that the inbound build before your patch (be1d72a1f228) does not exhibit this problem whereas the one after (ef5b7b1039ac) does, with or without the MOZ_LINKER_ONDEMAND. Maybe I didn't rebuild properly after reverting your patch in my build (I just did a ./mach build after, maybe I needed to clobber?). Anyway, I think we've found the source of the problem.

Comment 10

[Mike Hommey [:glandium] OoO until Dec 30]

Can you confirm that a clobber m-c or m-i build with MOZ_LINKER_ONDEMAND=0 set with am start works properly? There is no way the code path with that lock is used under that circumstance.

Comment 11

[William Lachance (:wlach)]

(In reply to Mike Hommey (high latency until June 25) [:glandium] from comment #10)
> Can you confirm that a clobber m-c or m-i build with MOZ_LINKER_ONDEMAND=0
> set with am start works properly? There is no way the code path with that
> lock is used under that circumstance.

MOZ_LINKER_ONDEMAND definitely does not work for me with the builds after your change. Here's the exact command I'm using to start:

am start -W -n org.mozilla.fennec/.App -a android.intent.action.VIEW --es env1 MOZ_LINKER_ONDEMAND=0 --es env0 MOZ_PROFILER_STARTUP=1 -d http://google.ca

I will test to see if a clobber build with the change reverted does work.

Comment 12

[William Lachance (:wlach)]

Sometime in the last few weeks, this started working again. Not sure of the cause.