905008 - [A/V] Crash on libxul.so!android::OmxDecoder::ReadAudio

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect, P1)

Description

[leo.bugzilla.gecko]

Tester's report about crash.
To reproduce the problem, I am receiving all of files from tester's sdcard.
(I cannot reproduce at all on my side with my files.)
(I am trying to get STR)

According to dump file, there're two types of call stack.
(There're seven crash reports on tester's device)

First one,
0  libxul.so!android::OmxDecoder::ReadAudio [OmxDecoder.cpp : 687 + 0x6]
     r4 = 0x42d6b980    r5 = 0x42910cd8    r6 = 0x42910c80    r7 = 0x00000000
     r8 = 0x0015c700    r9 = 0x42910d70   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910c78    lr = 0x4082eddd    pc = 0x4082ede4
    Found by: given as instruction pointer in context
 1  libxul.so!nsMediaOmxReader::DecodeAudioData [nsMediaOmxReader.cpp : 277 + 0x9]
     r4 = 0x42d46200    r5 = 0x00000000    r6 = 0x42d46208    r7 = 0x00000000
     r8 = 0x00000000    r9 = 0x42910d70   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910cd0    pc = 0x4082e0e1
    Found by: call frame info
 2  libxul.so!nsBuiltinDecoderReader::DecodeToTarget [nsBuiltinDecoderReader.cpp : 431 + 0x7]
     r4 = 0x42d46200    r5 = 0x00000000    r6 = 0x42d46208    r7 = 0x00000000
     r8 = 0x00000000    r9 = 0x42910d70   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910d28    pc = 0x40868671
    Found by: call frame info
 3  libxul.so!nsMediaOmxReader::Seek [nsMediaOmxReader.cpp : 315 + 0x9]
     r4 = 0x42d46200    r5 = 0x00000000    r6 = 0x42d46244    r7 = 0x42d46208
     r8 = 0x4458ab9c    r9 = 0x4458ab9c   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910db8    pc = 0x4082e095
    Found by: call frame info
 4  libxul.so!nsBuiltinDecoderStateMachine::DecodeSeek [nsBuiltinDecoderStateMachine.cpp : 1993 + 0x1d]
     r4 = 0x43b03d00    r5 = 0x43b03d08    r6 = 0x00000000    r7 = 0x0015c700
     r8 = 0x4458ab9c    r9 = 0x4458ab9c   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910dd8    pc = 0x40866d2b
    Found by: call frame info
 5  libxul.so!nsBuiltinDecoderStateMachine::DecodeThreadRun [nsBuiltinDecoderStateMachine.cpp : 537 + 0x5]
     r4 = 0x43b03d00    r5 = 0x4458ab9c    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x42910eb7    r9 = 0x41bb178c   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910e58    pc = 0x408677fb
    Found by: call frame info
 6  libxul.so!nsRunnableMethodImpl<nsrefcnt (mozilla::dom::workers::DOMBindingBase::*)(), false>::Run [nsThreadUtils.h : 366 + 0x5]
     r4 = 0x41bb1760    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x42910eb7    r9 = 0x41bb178c   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910e68    pc = 0x403db2dd
    Found by: call frame info
 7  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp : 620 + 0x5]
     r4 = 0x41bb1760    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x42910eb7    r9 = 0x41bb178c   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910e70    pc = 0x40bda0cf
    Found by: call frame info
 8  libxul.so!NS_ProcessNextEvent_P [nsThreadUtils.cpp : 237 + 0xb]
     r4 = 0x00000001    r5 = 0x42910ec4    r6 = 0x00000000    r7 = 0x41bb178c
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910eb0    pc = 0x40bba49f
    Found by: call frame info
 9  libxul.so!nsThread::ThreadFunc [nsThread.cpp : 258 + 0x7]
     r4 = 0x41bb1760    r5 = 0x42910ec4    r6 = 0x00000000    r7 = 0x41bb178c
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910ec0    pc = 0x40bda519
    Found by: call frame info
10  libnspr4.so!_pt_root [ptthread.c : 202 + 0x5]
     r4 = 0x41b079b0    r5 = 0x41596c0c    r6 = 0x00000000    r7 = 0x00000078
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910ee0    pc = 0x4158e9c9
    Found by: call frame info
11  libc.so!__thread_entry [pthread.c : 217 + 0x6]
     r4 = 0x42910f00    r5 = 0x4158e969    r6 = 0x41b079b0    r7 = 0x00000078
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910ef0    pc = 0x400f30ec
    Found by: call frame info
12  libc.so!pthread_create [pthread.c : 357 + 0xe]
     r4 = 0x42910f00    r5 = 0x00569060    r6 = 0x44effcfc    r7 = 0x00000078
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910f00    pc = 0x400f2c40
    Found by: call frame info

Second one,
0  libxul.so!android::OmxDecoder::ReadAudio [OmxDecoder.cpp : 687 + 0x6]
     r4 = 0x42d6b980    r5 = 0x42910cd8    r6 = 0x42910c80    r7 = 0x00000000
     r8 = 0x0015c700    r9 = 0x42910d70   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910c78    lr = 0x4082eddd    pc = 0x4082ede4
    Found by: given as instruction pointer in context
 1  libxul.so!nsMediaOmxReader::DecodeAudioData [nsMediaOmxReader.cpp : 277 + 0x9]
     r4 = 0x42d46200    r5 = 0x00000000    r6 = 0x42d46208    r7 = 0x00000000
     r8 = 0x00000000    r9 = 0x42910d70   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910cd0    pc = 0x4082e0e1
    Found by: call frame info
 2  libxul.so!nsBuiltinDecoderReader::DecodeToTarget [nsBuiltinDecoderReader.cpp : 431 + 0x7]
     r4 = 0x42d46200    r5 = 0x00000000    r6 = 0x42d46208    r7 = 0x00000000
     r8 = 0x00000000    r9 = 0x42910d70   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910d28    pc = 0x40868671
    Found by: call frame info
 3  libxul.so!nsMediaOmxReader::Seek [nsMediaOmxReader.cpp : 315 + 0x9]
     r4 = 0x42d46200    r5 = 0x00000000    r6 = 0x42d46244    r7 = 0x42d46208
     r8 = 0x4458ab9c    r9 = 0x4458ab9c   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910db8    pc = 0x4082e095
    Found by: call frame info
 4  libxul.so!nsBuiltinDecoderStateMachine::DecodeSeek [nsBuiltinDecoderStateMachine.cpp : 1993 + 0x1d]
     r4 = 0x43b03d00    r5 = 0x43b03d08    r6 = 0x00000000    r7 = 0x0015c700
     r8 = 0x4458ab9c    r9 = 0x4458ab9c   r10 = 0x00000001    fp = 0x42910e28
     sp = 0x42910dd8    pc = 0x40866d2b
    Found by: call frame info
 5  libxul.so!nsBuiltinDecoderStateMachine::DecodeThreadRun [nsBuiltinDecoderStateMachine.cpp : 537 + 0x5]
     r4 = 0x43b03d00    r5 = 0x4458ab9c    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x42910eb7    r9 = 0x41bb178c   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910e58    pc = 0x408677fb
    Found by: call frame info
 6  libxul.so!nsRunnableMethodImpl<nsrefcnt (mozilla::dom::workers::DOMBindingBase::*)(), false>::Run [nsThreadUtils.h : 366 + 0x5]
     r4 = 0x41bb1760    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x42910eb7    r9 = 0x41bb178c   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910e68    pc = 0x403db2dd
    Found by: call frame info
 7  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp : 620 + 0x5]
     r4 = 0x41bb1760    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x42910eb7    r9 = 0x41bb178c   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910e70    pc = 0x40bda0cf
    Found by: call frame info
 8  libxul.so!NS_ProcessNextEvent_P [nsThreadUtils.cpp : 237 + 0xb]
     r4 = 0x00000001    r5 = 0x42910ec4    r6 = 0x00000000    r7 = 0x41bb178c
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910eb0    pc = 0x40bba49f
    Found by: call frame info
 9  libxul.so!nsThread::ThreadFunc [nsThread.cpp : 258 + 0x7]
     r4 = 0x41bb1760    r5 = 0x42910ec4    r6 = 0x00000000    r7 = 0x41bb178c
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910ec0    pc = 0x40bda519
    Found by: call frame info
10  libnspr4.so!_pt_root [ptthread.c : 202 + 0x5]
     r4 = 0x41b079b0    r5 = 0x41596c0c    r6 = 0x00000000    r7 = 0x00000078
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910ee0    pc = 0x4158e9c9
    Found by: call frame info
11  libc.so!__thread_entry [pthread.c : 217 + 0x6]
     r4 = 0x42910f00    r5 = 0x4158e969    r6 = 0x41b079b0    r7 = 0x00000078
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910ef0    pc = 0x400f30ec
    Found by: call frame info
12  libc.so!pthread_create [pthread.c : 357 + 0xe]
     r4 = 0x42910f00    r5 = 0x00569060    r6 = 0x44effcfc    r7 = 0x00000078
     r8 = 0x4158e969    r9 = 0x41b079b0   r10 = 0x00020000    fp = 0x00000001
     sp = 0x42910f00    pc = 0x400f2c40
    Found by: call frame info

Comment 1

[leo.bugzilla.gecko]

Sorry for confusing. I wrote same call stack.
Second type is like below.

Thread 18 (crashed)
 0  libxul.so!android::OmxDecoder::ReadAudio [OmxDecoder.cpp : 689 + 0x6]
     r4 = 0x42e84aa0    r5 = 0x4252ed80    r6 = 0x00000000    r7 = 0xffffffff
     r8 = 0xffffffff    r9 = 0x00000000   r10 = 0x00000001    fp = 0x00000000
     sp = 0x4252ed20    lr = 0x408badbd    pc = 0x408badf2
    Found by: given as instruction pointer in context
 1  libxul.so!nsMediaOmxReader::DecodeAudioData [nsMediaOmxReader.cpp : 277 + 0x9]
     r4 = 0x42d45400    r5 = 0x00000000    r6 = 0x00000000    r7 = 0x00000001
     r8 = 0x000493e0    r9 = 0x00000000   r10 = 0x00000001    fp = 0x00000000
     sp = 0x4252ed78    pc = 0x408ba0e1
    Found by: call frame info
 2  libxul.so!nsBuiltinDecoderStateMachine::DecodeLoop [nsBuiltinDecoderStateMachine.cpp : 950 + 0x5]
     r4 = 0x43d03d00    r5 = 0x00000001    r6 = 0x00000000    r7 = 0x00000001
     r8 = 0x000493e0    r9 = 0x00000000   r10 = 0x00000001    fp = 0x00000000
     sp = 0x4252edd0    pc = 0x408f319f
    Found by: call frame info
 3  libxul.so!nsBuiltinDecoderStateMachine::DecodeThreadRun [nsBuiltinDecoderStateMachine.cpp : 535 + 0x5]
     r4 = 0x43d03d00    r5 = 0x45cba71c    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x4252eeb7    r9 = 0x4432d0cc   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252ee58    pc = 0x408f37ef
    Found by: call frame info
 4  libxul.so!nsRunnableMethodImpl<nsrefcnt (mozilla::dom::workers::DOMBindingBase::*)(), false>::Run [nsThreadUtils.h : 366 + 0x5]
     r4 = 0x4432d0a0    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x4252eeb7    r9 = 0x4432d0cc   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252ee68    pc = 0x404672dd
    Found by: call frame info
 5  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp : 620 + 0x5]
     r4 = 0x4432d0a0    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x00000000
     r8 = 0x4252eeb7    r9 = 0x4432d0cc   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252ee70    pc = 0x40c660cf
    Found by: call frame info
 6  libxul.so!NS_ProcessNextEvent_P [nsThreadUtils.cpp : 237 + 0xb]
     r4 = 0x00000001    r5 = 0x4252eec4    r6 = 0x00000000    r7 = 0x4432d0cc
     r8 = 0x41584969    r9 = 0x4435d710   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252eeb0    pc = 0x40c4649f
    Found by: call frame info
 7  libxul.so!nsThread::ThreadFunc [nsThread.cpp : 258 + 0x7]
     r4 = 0x4432d0a0    r5 = 0x4252eec4    r6 = 0x00000000    r7 = 0x4432d0cc
     r8 = 0x41584969    r9 = 0x4435d710   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252eec0    pc = 0x40c66519
    Found by: call frame info
 8  libnspr4.so!_pt_root [ptthread.c : 202 + 0x5]
     r4 = 0x4435d710    r5 = 0x4158cc0c    r6 = 0x00000000    r7 = 0x00000078
     r8 = 0x41584969    r9 = 0x4435d710   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252eee0    pc = 0x415849c9
    Found by: call frame info
 9  libc.so!__thread_entry [pthread.c : 217 + 0x6]
     r4 = 0x4252ef00    r5 = 0x41584969    r6 = 0x4435d710    r7 = 0x00000078
     r8 = 0x41584969    r9 = 0x4435d710   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252eef0    pc = 0x400790ec
    Found by: call frame info
10  libc.so!pthread_create [pthread.c : 357 + 0xe]
     r4 = 0x4252ef00    r5 = 0x01e97fe8    r6 = 0x445ffcfc    r7 = 0x00000078
     r8 = 0x41584969    r9 = 0x4435d710   r10 = 0x00020000    fp = 0x00000001
     sp = 0x4252ef00    pc = 0x40078c40
    Found by: call frame info

Comment 2

[leo.bugzilla.gecko]

FYI, Bug 890394 and Bug 884182 is already applied on LG side.

Comment 3

[leo.bugzilla.gecko]

I think I found the STR

Precondition : Put enough video files on SD card and follow each steps before scanning is over by video app.

1. Start video app.
2. Long push of home button.
3. Kill the video app on card view.

It's little hard to reproduce but I cannot get same crash by doing that.

According to logcat, MediaElement or Statemachine try to do something after H/W codec is released.

Comment 4

[leo.bugzilla.gecko]

Attachment: CrashLog.txt

Comment 5

[leo.bugzilla.gecko]

In Attachment #790105 [details], there're six try of thumbnail.

Please check last one, right before this line (the crash).
01-06 00:29:12.559 E       132      GeckoConsole                                 Content JS LOG at app://system.gaiamobile.org/shared/js/event_helper.js:4 in handle_mozChromeEvent: event_helper :handle_mozChromeEvent: handle-crash

Comment 6

[Sotaro Ikeda [:sotaro]]

From STR in comment #3, it seems to happens only when a application is killed and the application is not in hide state. Current nsBuiltinDecoderStateMachine's shut down is dependent on calling nsBuiltinDecoder::Shutdown() by nsHTMLMediaElement before thensBuiltinDecoderStateMachine's deletion.
It seems that nsBuiltinDecoder::Shutdown() is not called by nsHTMLMediaElement when app is killed via card view.

Comment 7

[Sotaro Ikeda [:sotaro]]

If From STR in comment #3 is the only STR, the defect create crash logs, but it does not make other side effect to a b2g phone user.

Comment 8

[leo.bugzilla.gecko]

(In reply to Sotaro Ikeda [:sotaro] from comment #7)
> If From STR in comment #3 is the only STR, the defect create crash logs, but
> it does not make other side effect to a b2g phone user.

Yes, even though there's crash popup, there's nothing to bother user's behavior.

Do you think that it can be fixed easily?
Or is it better to remove leo+ flag and fix it next version?

Please make some decision for this problem.

Comment 9

[Sotaro Ikeda [:sotaro]]

I feel that it is not possible to remove all crashes when killing an app from card view. Current gecko do not have a way to politey shutdown a content process.
But I think that to fix the crash around media/omx could be possible.

Comment 10

[Sotaro Ikeda [:sotaro]]

Attachment: temporary patch - add logout around content process' shutdown

Created the patch to analyze the content process's shutdown sequence.

Comment 11

[Sotaro Ikeda [:sotaro]]

Attachment: logout - STR in comment3 by using attachment 791727

Check STR in comment3 by using attachment 791727 [details] [diff] [review] on moz build v1.1 leo.

Comment 12

[Sotaro Ikeda [:sotaro]]

In attachment 791734 [details], nsBuiltinDecoder is set to dormant state after TabChild::DestroyWindow(). Therefore omx decoder's activity should be stopped after that.

Comment 13

[Sotaro Ikeda [:sotaro]]

I tried a lot of time the STR in comment3. I never got crash like comment #0. sometimes different type of crash happen during app killing.

Comment 14

[Sotaro Ikeda [:sotaro]]

It seems that I can regenerate the crash in comment #0. In the logout, there was only following log. When the log happens mAudioSource is NULL in OmxDecoder::ReadAudio().

>Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1)

Comment 15

[Sotaro Ikeda [:sotaro]]

By analyzing more, confirmed that current nsBuiltinDecoder can not handle dormant request in PLAY_STATE_LOADING state.

Comment 16

[Sotaro Ikeda [:sotaro]]

(In reply to leo.bugzilla.gecko from comment #8)
> Do you think that it can be fixed easily?
> Or is it better to remove leo+ flag and fix it next version?

This bug need to be fixed. leo+ bug.

Comment 17

[Sotaro Ikeda [:sotaro]]

nsBuiltinDecoder's dormant implementation re-uses PLAY_STATE_LOADING state. Then current implementation can not differentiate the following situations
 - [1] Set to dormant state during playback (other state than PLAY_STATE_LOADING)
 - [2] Set to dormant state when in PLAY_STATE_LOADING state.

In [2] case, MetadataLoaded() is called later. Need to cancel it.

Comment 18

[Sotaro Ikeda [:sotaro]]

Attachment: patch for b2g18 - block bogus MetadataLoaded event

Comment 19

[Sotaro Ikeda [:sotaro]]

Confirmed that attachment 791764 [details] [diff] [review] fixes the crash in comment #0 on v1.1 leo.

Comment 20

[Sotaro Ikeda [:sotaro]]

Leo, can you confirm if attachment 791764 [details] [diff] [review] works. n.b. it fixes only the crash in comment #0 on v1.1 leo.

Comment 21

[leo.bugzilla.gecko]

(In reply to Sotaro Ikeda [:sotaro] from comment #20)
> Leo, can you confirm if attachment 791764 [details] [diff] [review] works.
> n.b. it fixes only the crash in comment #0 on v1.1 leo.

I also confirm that the crash doesn't happen any more after patching it.

Comment 22

[Sotaro Ikeda [:sotaro]]

Attachment: patch for master - block bogus MetadataLoaded event

Comment 23

[Sotaro Ikeda [:sotaro]]

Comment on attachment 791764 [details] [diff] [review]
patch for b2g18 - block bogus MetadataLoaded event

Doublec, can you review the patch?

Comment 24

[cajbir (:cajbir)]

Comment on attachment 792242 [details] [diff] [review]
patch for master - block bogus MetadataLoaded event

Please provide a link to the results of a try server run with tests across all platforms then re-request review.

Comment 25

[cajbir (:cajbir)]

Comment on attachment 791764 [details] [diff] [review]
patch for b2g18 - block bogus MetadataLoaded event

Please re-request review once a try server run is provided as per the previous comment, thanks.

Comment 26

[cajbir (:cajbir)]

Is it possible to get a test for this?

Comment 27

[Sotaro Ikeda [:sotaro]]

Attachment: patch v2 for master - block bogus MetadataLoaded event

Fix the flag as consistent.

Comment 28

[Sotaro Ikeda [:sotaro]]

Attachment: patch v2 for b2g18 - block bogus MetadataLoaded event

Fix the flag as consistent.

Comment 29

[Sotaro Ikeda [:sotaro]]

https://tbpl.mozilla.org/?tree=Try&rev=f76f62caa7bb

Comment 30

[Sotaro Ikeda [:sotaro]]

(In reply to Chris Double (:doublec) from comment #26)
> Is it possible to get a test for this?

It seems possible. Is it OK to add a test in a follow up bug?

Comment 31

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #30)
> (In reply to Chris Double (:doublec) from comment #26)
> > Is it possible to get a test for this?
> 
> It seems possible. Is it OK to add a test in a follow up bug?

Sorry, I re-think about it. It seems more difficult than I thought. The timing needs to be following. Always call this timing using mochi-test seems not possible.
 - Set video element in hide state when MediaDecoder is in PLAY_STATE_LOADING and OmxDecoder allocated the video decoder.

Comment 32

[Sotaro Ikeda [:sotaro]]

doublec, from comment #31, I think that it is not easy to implement test for it. Is it ok not to add a test for it?

Comment 33

[cajbir (:cajbir)]

If you think it's not possible, that's fine.

Comment 34

[Sotaro Ikeda [:sotaro]]

Attachment: patch v3 for master - block bogus MetadataLoaded event

Add a header to the patach. Carry "chris.double: review+".

Comment 35

[Sotaro Ikeda [:sotaro]]

Attachment: patch v3 for b2g18 - block bogus MetadataLoaded event

Add a header to the patch. Carry "chris.double: review+".

Comment 36

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/b2g-inbound/rev/7017360dbdf5

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7017360dbdf5

Comment 38

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/328b3b8158ee

Comment 39

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/328b3b8158ee