1324530 - Intermittent dom/media/mediasource/test/test_EndedEvent.html,test_Eviction_mp4.html,test_BufferedSeek.html | Test timed out.

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=40785270&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/PYNAOtQiTBOObutik0w8-Q/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Intermittent Failures Robot]

17 failures in 117 pushes (0.145 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6
* try: 1
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2016-12-20&endday=2016-12-20&tree=all

Comment 2

[JW Wang [:jwwang] [:jw_wang]]

I think this has something to do with recent Android changes.

Comment 3

[John Lin [:jhlin][:jolin]]

(In reply to JW Wang [:jwwang] [:jw_wang] from comment #2)
> I think this has something to do with recent Android changes.

 Couldn't see how remote decoding would cause the problem. The log below shows fetchWithXHR() took > 5 minutes to load data [1] and threw error even before appending data to source buffer.
 Looks like bug 1324491 have the same symptom too.
 
 Is there log for XMLHttpRequest processing that can be enabled to help investigating why it took so long?

[task 2016-12-19T18:06:48.963192Z] 18:06:48     INFO -  11 INFO TEST-START | dom/media/mediasource/test/test_EndedEvent.html
[task 2016-12-19T18:12:00.091479Z] 18:12:00     INFO -  Buffered messages logged at 18:06:45
[task 2016-12-19T18:12:00.091590Z] 18:12:00     INFO -  12 INFO TEST-PASS | dom/media/mediasource/test/test_EndedEvent.html | fetchWithXHR load uri='seek.webm' status=200
[task 2016-12-19T18:12:00.091651Z] 18:12:00     INFO -  Buffered messages finished
[task 2016-12-19T18:12:00.091732Z] 18:12:00     INFO -  13 INFO TEST-UNEXPECTED-FAIL | dom/media/mediasource/test/test_EndedEvent.html | Test timed out.
[task 2016-12-19T18:12:00.091785Z] 18:12:00     INFO -      reportError@SimpleTest/TestRunner.js:114:7
[task 2016-12-19T18:12:00.091839Z] 18:12:00     INFO -      TestRunner._checkForHangs@SimpleTest/TestRunner.js:135:7
[task 2016-12-19T18:12:00.091900Z] 18:12:00     INFO -  14 INFO TEST-OK | dom/media/mediasource/test/test_EndedEvent.html | took 312381ms

[1] http://searchfox.org/mozilla-central/source/dom/media/mediasource/test/mediasource.js#42

Comment 4

[JW Wang [:jwwang] [:jw_wang]]

The logs are dumped when the test case times out. So the timestamps are not correct.

Comment 5

[John Lin [:jhlin][:jolin]]

(In reply to JW Wang [:jwwang] [:jw_wang] from comment #4)
> The logs are dumped when the test case times out. So the timestamps are not
> correct.

 Logcat shows some decoder error message. Will continue to investigate.

Comment 6

[Intermittent Failures Robot]

16 failures in 123 pushes (0.13 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 10
* mozilla-inbound: 6

Platform breakdown:
* android-4-3-armv7-api15: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2016-12-21&endday=2016-12-21&tree=all

Comment 8

[John Lin [:jhlin][:jolin]]

The remote decoding process crashed because of a race condition that ICodec.flush() was executed between ICodec.dequeueInput() and ICodec.queueInput().

Comment 9

[Intermittent Failures Robot]

68 failures in 609 pushes (0.112 failures/push) were associated with this bug in the last 7 days. 

This is the #4 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 38
* mozilla-inbound: 23
* mozilla-central: 3
* try: 2
* graphics: 2

Platform breakdown:
* android-4-3-armv7-api15: 68

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2016-12-19&endday=2016-12-25&tree=all

Comment 10

[John Lin [:jhlin][:jolin]]

Besides the problem in comment 8, there is another issue in RemoteManager.waitConnection(): when serivce is ready and onServiceConnected() finished before latchCreation [1], it will be blocked for 5s.

[1] http://searchfox.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/media/RemoteManager.java#99

Comment 11

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1324530 - part 1: add method to AsyncCodec for resuming receiving input buffers after flush().

2 birds with 1 stone:
 - according to MediaCodec doc, when running in async mode start() must be called after flush() to 'resume receiving input buffers'.
 - in JellyBeanAsyncCodec's case, explicit resuming polling input buffers helps Codec decide which buffers are invalid by flush().

Review commit: https://reviewboard.mozilla.org/r/101256/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/101256/

Comment 12

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1324530 - part 2: rewrite the logic of discarding invalid buffers for flush.

Review commit: https://reviewboard.mozilla.org/r/101258/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/101258/

Comment 13

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1324530 - part 3: make codec methods that change state(start, stop, flush, release) synchronous.

Review commit: https://reviewboard.mozilla.org/r/101260/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/101260/

Comment 14

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1324530 - part 4: improve remote manager init wait time.

Fix the issue that RemoteManager.init() will be block for 5s when onServiceConnected() is
finished before latch creation.

Review commit: https://reviewboard.mozilla.org/r/101262/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/101262/

Comment 15

[John Lin [:jhlin][:jolin]]

Also observed that sometimes 'SimpleTest.finish.bind(null)' [1] doesn't invoke SimpleTest.finish() and causes time-out. Directly calling it seems fix the problem.

Comment 16

[John Lin [:jhlin][:jolin]]

^ [1] http://searchfox.org/mozilla-central/source/dom/media/mediasource/test/test_EndedEvent.html#26

Comment 17

[John Lin [:jhlin][:jolin]]

(In reply to John Lin [:jolin][:jhlin] from comment #15)
> Also observed that sometimes 'SimpleTest.finish.bind(null)' [1] doesn't
> invoke SimpleTest.finish() and causes time-out. Directly calling it seems
> fix the problem.

Never mind. Turns out that's a mistake in my try push.

Comment 18

[Intermittent Failures Robot]

33 failures in 305 pushes (0.108 failures/push) were associated with this bug in the last 7 days. 

This is the #13 most frequent failure this week. 

Repository breakdown:
* autoland: 15
* mozilla-inbound: 8
* graphics: 4
* try: 3
* mozilla-central: 3

Platform breakdown:
* android-4-3-armv7-api15: 33

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2016-12-26&endday=2017-01-01&tree=all

Comment 19

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822316 [details]
Bug 1324530 - part 1: add method to AsyncCodec for resuming receiving input buffers after flush().

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101256/diff/1-2/

Comment 20

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822317 [details]
Bug 1324530 - part 2: rewrite the logic of discarding invalid buffers for flush.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101258/diff/1-2/

Comment 21

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822318 [details]
Bug 1324530 - part 3: make codec methods that change state(start, stop, flush, release) synchronous.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101260/diff/1-2/

Comment 22

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822319 [details]
Bug 1324530 - part 4: improve remote manager init wait time.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101262/diff/1-2/

Comment 23

[Eugen Sawin [:esawin]]

Comment on attachment 8822316 [details]
Bug 1324530 - part 1: add method to AsyncCodec for resuming receiving input buffers after flush().

https://reviewboard.mozilla.org/r/101256/#review102518

::: mobile/android/base/java/org/mozilla/gecko/media/AsyncCodec.java:32
(Diff revision 2)
>      public abstract void configure(MediaFormat format, Surface surface, MediaCrypto crypto, int flags);
>      public abstract boolean isAdaptivePlaybackSupported(String mimeType);
>      public abstract void start();
>      public abstract void stop();
>      public abstract void flush();
> +    public abstract void resumeReceivingInputs(); // Must be called after flush().

Can you move the comment on its own line on top of it?

::: mobile/android/base/java/org/mozilla/gecko/media/Codec.java:415
(Diff revision 2)
>          } catch (Exception e) {
>              reportError(Error.FATAL, e);
>          }
>  
>          mFlushing = false;
>          if (DEBUG) Log.d(LOGTAG, "flushed " + this);

Unrelated to patch: please use curly brackets and indentation for single-line blocks, too.

::: mobile/android/base/java/org/mozilla/gecko/media/JellyBeanAsyncCodec.java:328
(Diff revision 2)
>          mInputBuffers = mCodec.getInputBuffers();
> +        startPollingInputs();
> +        mOutputBuffers = mCodec.getOutputBuffers();
> +    }
> +
> +    private void startPollingInputs() {

Can't we just use resumeReceivingInput instead and implement it there?

Comment 24

[Eugen Sawin [:esawin]]

Comment on attachment 8822317 [details]
Bug 1324530 - part 2: rewrite the logic of discarding invalid buffers for flush.

https://reviewboard.mozilla.org/r/101258/#review102524

Comment 25

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822316 [details]
Bug 1324530 - part 1: add method to AsyncCodec for resuming receiving input buffers after flush().

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101256/diff/2-3/

Comment 26

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822317 [details]
Bug 1324530 - part 2: rewrite the logic of discarding invalid buffers for flush.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101258/diff/2-3/

Comment 27

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822318 [details]
Bug 1324530 - part 3: make codec methods that change state(start, stop, flush, release) synchronous.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101260/diff/2-3/

Comment 28

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822319 [details]
Bug 1324530 - part 4: improve remote manager init wait time.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101262/diff/2-3/

Comment 29

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822316 [details]
Bug 1324530 - part 1: add method to AsyncCodec for resuming receiving input buffers after flush().

https://reviewboard.mozilla.org/r/101256/#review102518

> Can you move the comment on its own line on top of it?

Will do in next revision.

> Unrelated to patch: please use curly brackets and indentation for single-line blocks, too.

Will do.

> Can't we just use resumeReceivingInput instead and implement it there?

Okay.

Comment 30

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8822318 [details]
Bug 1324530 - part 3: make codec methods that change state(start, stop, flush, release) synchronous.

https://reviewboard.mozilla.org/r/101260/#review103066

Comment 31

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8822319 [details]
Bug 1324530 - part 4: improve remote manager init wait time.

https://reviewboard.mozilla.org/r/101262/#review103072

::: mobile/android/base/java/org/mozilla/gecko/media/RemoteManager.java:95
(Diff revision 3)
> +            while (mRemote != null) {
> +                try {
> +                    wait(1000);
> -        } catch (InterruptedException e) {
> +                } catch (InterruptedException e) {
> -            Log.e(LOGTAG, "service not connected in 5 seconds. Stop waiting.");
> -            e.printStackTrace();
> +                    e.printStackTrace();

wrap in 'if (DEBUG)'?

Comment 32

[Intermittent Failures Robot]

21 failures in 132 pushes (0.159 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 13
* mozilla-inbound: 7
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-05&endday=2017-01-05&tree=all

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cce21c0cb2dd
part 1: add method to AsyncCodec for resuming receiving input buffers after flush(). r=esawin
https://hg.mozilla.org/integration/autoland/rev/3e71466a060d
part 2: rewrite the logic of discarding invalid buffers for flush. r=esawin
https://hg.mozilla.org/integration/autoland/rev/bedf2a853f91
part 3: make codec methods that change state(start, stop, flush, release) synchronous. r=snorp
https://hg.mozilla.org/integration/autoland/rev/c520d79f9611
part 4: improve remote manager init wait time. r=snorp

Comment 34

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822319 [details]
Bug 1324530 - part 4: improve remote manager init wait time.

https://reviewboard.mozilla.org/r/101262/#review103072

> wrap in 'if (DEBUG)'?

Oops, I missed this one before landing. :$ Will add it in a later patch.

Comment 35

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backe dout for failing passwd-1.xul on Android 4.3 debug and number-2.xul on Android 4.3 opt:

https://hg.mozilla.org/integration/autoland/rev/7f57c8a2f5259d30a8fc6c1804d9ff3bb5312c68
https://hg.mozilla.org/integration/autoland/rev/7f57c8a2f5259d30a8fc6c1804d9ff3bb5312c68
https://hg.mozilla.org/integration/autoland/rev/89cccf51902d2d6dc7710beaf2c0a086b1206e0f
https://hg.mozilla.org/integration/autoland/rev/f869523d6f1f9a6e5fd7e68b4e58384372a9c603

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c520d79f9611e41a7b77584dd694c97012aebb99
Failure log debug: https://treeherder.mozilla.org/logviewer.html#?job_id=66678470&repo=autoland
[task 2017-01-06T06:47:26.716691Z] 06:47:26     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/editor/reftests/xul/passwd-1.xul | 44 / 304 (14%)
[task 2017-01-06T06:47:38.295114Z] 06:47:38     INFO -  INFO | automation.py | Application ran for: 0:10:38.079740
[task 2017-01-06T06:47:38.295359Z] 06:47:38     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpCFu3Rspidlog
[task 2017-01-06T06:47:38.833635Z] 06:47:38     INFO -  /data/tombstones does not exist; tombstone check skipped
[task 2017-01-06T06:47:39.793147Z] 06:47:39     INFO -  REFTEST INFO | Copy/paste: /usr/local/bin/linux64-minidump_stackwalk /tmp/tmp7_0VVI/32633aed-c648-58a1-32cd3c31-4369245f.dmp /home/worker/workspace/build/symbols
[task 2017-01-06T06:47:47.038212Z] 06:47:47     INFO -  REFTEST INFO | Saved minidump as /home/worker/workspace/build/blobber_upload_dir/32633aed-c648-58a1-32cd3c31-4369245f.dmp
[task 2017-01-06T06:47:47.038503Z] 06:47:47     INFO -  REFTEST INFO | Saved app info as /home/worker/workspace/build/blobber_upload_dir/32633aed-c648-58a1-32cd3c31-4369245f.extra
[task 2017-01-06T06:47:47.042093Z] 06:47:47     INFO -  REFTEST PROCESS-CRASH | http://10.0.2.2:8854/tests/editor/reftests/xul/passwd-1.xul == http://10.0.2.2:8854/tests/editor/reftests/xul/passwd-ref.xul | application crashed [@ mozilla::java::CodecProxy::Flush]
[task 2017-01-06T06:47:47.042328Z] 06:47:47     INFO -  Crash dump filename: /tmp/tmp7_0VVI/32633aed-c648-58a1-32cd3c31-4369245f.dmp
[task 2017-01-06T06:47:47.042512Z] 06:47:47     INFO -  Operating system: Android
[task 2017-01-06T06:47:47.042732Z] 06:47:47     INFO -                    0.0.0 Linux 2.6.29-gea477bb #1 Wed Sep 26 11:04:45 PDT 2012 armv7l
[task 2017-01-06T06:47:47.042963Z] 06:47:47     INFO -  CPU: arm
[task 2017-01-06T06:47:47.043226Z] 06:47:47     INFO -       ARMv7 ARM Cortex-A8 features: swp,half,thumb,fastmult,vfpv2,edsp,neon,vfpv3
[task 2017-01-06T06:47:47.043449Z] 06:47:47     INFO -       1 CPU
[task 2017-01-06T06:47:47.043641Z] 06:47:47     INFO -  GPU: UNKNOWN
[task 2017-01-06T06:47:47.043877Z] 06:47:47     INFO -  Crash reason:  SIGSEGV
[task 2017-01-06T06:47:47.044074Z] 06:47:47     INFO -  Crash address: 0x0
[task 2017-01-06T06:47:47.044303Z] 06:47:47     INFO -  Process uptime: not available
[task 2017-01-06T06:47:47.045376Z] 06:47:47     INFO -  Thread 54 (crashed)
[task 2017-01-06T06:47:47.045445Z] 06:47:47     INFO -   0  libxul.so!mozilla::java::CodecProxy::Flush [Accessors.h:c520d79f9611 : 59 + 0x2]
[task 2017-01-06T06:47:47.045521Z] 06:47:47     INFO -       r0 = 0x00000081    r1 = 0xea2fe500    r2 = 0x5cb78587    r3 = 0x00000000
[task 2017-01-06T06:47:47.045581Z] 06:47:47     INFO -       r4 = 0x6143fc38    r5 = 0x00000001    r6 = 0x2a42b518    r7 = 0x5d78719c
[task 2017-01-06T06:47:47.045640Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000   r12 = 0x00000003
[task 2017-01-06T06:47:47.045822Z] 06:47:47     INFO -       fp = 0x00000001    sp = 0x6143fc10    lr = 0x5b6deb03    pc = 0x5b6df82c
[task 2017-01-06T06:47:47.045902Z] 06:47:47     INFO -      Found by: given as instruction pointer in context
[task 2017-01-06T06:47:47.045968Z] 06:47:47     INFO -   1  libxul.so!mozilla::RemoteDataDecoder::Flush [RemoteDataDecoder.cpp:c520d79f9611 : 583 + 0x5]
[task 2017-01-06T06:47:47.046025Z] 06:47:47     INFO -       r4 = 0x54467f00    r5 = 0x00000002    r6 = 0x0247d53c    r7 = 0x60a9f588
[task 2017-01-06T06:47:47.046082Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000    fp = 0x00000001
[task 2017-01-06T06:47:47.046256Z] 06:47:47     INFO -       sp = 0x6143fc30    pc = 0x5b3827bd
[task 2017-01-06T06:47:47.046418Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.046661Z] 06:47:47     INFO -   2  libxul.so!mozilla::MediaFormatReader::DecoderData::Flush [MediaFormatReader.h:c520d79f9611 : 365 + 0x3]
[task 2017-01-06T06:47:47.046892Z] 06:47:47     INFO -       r4 = 0x60a9f588    r5 = 0x00000002    r6 = 0x0247d53c    r7 = 0x60a9f588
[task 2017-01-06T06:47:47.047101Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000    fp = 0x00000001
[task 2017-01-06T06:47:47.047375Z] 06:47:47     INFO -       sp = 0x6143fc48    pc = 0x5b2e99ff
[task 2017-01-06T06:47:47.047527Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.047751Z] 06:47:47     INFO -   3  libxul.so!mozilla::MediaFormatReader::Reset [MediaFormatReader.cpp:c520d79f9611 : 2159 + 0x3]
[task 2017-01-06T06:47:47.048005Z] 06:47:47     INFO -       r3 = 0x00000000    r4 = 0x60a9f000    r5 = 0x00000002    r6 = 0x0247d53c
[task 2017-01-06T06:47:47.048202Z] 06:47:47     INFO -       r7 = 0x60a9f588    r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000
[task 2017-01-06T06:47:47.048407Z] 06:47:47     INFO -       fp = 0x00000001    sp = 0x6143fc58    pc = 0x5b2e9c89
[task 2017-01-06T06:47:47.048650Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.048872Z] 06:47:47     INFO -   4  libxul.so!mozilla::MediaFormatReader::ResetDecode [MediaFormatReader.cpp:c520d79f9611 : 2078 + 0x7]
[task 2017-01-06T06:47:47.049078Z] 06:47:47     INFO -       r4 = 0x60a9f000    r5 = 0x6143fc78    r6 = 0x6143fca8    r7 = 0x60a9f588
[task 2017-01-06T06:47:47.049274Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000    fp = 0x00000001
[task 2017-01-06T06:47:47.049473Z] 06:47:47     INFO -       sp = 0x6143fc70    pc = 0x5b30922f
[task 2017-01-06T06:47:47.049650Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.049961Z] 06:47:47     INFO -   5  libxul.so!mozilla::detail::RunnableMethodImpl<nsresult (mozilla::MediaDecoderReader::*)(mozilla::EnumSet<mozilla::TrackInfo::TrackType>), true, false, mozilla::EnumSet<mozilla::TrackInfo::TrackType> >::Run [nsThreadUtils.h:c520d79f9611 : 791 + 0x3]
[task 2017-01-06T06:47:47.050140Z] 06:47:47     INFO -       r4 = 0x00000000    r5 = 0x00000000    r6 = 0x5d73be54    r7 = 0x54021a60
[task 2017-01-06T06:47:47.050357Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000    fp = 0x00000001
[task 2017-01-06T06:47:47.050582Z] 06:47:47     INFO -       sp = 0x6143fca8    pc = 0x5b2a01cd
[task 2017-01-06T06:47:47.050805Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.051177Z] 06:47:47     INFO -   6  libxul.so!mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run [TaskDispatcher.h:c520d79f9611 : 193 + 0x3]
[task 2017-01-06T06:47:47.051527Z] 06:47:47     INFO -       r4 = 0x00000001    r5 = 0x54021a60    r6 = 0x5d73be54    r7 = 0x54021a60
[task 2017-01-06T06:47:47.052127Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000    fp = 0x00000001
[task 2017-01-06T06:47:47.052234Z] 06:47:47     INFO -       sp = 0x6143fcc8    pc = 0x5a4b7acd
[task 2017-01-06T06:47:47.052324Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.052581Z] 06:47:47     INFO -   7  libxul.so!mozilla::TaskQueue::Runner::Run [TaskQueue.cpp:c520d79f9611 : 232 + 0x5]
[task 2017-01-06T06:47:47.052779Z] 06:47:47     INFO -       r3 = 0x5a4b7a79    r4 = 0x54021e54    r5 = 0x6143fce0    r6 = 0x5d73be54
[task 2017-01-06T06:47:47.053028Z] 06:47:47     INFO -       r7 = 0x54021a60    r8 = 0x00000000    r9 = 0x54021e40   r10 = 0x00000000
[task 2017-01-06T06:47:47.053246Z] 06:47:47     INFO -       fp = 0x00000001    sp = 0x6143fcd8    pc = 0x5a4bf6b1
[task 2017-01-06T06:47:47.053474Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.053735Z] 06:47:47     INFO -   8  libxul.so!nsThreadPool::Run [nsThreadPool.cpp:c520d79f9611 : 226 + 0x5]
[task 2017-01-06T06:47:47.053948Z] 06:47:47     INFO -       r4 = 0x5527a200    r5 = 0x00000000    r6 = 0x6143fd78    r7 = 0x6143fd68
[task 2017-01-06T06:47:47.054194Z] 06:47:47     INFO -       r8 = 0x6143fd74    r9 = 0x00000000   r10 = 0x00000000    fp = 0x54021e40
[task 2017-01-06T06:47:47.054404Z] 06:47:47     INFO -       sp = 0x6143fd40    pc = 0x5a4c2479
[task 2017-01-06T06:47:47.054557Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.054764Z] 06:47:47     INFO -   9  libxul.so!nsThread::ProcessNextEvent [nsThread.cpp:c520d79f9611 : 1213 + 0x3]
[task 2017-01-06T06:47:47.055017Z] 06:47:47     INFO -       r4 = 0x53fba870    r5 = 0x00000000    r6 = 0x00000001    r7 = 0x6143fdc4
[task 2017-01-06T06:47:47.055180Z] 06:47:47     INFO -       r8 = 0x6143fdc0    r9 = 0x6143fdbc   r10 = 0x00000000    fp = 0x6143fe1f
[task 2017-01-06T06:47:47.055380Z] 06:47:47     INFO -       sp = 0x6143fda8    pc = 0x5a4bde1f
[task 2017-01-06T06:47:47.055562Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.055804Z] 06:47:47     INFO -  10  libxul.so!NS_ProcessNextEvent [nsThreadUtils.cpp:c520d79f9611 : 381 + 0xd]
[task 2017-01-06T06:47:47.056012Z] 06:47:47     INFO -       r4 = 0x53fba870    r5 = 0x00000001    r6 = 0x53f4edd0    r7 = 0x5d73be54
[task 2017-01-06T06:47:47.056219Z] 06:47:47     INFO -       r8 = 0x54172ed0    r9 = 0x53fba870   r10 = 0xffffffff    fp = 0x2a429310
[task 2017-01-06T06:47:47.056380Z] 06:47:47     INFO -       sp = 0x6143fe10    pc = 0x5a4d9981
[task 2017-01-06T06:47:47.056564Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.056800Z] 06:47:47     INFO -  11  libxul.so!mozilla::ipc::MessagePumpForNonMainThreads::Run [MessagePump.cpp:c520d79f9611 : 368 + 0x7]
[task 2017-01-06T06:47:47.057003Z] 06:47:47     INFO -       r4 = 0x53f4edc0    r5 = 0x00000000    r6 = 0x53f4edd0    r7 = 0x5d73be54
[task 2017-01-06T06:47:47.057225Z] 06:47:47     INFO -       r8 = 0x54172ed0    r9 = 0x53fba870   r10 = 0xffffffff    fp = 0x2a429310
[task 2017-01-06T06:47:47.057511Z] 06:47:47     INFO -       sp = 0x6143fe30    pc = 0x5a6ce409
[task 2017-01-06T06:47:47.057725Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.057948Z] 06:47:47     INFO -  12  libxul.so!MessageLoop::RunInternal [message_loop.cc:c520d79f9611 : 238 + 0x5]
[task 2017-01-06T06:47:47.058156Z] 06:47:47     INFO -       r4 = 0x54172ed0    r5 = 0x6143fe98    r6 = 0x6143fea0    r7 = 0x6143fe9c
[task 2017-01-06T06:47:47.058360Z] 06:47:47     INFO -       r8 = 0x54172ed0    r9 = 0x53fba880   r10 = 0xffffffff    fp = 0x2a429310
[task 2017-01-06T06:47:47.058589Z] 06:47:47     INFO -       sp = 0x6143fe60    pc = 0x5a6b3d79
[task 2017-01-06T06:47:47.058816Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.059046Z] 06:47:47     INFO -  13  libxul.so!MessageLoop::Run [message_loop.cc:c520d79f9611 : 231 + 0x5]
[task 2017-01-06T06:47:47.059215Z] 06:47:47     INFO -       r4 = 0x54172ed0    r5 = 0x6143fe98    r6 = 0x6143fea0    r7 = 0x6143fe9c
[task 2017-01-06T06:47:47.059421Z] 06:47:47     INFO -       r8 = 0x54172ed0    r9 = 0x53fba880   r10 = 0xffffffff    fp = 0x2a429310
[task 2017-01-06T06:47:47.059624Z] 06:47:47     INFO -       sp = 0x6143fe78    pc = 0x5a6b3ec1
[task 2017-01-06T06:47:47.059875Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.060071Z] 06:47:47     INFO -  14  libxul.so!nsThread::ThreadFunc [nsThread.cpp:c520d79f9611 : 467 + 0x3]
[task 2017-01-06T06:47:47.060312Z] 06:47:47     INFO -       r4 = 0x53fba870    r5 = 0x6143fe98    r6 = 0x6143fea0    r7 = 0x6143fe9c
[task 2017-01-06T06:47:47.060540Z] 06:47:47     INFO -       r8 = 0x54172ed0    r9 = 0x53fba880   r10 = 0xffffffff    fp = 0x2a429310
[task 2017-01-06T06:47:47.060721Z] 06:47:47     INFO -       sp = 0x6143fe90    pc = 0x5a4b9723
[task 2017-01-06T06:47:47.060975Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.061201Z] 06:47:47     INFO -  15  libnss3.so!_pt_root [ptthread.c:c520d79f9611 : 216 + 0x5]
[task 2017-01-06T06:47:47.061569Z] 06:47:47     INFO -       r4 = 0x53f60c80    r5 = 0x00000000    r6 = 0x52e788d4    r7 = 0x2a429310
[task 2017-01-06T06:47:47.061818Z] 06:47:47     INFO -       r8 = 0x00000000    r9 = 0x61400000   r10 = 0x6143ff00    fp = 0x2a429310
[task 2017-01-06T06:47:47.061953Z] 06:47:47     INFO -       sp = 0x6143fec8    pc = 0x52da994d
[task 2017-01-06T06:47:47.062185Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.062422Z] 06:47:47     INFO -  16  libc.so + 0xca5a
[task 2017-01-06T06:47:47.062591Z] 06:47:47     INFO -       r4 = 0x6143ff00    r5 = 0x2a429310    r6 = 0x52da9869    r7 = 0x53f60c80
[task 2017-01-06T06:47:47.062824Z] 06:47:47     INFO -       r8 = 0x6113fad4    r9 = 0x61400000   r10 = 0x6143ff00    fp = 0x2a429310
[task 2017-01-06T06:47:47.063008Z] 06:47:47     INFO -       sp = 0x6143fee8    pc = 0x40033a5c
[task 2017-01-06T06:47:47.063182Z] 06:47:47     INFO -      Found by: call frame info
[task 2017-01-06T06:47:47.063353Z] 06:47:47     INFO -  17  libc.so + 0xcbd6
[task 2017-01-06T06:47:47.063531Z] 06:47:47     INFO -       sp = 0x6143ff00    pc = 0x40033bd8
[task 2017-01-06T06:47:47.063747Z] 06:47:47     INFO -      Found by: stack scanning

Comment 36

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/415703322ec5
Backed out changeset 3e71466a060d

Comment 37

[John Lin [:jhlin][:jolin]]

The crashes were in fact caused by dom/html/reftest/bug1228601-video-rotation-90.html.
In that test, the SW H.264/AVC decoder failed to decode input samples and trigger a bug in Codec.InputProcessor that EOS sample are sent to SamplePool to be recycled and cause NullPointerException at [1].

[1] http://searchfox.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/media/SamplePool.java#55

Comment 38

[John Lin [:jhlin][:jolin]]

Attachment: bug 1324530 - part 5: do not recycle EOS input sample.

Review commit: https://reviewboard.mozilla.org/r/102922/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/102922/

Comment 39

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822316 [details]
Bug 1324530 - part 1: add method to AsyncCodec for resuming receiving input buffers after flush().

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101256/diff/3-4/

Comment 40

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822317 [details]
Bug 1324530 - part 2: rewrite the logic of discarding invalid buffers for flush.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101258/diff/3-4/

Comment 41

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822318 [details]
Bug 1324530 - part 3: make codec methods that change state(start, stop, flush, release) synchronous.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101260/diff/3-4/

Comment 42

[John Lin [:jhlin][:jolin]]

Comment on attachment 8822319 [details]
Bug 1324530 - part 4: improve remote manager init wait time.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/101262/diff/3-4/

Comment 43

[Intermittent Failures Robot]

15 failures in 134 pushes (0.112 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 7
* autoland: 6
* try: 2

Platform breakdown:
* android-4-3-armv7-api15: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-06&endday=2017-01-06&tree=all

Comment 44

[Intermittent Failures Robot]

23 failures in 57 pushes (0.404 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* graphics: 12
* autoland: 6
* mozilla-inbound: 4
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 12
* linux64: 5
* windows8-64: 4
* osx-10-7: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-07&endday=2017-01-07&tree=all

Comment 45

[Intermittent Failures Robot]

84 failures in 563 pushes (0.149 failures/push) were associated with this bug in the last 7 days. 

This is the #12 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* autoland: 36
* mozilla-inbound: 29
* graphics: 13
* try: 3
* mozilla-central: 3

Platform breakdown:
* android-4-3-armv7-api15: 73
* linux64: 5
* windows8-64: 4
* osx-10-7: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-02&endday=2017-01-08&tree=all

Comment 47

[Intermittent Failures Robot]

16 failures in 128 pushes (0.125 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 10
* autoland: 6

Platform breakdown:
* android-4-3-armv7-api15: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-10&endday=2017-01-10&tree=all

Comment 48

[Intermittent Failures Robot]

22 failures in 155 pushes (0.142 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 11
* autoland: 10
* try: 1

Platform breakdown:
* android-4-3-armv7-api15: 22

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-11&endday=2017-01-11&tree=all

Comment 49

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8824411 [details]
bug 1324530 - part 5: do not recycle EOS input sample.

https://reviewboard.mozilla.org/r/102922/#review104954

Comment 50

[Intermittent Failures Robot]

30 failures in 137 pushes (0.219 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* autoland: 13
* mozilla-inbound: 10
* try: 4
* mozilla-central: 2
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 30

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-12&endday=2017-01-12&tree=all

Comment 51

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6179997594f3
part 1: add method to AsyncCodec for resuming receiving input buffers after flush(). r=esawin
https://hg.mozilla.org/integration/autoland/rev/7398e758604e
part 2: rewrite the logic of discarding invalid buffers for flush. r=esawin
https://hg.mozilla.org/integration/autoland/rev/435b42d165ee
part 3: make codec methods that change state(start, stop, flush, release) synchronous. r=snorp
https://hg.mozilla.org/integration/autoland/rev/d9544d6b0b5d
part 4: improve remote manager init wait time. r=snorp
https://hg.mozilla.org/integration/autoland/rev/38e88586d122
part 5: do not recycle EOS input sample. r=snorp

Comment 52

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/6179997594f3
https://hg.mozilla.org/mozilla-central/rev/7398e758604e
https://hg.mozilla.org/mozilla-central/rev/435b42d165ee
https://hg.mozilla.org/mozilla-central/rev/d9544d6b0b5d
https://hg.mozilla.org/mozilla-central/rev/38e88586d122

Comment 53

[Intermittent Failures Robot]

104 failures in 722 pushes (0.144 failures/push) were associated with this bug in the last 7 days. 

This is the #13 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-inbound: 49
* autoland: 42
* try: 5
* mozilla-central: 5
* graphics: 3

Platform breakdown:
* android-4-3-armv7-api15: 104

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324530&startday=2017-01-09&endday=2017-01-15&tree=all