[Mdm2] Intermittent test_peerConnection_basicVideoRemoteHwEncoder.html | java-exception org.mozilla.gecko.mozglue.GeckoLoader$AbortException: abort() called from :0xdd6216d9 () at org.mozilla.gecko.mozglue.GeckoLoader.abort(GeckoLoader.java:532)

RESOLVED FIXED in Firefox 56

Status

()

Core
WebRTC
P1
normal
Rank:
18
RESOLVED FIXED
4 months ago
3 months ago

People

(Reporter: Treeherder Bug Filer, Assigned: mchiang)

Tracking

(Blocks: 1 bug, {crash, intermittent-failure})

unspecified
mozilla56
Unspecified
Android
crash, intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox54 unaffected, firefox55 unaffected, firefox56 fixed)

Details

(Whiteboard: [stockwell fixed:product])

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

4 months ago
treeherder
Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=106709160&repo=mozilla-inbound

https://autophone.s3.amazonaws.com/v1/task/dEj-bVn_RLiq82A0fya16w/runs/0/artifacts/public/build/adedeeeb-5d2e-400b-83b9-2c7f8a3860dd-autophone.log

Updated

4 months ago
Duplicate of this bug: 1372688
Rank: 25
OS: Unspecified → Android
Priority: -- → P2

Comment 2

4 months ago
17 failures in 147 pushes (0.116 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 9
* autoland: 6
* mozilla-central: 2

Platform breakdown:
* android-6-0-armv8-api15: 9
* android-7-1-armv8-api15: 7
* android-5-1-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-16&endday=2017-06-16&tree=all

Comment 3

4 months ago
45 failures in 814 pushes (0.055 failures/push) were associated with this bug in the last 7 days. 

This is the #45 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 23
* mozilla-inbound: 12
* mozilla-central: 10

Platform breakdown:
* android-6-0-armv8-api15: 22
* android-7-1-armv8-api15: 20
* android-5-1-armv7-api15: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-12&endday=2017-06-18&tree=all
this is autophone specifc and started around June 12th.  Given the high failure rate I would like to look into this and resolve this failure (fix, reduce, disable) in the next 2 weeks.

:bc - who is your best point of contact for dom-media tests?  (I see :bechen working on a av/playback bug, but this looks like webrtc).
Flags: needinfo?(bob)
Whiteboard: [stockwell needswork]

Comment 5

4 months ago
Al, I know you have been working on Mdm1 to get it back running. Could you look into this one as well?
Blocks: 1232305
Flags: needinfo?(bob) → needinfo?(alwu)
Summary: Intermittent test_peerConnection_basicVideoRemoteHwEncoder.html | java-exception org.mozilla.gecko.mozglue.GeckoLoader$AbortException: abort() called from :0xdd6216d9 () at org.mozilla.gecko.mozglue.GeckoLoader.abort(GeckoLoader.java:532) → [Mdm2] Intermittent test_peerConnection_basicVideoRemoteHwEncoder.html | java-exception org.mozilla.gecko.mozglue.GeckoLoader$AbortException: abort() called from :0xdd6216d9 () at org.mozilla.gecko.mozglue.GeckoLoader.abort(GeckoLoader.java:532)

Comment 6

4 months ago
17 failures in 173 pushes (0.098 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-central: 8
* mozilla-inbound: 6
* autoland: 3

Platform breakdown:
* android-6-0-armv8-api15: 8
* android-7-1-armv8-api15: 7
* android-5-1-armv7-api15: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-20&endday=2017-06-20&tree=all
Hi, Munro,
Could you help me take a look about it?
It seems related with WebRTC.
Thanks!
Flags: needinfo?(alwu) → needinfo?(mchiang)

Comment 8

4 months ago
26 failures in 170 pushes (0.153 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 16
* mozilla-inbound: 8
* mozilla-central: 2

Platform breakdown:
* android-6-0-armv8-api15: 14
* android-7-1-armv8-api15: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-21&endday=2017-06-21&tree=all
(Assignee)

Updated

4 months ago
Assignee: nobody → mchiang
Flags: needinfo?(mchiang)
(Assignee)

Updated

4 months ago
Keywords: regressionwindow-wanted
(Assignee)

Updated

4 months ago
Keywords: regressionwindow-wanted

Comment 9

4 months ago
15 failures in 175 pushes (0.086 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 6
* mozilla-central: 5
* mozilla-inbound: 4

Platform breakdown:
* android-6-0-armv8-api15: 8
* android-7-1-armv8-api15: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-22&endday=2017-06-22&tree=all
:mchiang- do you think this will be fixed before or during the all hands next week?  The failure rate is pretty high and I would like to temporarily disable this if we don't have a short term plan for a fix.
Flags: needinfo?(mchiang)
Duplicate of this bug: 1375634
Rank: 25 → 18
Priority: P2 → P1

Comment 12

4 months ago
74 failures in 892 pushes (0.083 failures/push) were associated with this bug in the last 7 days. 

This is the #22 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 32
* mozilla-inbound: 21
* mozilla-central: 21

Platform breakdown:
* android-6-0-armv8-api15: 36
* android-7-1-armv8-api15: 34
* android-5-1-armv7-api15: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-19&endday=2017-06-25&tree=all
(Assignee)

Comment 13

4 months ago
(In reply to Joel Maher ( :jmaher) from comment #10)
> :mchiang- do you think this will be fixed before or during the all hands
> next week?  The failure rate is pretty high and I would like to temporarily
> disable this if we don't have a short term plan for a fix.

Sorry to response late. I am heading to SF.
This bug is my first priority during the all hands.
Flags: needinfo?(mchiang)
(Assignee)

Comment 14

4 months ago
Regresion window show that this bug is caused by bug 1341285
(Assignee)

Updated

4 months ago
Blocks: 1341285
Dan - regression from the 57 landing on android - coordinate with munro

Comment 16

4 months ago
Munro, let me know if there is anything I can do to help!

Comment 17

4 months ago
Looks like an invalid rtp_time:

6-13 11:15:28.927 E/rtc     ( 8594): #
2017-06-13 08:16:51,506      nexus-6p-5 INFO     UnitTestJob mozilla-inbound 20170613140230 debug api-23 android-api-15 mochitest-dom-media-tests 06-13 11:15:28.927 E/rtc     ( 8594): # Fatal error in /home/worker/workspace/build/src/media/webrtc/trunk/webrtc/modules/rtp_rtcp/source/rtp_header_extensions.cc, line 106
2017-06-13 08:16:51,506      nexus-6p-5 INFO     UnitTestJob mozilla-inbound 20170613140230 debug api-23 android-api-15 mochitest-dom-media-tests 06-13 11:15:28.927 E/rtc     ( 8594): # last system error: 22
2017-06-13 08:16:51,507      nexus-6p-5 INFO     UnitTestJob mozilla-inbound 20170613140230 debug api-23 android-api-15 mochitest-dom-media-tests 06-13 11:15:28.927 E/rtc     ( 8594): # Check failed: rtp_time <= 0x00ffffff (752870786 vs. 16777215)
2017-06-13 08:16:51,507      nexus-6p-5 INFO     UnitTestJob mozilla-inbound 20170613140230 debug api-23 android-api-15 mochitest-dom-media-tests 06-13 11:15:28.927 E/rtc     ( 8594): #
2017-06-13 08:16:51,507      nexus-6p-5 INFO     UnitTestJob mozilla-inbound 20170613140230 debug api-23 android-api-15 mochitest-dom-media-tests 06-13 11:15:28.927 E/rtc     ( 8594): #
2017-06-13 08:16:51,507      nexus-6p-5 INFO     UnitTestJob mozilla-inbound 20170613140230 debug api-23 android-api-15 mochitest-dom-media-tests 06-13 11:15:28.927 E/Gecko   ( 8594): mozalloc_abort: abort() called from :0xdd6216d9 ()
(Assignee)

Comment 18

4 months ago
This check is added in 57 release.
Does rtp_time come from encoder?
See rtp_sender.cc:

    if (capture_time_ms > 0) {
      padding_packet->SetExtension<TransmissionOffset>(
          (now_ms - capture_time_ms) * kTimestampTicksPerMs);
    }

  int64_t capture_time_ms = packet->capture_time_ms();
...
  int64_t diff_ms = now_ms - capture_time_ms;
  packet_to_send->SetExtension<TransmissionOffset>(kTimestampTicksPerMs *
                                                   diff_ms);

  if (packet->capture_time_ms() > 0) {
    packet->SetExtension<TransmissionOffset>(
        kTimestampTicksPerMs * (now_ms - packet->capture_time_ms()));
  }

So since 'now' is unlikely to be bad, either the capture_time_ms is bad, or maybe it's 0 -- note that one of these doesn't check for 0

Comment 20

4 months ago
15 failures in 138 pushes (0.109 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 9
* autoland: 6

Platform breakdown:
* android-6-0-armv8-api15: 8
* android-7-1-armv8-api15: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-28&endday=2017-06-28&tree=all
Comment hidden (mozreview-request)
(Assignee)

Comment 22

4 months ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8104f818ed3c9ad0ff73155e853dc2bdb2766e0f
(Assignee)

Comment 23

4 months ago
This bug is similar to bug 1030112.
We should feed android native encoder with timestamp in millisecond (inputImage.render_time_ms()), not in 90kHz format (inputImage.timestamp()).
Attachment #8882092 - Flags: review?(jib) → review?(rjesup)
Lgtm, but lateraling to jesup.

Comment 25

4 months ago
mozreview-review
Comment on attachment 8882092 [details]
Bug 1372687 - replace timestamp() (90kHz) with inputImage.render_time_ms() (Milliseconds).

https://reviewboard.mozilla.org/r/153192/#review158440

from looking at the patch, it appears the changes is us vs ms, not 90K vs ms.   If this was the intention, r+ with a commit message change.  Else, please check and re-submit
Attachment #8882092 - Flags: review?(rjesup) → review+
(Assignee)

Comment 26

4 months ago
mozreview-review
Comment on attachment 8882092 [details]
Bug 1372687 - replace timestamp() (90kHz) with inputImage.render_time_ms() (Milliseconds).

https://reviewboard.mozilla.org/r/153192/#review158492

::: media/webrtc/signaling/src/media-conduit/WebrtcMediaCodecVP8VideoCodec.cpp:108
(Diff revision 1)
>  
>        int64_t presentationTimeUs;
>        ok = NS_SUCCEEDED(info->PresentationTimeUs(&presentationTimeUs));
>        MOZ_RELEASE_ASSERT(ok);
>  
> -      mEncodedImage._timeStamp = presentationTimeUs;
> +      mEncodedImage._timeStamp = presentationTimeUs / PR_USEC_PER_MSEC;

us -> ms

::: media/webrtc/signaling/src/media-conduit/WebrtcMediaCodecVP8VideoCodec.cpp:1079
(Diff revision 1)
> -    bufferInfo->Set(0, size, inputImage.timestamp(), MediaCodec::BUFFER_FLAG_SYNC_FRAME);
> +    bufferInfo->Set(0, size, inputImage.render_time_ms() * PR_USEC_PER_MSEC, MediaCodec::BUFFER_FLAG_SYNC_FRAME);
>    } else {
> -    bufferInfo->Set(0, size, inputImage.timestamp(), 0);
> +    bufferInfo->Set(0, size, inputImage.render_time_ms() * PR_USEC_PER_MSEC, 0);

timestamp() returns timestamp in 90kHz format
http://searchfox.org/mozilla-central/rev/152c0296f8a10f81185ee88dfb4114ec3882b4c6/media/webrtc/trunk/webrtc/api/video/video_frame.h#74-75

render_time_ms() returns timestamp in ms format
http://searchfox.org/mozilla-central/rev/152c0296f8a10f81185ee88dfb4114ec3882b4c6/media/webrtc/trunk/webrtc/api/video/video_frame.cc#62-64
(Assignee)

Updated

4 months ago
Keywords: checkin-needed

Comment 27

4 months ago
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/6ec07b1cf721
replace timestamp() (90kHz) with inputImage.render_time_ms() (Milliseconds). r=jesup
Keywords: checkin-needed

Comment 28

4 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/6ec07b1cf721
Status: NEW → RESOLVED
Last Resolved: 4 months ago
status-firefox56: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56

Comment 29

4 months ago
50 failures in 718 pushes (0.07 failures/push) were associated with this bug in the last 7 days. 

This is the #24 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 32
* autoland: 14
* mozilla-central: 4

Platform breakdown:
* android-6-0-armv8-api15: 26
* android-7-1-armv8-api15: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1372687&startday=2017-06-26&endday=2017-07-02&tree=all

Updated

4 months ago
Whiteboard: [stockwell needswork] → [stockwell fixed]
Whiteboard: [stockwell fixed] → [stockwell fixed:product]
status-firefox54: --- → unaffected
status-firefox55: --- → unaffected
status-firefox-esr52: --- → unaffected
Keywords: crash
You need to log in before you can comment on or make changes to this bug.