Intermittent mediapipeline_unittest | test failed with return code 1 due to MediaPipelineTest.TestAudioSendMux failure

RESOLVED FIXED in Firefox -esr52

Status

()

Core
WebRTC: Signaling
P2
normal
Rank:
25
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: Treeherder Bug Filer, Assigned: dminor)

Tracking

({intermittent-failure})

unspecified
mozilla53
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox51 wontfix, firefox52 disabled, firefox-esr52 fixed, firefox53 fixed)

Details

Attachments

(3 attachments)

(Reporter)

Description

2 years ago
treeherder
Filed by: rvandermeulen [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=3734995&repo=mozilla-aurora

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-aurora-linux-pgo/1475547840/mozilla-aurora_ubuntu32_vm_test_pgo-cppunit-bm05-tests1-linux32-build2.txt.gz

[ RUN      ] MediaPipelineTest.TestAudioSendMux
/builds/slave/m-aurora-lx-000000000000000000/build/src/media/webrtc/signaling/test/mediapipeline_unittest.cpp:453: Failure
Value of: p2_.GetAudioRtpCountReceived()
  Actual: 233
Expected: p1_.GetAudioRtpCountSent()
Which is: 970
[  FAILED  ] MediaPipelineTest.TestAudioSendMux (11146 ms)
status-firefox51: --- → affected
Not sure this is a great check as written.
Rank: 25
Priority: -- → P2
status-firefox52: --- → affected

Comment 2

2 years ago
12 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 6
* mozilla-inbound: 5
* mozilla-central: 1

Platform breakdown:
* linux64: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-10-24&endday=2016-10-30&tree=all

Comment 3

2 years ago
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* autoland: 1

Platform breakdown:
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-11-07&endday=2016-11-13&tree=all

Comment 4

2 years ago
11 failures in 715 pushes (0.015 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 5
* autoland: 5
* mozilla-central: 1

Platform breakdown:
* linux64: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-11-14&endday=2016-11-20&tree=all

Comment 5

2 years ago
6 failures in 623 pushes (0.01 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 3
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-11-21&endday=2016-11-27&tree=all

Comment 6

2 years ago
19 failures in 694 pushes (0.027 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 9
* autoland: 9
* mozilla-central: 1

Platform breakdown:
* linux64: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-11-28&endday=2016-12-04&tree=all

Comment 7

2 years ago
7 failures in 289 pushes (0.024 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 3
* autoland: 3
* graphics: 1

Platform breakdown:
* linux64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-12-05&endday=2016-12-11&tree=all

Comment 8

2 years ago
6 failures in 526 pushes (0.011 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 5
* mozilla-aurora: 1

Platform breakdown:
* linux64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1307461&startday=2016-12-12&endday=2016-12-18&tree=all
We stopped building this test on trunk via bug 1322707. However, 52 continues to be affected (and is destined to be our next ESR).
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox51: affected → wontfix
status-firefox53: --- → fixed
Depends on: 1322707
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
If disabling the entire test isn't desirable, can we maybe skip just the one problematic subtest?
Flags: needinfo?(rjesup)
Dan, can you look at this?  thanks
Flags: needinfo?(rjesup) → needinfo?(dminor)
(Assignee)

Comment 12

2 years ago
Created attachment 8831749 [details] [diff] [review]
Disable TestAudioSendMux

Try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=44a5cb4d2f19833bbf70f0e044259afac8bb4858
Flags: needinfo?(dminor)
Attachment #8831749 - Flags: review?(rjesup)

Updated

2 years ago
Attachment #8831749 - Flags: review?(rjesup) → review+
(Assignee)

Comment 13

2 years ago
Comment on attachment 8831749 [details] [diff] [review]
Disable TestAudioSendMux

Approval Request Comment
[Feature/Bug causing the regression]: Not sure, we're working around an intermittent failure.
[User impact if declined]: None
[Is this code covered by automated tests?]: Yes.
[Has the fix been verified in Nightly?]: This test is completely disabled in aurora and nightly.
[Needs manual test from QE? If yes, steps to reproduce]: No.
[List of other uplifts needed for the feature/fix]: None.
[Is the change risky?]: No.
[Why is the change risky/not risky?]: We're disabling an intermittent test. There is a slight risk that we could introduce a regression that this test would have caught, but since this test is disabled on aurora and currently disabled on nightly while it is being rewrittent, I think we're ok with that risk.
[String changes made/needed]: None
Attachment #8831749 - Flags: approval-mozilla-beta?
Comment on attachment 8831749 [details] [diff] [review]
Disable TestAudioSendMux

test-only changes don't need approval, thanks Dan!
Attachment #8831749 - Flags: approval-mozilla-beta?
Whiteboard: [checkin-needed-beta]

Comment 15

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/95e076f66b6b
status-firefox52: affected → fixed
Whiteboard: [checkin-needed-beta]
Bah, it just moved to TestAudioSendNoMux. Guess that's not surprising given bug 1317044, in retrospect.
https://treeherder.mozilla.org/logviewer.html#?job_id=73368952&repo=mozilla-beta
Assignee: nobody → dminor
Flags: needinfo?(dminor)
(Assignee)

Comment 17

2 years ago
Created attachment 8832505 [details] [diff] [review]
whack-a-mole

If we see any more problems after this, I'll just disable the test completely.
Flags: needinfo?(dminor)
Attachment #8832505 - Flags: review?(rjesup)

Updated

2 years ago
Attachment #8832505 - Flags: review?(rjesup) → review+
(Assignee)

Updated

2 years ago
Status: RESOLVED → REOPENED
status-firefox52: fixed → affected
Resolution: FIXED → ---
Whiteboard: checkin-needed-beta
Bug resolution tracks status on trunk.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED

Comment 19

2 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/1ad9c3c00e7d
status-firefox52: affected → fixed
status-firefox52: fixed → disabled
Whiteboard: checkin-needed-beta
Blocks: 1317044
(Assignee)

Comment 21

2 years ago
Created attachment 8835499 [details] [diff] [review]
Disable mediapiple_unittest on beta

Just disable the test completely. Ryan said he would be willing to take the gtest version of this test once it is ready if it ends up being less prone to intermittent failures.
Flags: needinfo?(dminor)
Attachment #8835499 - Flags: review?(rjesup)

Updated

2 years ago
Attachment #8835499 - Flags: review?(rjesup) → review+
(Assignee)

Updated

a year ago
Whiteboard: checkin-needed-beta

Comment 22

a year ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/dd0c3648541f
Whiteboard: checkin-needed-beta

Comment 23

a year ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-esr52/rev/dd0c3648541f
status-firefox-esr52: --- → fixed
You need to log in before you can comment on or make changes to this bug.