Closed Bug 1375389 Opened 3 years ago Closed 3 years ago

Enable Mdm1 on Autophone

Categories

(Firefox for Android :: Audio/Video, enhancement, P3)

enhancement

Tracking

()

RESOLVED FIXED

People

(Reporter: alwu, Assigned: bc)

References

(Blocks 2 open bugs)

Details

Attachments

(1 file)

After landing bug 1373177, I'll check again whether current media tests can run successfully on Autophone and no any significant intermittent failure.

If everything is great, then we can re-enable the mdm1 on Autophone.
Cool!
Priority: -- → P3
From the result in comment2, although now we won't get the permanent fail on mdm1, but there are still some high possibility intermittent failures.
We need to make them more stable before we re-enable mdm1.
Summary: Enable mochitest-dom-media test → Enable mochitest-dom-media test on Autophone
Let's waiting for the result, I'm sure we don't have permanent failure now.
But I guess there are still some high possibility intermittent failures, I'll prefer to disable them and make the mdm1 back to Autophone again.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b64e5bc1d12e6590663b44d663295cc971805c8b
After fixing lots of bugs, we do reduce the number of fail test cases, and I think it's time to open mdm1 on Autophone.

Because (1) we really need to have test coverage for media codes on Android, (2) for those intermittent, we need to know their fail frequency to decide whether it's serious or not.

Here are the try results, 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a087af870d5daaa108f301dac89cb1c61e8ffd52
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d

---

Hi, Bob,
How do you think?
Thanks!
Flags: needinfo?(bob)
Summary: Enable mochitest-dom-media test on Autophone → Enable Mdm1 on Autophone
I've retriggered Mdm1 on your try builds. Lets see how they do and ask Joel what he thinks about the orange factor.
Flags: needinfo?(jmaher)
those seem to have a lot of failures still- I would like to wait for more data to know this is <10% of the time we fail.
Flags: needinfo?(jmaher)
(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #8)
> those seem to have a lot of failures still- I would like to wait for more
> data to know this is <10% of the time we fail.

Ok, let's set that as a goal. Reopen it when the fail rate is under 10%.

(In reply to Bob Clary [:bc:] from comment #7)
> One thing I can say initially is that I can't afford 38 minutes like
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d&group_state=e
> xpanded&selectedJob=115843014

Now almost all failures on the Fennec is timeout, that's why it spends so much time.
The duration of pass Mdm1 only takes under 10 mins, maybe we can reduce the timeout time on Fennec...?

Anyway, I'll try to reduce the fail rate before asking another request.

Thanks!
Flags: needinfo?(bob)
 Looking at the retriggers I did on your try runs, you are pretty close to making it under 10%.
(In reply to Bob Clary [:bc:] from comment #10)
>  Looking at the retriggers I did on your try runs, you are pretty close to
> making it under 10%.

I saw some tests have relatively high fail rate than others, let me try to disable them and see the result.
Depends on: 1373177
In this result [1], we found the crash in JellyBeanAsyncCodec$CallbackSender.notifyError, and it might be the root cause which cause the Android 4.2 & 5.0 failure.

After offline discussion with John, he will take a look that issue.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=6b94808793ba87b38b3a4505e28fd2665de15932&selectedJob=116280017
Depends on: 1384495
Depends on: 1387963
Now mdm1 would permanent fail due to the crash of HLS tests, we need to fix it in bug1387963.
Depends on: 1387702
The result in comment14 looks good, we can consider to enable mdm1 after fixing bug1387702.
Depends on: 1388633
Hi, BC,
This is the try result [1] from yesterday's build, and it seems we've achieved the fail rate under the 10%.
Could we re-enable mdm1 now? How do you think?
Thanks!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=75d08540553f3db769b93085e348ad1933eee0e7
Flags: needinfo?(bob)
opt looks clean across the board. debug is definitely out for android 4.2 (nexus 4) and 7.1 (pixel). I'll look into the load and where we can run these. Thanks!
FYI, :jolin has a patch to fix the fail on 4.2, and I think he will file bug and fix it soon.
(In reply to Alastor Wu [:alwu][please needinfo me][GMT+8] from comment #18)
> FYI, :jolin has a patch to fix the fail on 4.2, and I think he will file bug
> and fix it soon.

 Unfortunately my theory about 4.2 timeouts is wrong. Need more time to investigate the real cause.
If possible, can we enable tests first without being blocked by 4.2? We could enable it on 4.2 later.
Yes, my plan is to enable everything I can support minus debug 4.2 and debug 7.1 until we get a handle on those failures.
Hi, BC,
How was the status of enabling test? Is everything good?
Thanks!
This will enable Mdm1 except on Nexus 4 (I don't have enough devices) and Pixel debug. r=self. I'll land this later today.
Attachment #8899516 - Flags: review+
https://github.com/mozilla/autophone/commit/75d0f76f10626646ffb45799f56da8c3e2eea9cb
deployed 2017-08-21 09:30 PT. Marking this fixed. I'll follow up if there are any issues.
Status: NEW → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bob)
Resolution: --- → FIXED
(In reply to Bob Clary [:bc:] from comment #24)
> https://github.com/mozilla/autophone/commit/
> 75d0f76f10626646ffb45799f56da8c3e2eea9cb
> deployed 2017-08-21 09:30 PT. Marking this fixed. I'll follow up if there
> are any issues.

Hooray! Super thank for your help :)
Blocks: 1392747
I shouldn't have enabled Mdm1 on beta or release. I'll work up a patch later today.
Flags: needinfo?(bob)
You need to log in before you can comment on or make changes to this bug.