Status

()

enhancement
P3
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: alwu, Assigned: bc)

Tracking

(Blocks 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Reporter

Description

2 years ago
After landing bug 1373177, I'll check again whether current media tests can run successfully on Autophone and no any significant intermittent failure.

If everything is great, then we can re-enable the mdm1 on Autophone.
Cool!
Priority: -- → P3
Reporter

Comment 3

2 years ago
From the result in comment2, although now we won't get the permanent fail on mdm1, but there are still some high possibility intermittent failures.
We need to make them more stable before we re-enable mdm1.
Reporter

Updated

2 years ago
Summary: Enable mochitest-dom-media test → Enable mochitest-dom-media test on Autophone
Reporter

Comment 4

2 years ago
Let's waiting for the result, I'm sure we don't have permanent failure now.
But I guess there are still some high possibility intermittent failures, I'll prefer to disable them and make the mdm1 back to Autophone again.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b64e5bc1d12e6590663b44d663295cc971805c8b
Reporter

Comment 5

2 years ago
After fixing lots of bugs, we do reduce the number of fail test cases, and I think it's time to open mdm1 on Autophone.

Because (1) we really need to have test coverage for media codes on Android, (2) for those intermittent, we need to know their fail frequency to decide whether it's serious or not.

Here are the try results, 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a087af870d5daaa108f301dac89cb1c61e8ffd52
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d

---

Hi, Bob,
How do you think?
Thanks!
Flags: needinfo?(bob)
Reporter

Updated

2 years ago
Summary: Enable mochitest-dom-media test on Autophone → Enable Mdm1 on Autophone
Assignee

Comment 6

2 years ago
I've retriggered Mdm1 on your try builds. Lets see how they do and ask Joel what he thinks about the orange factor.
Flags: needinfo?(jmaher)
those seem to have a lot of failures still- I would like to wait for more data to know this is <10% of the time we fail.
Flags: needinfo?(jmaher)
Reporter

Comment 9

2 years ago
(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #8)
> those seem to have a lot of failures still- I would like to wait for more
> data to know this is <10% of the time we fail.

Ok, let's set that as a goal. Reopen it when the fail rate is under 10%.

(In reply to Bob Clary [:bc:] from comment #7)
> One thing I can say initially is that I can't afford 38 minutes like
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d&group_state=e
> xpanded&selectedJob=115843014

Now almost all failures on the Fennec is timeout, that's why it spends so much time.
The duration of pass Mdm1 only takes under 10 mins, maybe we can reduce the timeout time on Fennec...?

Anyway, I'll try to reduce the fail rate before asking another request.

Thanks!
Flags: needinfo?(bob)
Assignee

Comment 10

2 years ago
 Looking at the retriggers I did on your try runs, you are pretty close to making it under 10%.
Reporter

Comment 11

2 years ago
(In reply to Bob Clary [:bc:] from comment #10)
>  Looking at the retriggers I did on your try runs, you are pretty close to
> making it under 10%.

I saw some tests have relatively high fail rate than others, let me try to disable them and see the result.
Reporter

Updated

2 years ago
Reporter

Updated

2 years ago
Depends on: 1373177
Reporter

Comment 12

2 years ago
In this result [1], we found the crash in JellyBeanAsyncCodec$CallbackSender.notifyError, and it might be the root cause which cause the Android 4.2 & 5.0 failure.

After offline discussion with John, he will take a look that issue.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=6b94808793ba87b38b3a4505e28fd2665de15932&selectedJob=116280017
Reporter

Updated

2 years ago
Depends on: 1384495
Reporter

Updated

2 years ago
Depends on: 1387963
Reporter

Comment 13

2 years ago
Now mdm1 would permanent fail due to the crash of HLS tests, we need to fix it in bug1387963.
Reporter

Updated

2 years ago
Depends on: 1387702
Reporter

Comment 15

2 years ago
The result in comment14 looks good, we can consider to enable mdm1 after fixing bug1387702.
Reporter

Updated

2 years ago
Depends on: 1388633
Reporter

Comment 16

2 years ago
Hi, BC,
This is the try result [1] from yesterday's build, and it seems we've achieved the fail rate under the 10%.
Could we re-enable mdm1 now? How do you think?
Thanks!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=75d08540553f3db769b93085e348ad1933eee0e7
Flags: needinfo?(bob)
Assignee

Comment 17

2 years ago
opt looks clean across the board. debug is definitely out for android 4.2 (nexus 4) and 7.1 (pixel). I'll look into the load and where we can run these. Thanks!
Reporter

Comment 18

2 years ago
FYI, :jolin has a patch to fix the fail on 4.2, and I think he will file bug and fix it soon.
(In reply to Alastor Wu [:alwu][please needinfo me][GMT+8] from comment #18)
> FYI, :jolin has a patch to fix the fail on 4.2, and I think he will file bug
> and fix it soon.

 Unfortunately my theory about 4.2 timeouts is wrong. Need more time to investigate the real cause.
If possible, can we enable tests first without being blocked by 4.2? We could enable it on 4.2 later.
Assignee

Comment 21

2 years ago
Yes, my plan is to enable everything I can support minus debug 4.2 and debug 7.1 until we get a handle on those failures.
Reporter

Comment 22

2 years ago
Hi, BC,
How was the status of enabling test? Is everything good?
Thanks!
Assignee

Comment 23

2 years ago
This will enable Mdm1 except on Nexus 4 (I don't have enough devices) and Pixel debug. r=self. I'll land this later today.
Attachment #8899516 - Flags: review+
Assignee

Comment 24

2 years ago
https://github.com/mozilla/autophone/commit/75d0f76f10626646ffb45799f56da8c3e2eea9cb
deployed 2017-08-21 09:30 PT. Marking this fixed. I'll follow up if there are any issues.
Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bob)
Resolution: --- → FIXED
Reporter

Comment 25

2 years ago
(In reply to Bob Clary [:bc:] from comment #24)
> https://github.com/mozilla/autophone/commit/
> 75d0f76f10626646ffb45799f56da8c3e2eea9cb
> deployed 2017-08-21 09:30 PT. Marking this fixed. I'll follow up if there
> are any issues.

Hooray! Super thank for your help :)
Blocks: 1392747
Assignee

Comment 26

2 years ago
I shouldn't have enabled Mdm1 on beta or release. I'll work up a patch later today.
Flags: needinfo?(bob)
You need to log in before you can comment on or make changes to this bug.