1375389 - Enable Mdm1 on Autophone

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Audio/Video, enhancement, P3)

Description

[Alastor Wu [:alwu]]

After landing bug 1373177, I'll check again whether current media tests can run successfully on Autophone and no any significant intermittent failure.

If everything is great, then we can re-enable the mdm1 on Autophone.

Comment 1

[Blake Wu [:bwu][:blakewu]]

Cool!

Comment 2

[Alastor Wu [:alwu]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ee4dcd36f89c1e8a8ee11907826100175c7aaa83

Comment 3

[Alastor Wu [:alwu]]

From the result in comment2, although now we won't get the permanent fail on mdm1, but there are still some high possibility intermittent failures.
We need to make them more stable before we re-enable mdm1.

Comment 4

[Alastor Wu [:alwu]]

Let's waiting for the result, I'm sure we don't have permanent failure now.
But I guess there are still some high possibility intermittent failures, I'll prefer to disable them and make the mdm1 back to Autophone again.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b64e5bc1d12e6590663b44d663295cc971805c8b

Comment 5

[Alastor Wu [:alwu]]

After fixing lots of bugs, we do reduce the number of fail test cases, and I think it's time to open mdm1 on Autophone.

Because (1) we really need to have test coverage for media codes on Android, (2) for those intermittent, we need to know their fail frequency to decide whether it's serious or not.

Here are the try results, 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a087af870d5daaa108f301dac89cb1c61e8ffd52
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d

---

Hi, Bob,
How do you think?
Thanks!

Comment 6

[Bob Clary [:bc] (inactive)]

I've retriggered Mdm1 on your try builds. Lets see how they do and ask Joel what he thinks about the orange factor.

Comment 7

[Bob Clary [:bc] (inactive)]

One thing I can say initially is that I can't afford 38 minutes like https://treeherder.mozilla.org/#/jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d&group_state=expanded&selectedJob=115843014

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

those seem to have a lot of failures still- I would like to wait for more data to know this is <10% of the time we fail.

Comment 9

[Alastor Wu [:alwu]]

(In reply to Joel Maher ( :jmaher) (UTC-8) from comment #8)
> those seem to have a lot of failures still- I would like to wait for more
> data to know this is <10% of the time we fail.

Ok, let's set that as a goal. Reopen it when the fail rate is under 10%.

(In reply to Bob Clary [:bc:] from comment #7)
> One thing I can say initially is that I can't afford 38 minutes like
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=e41b5f7ae88e27cd9c45bf782b3d666416cbbd3d&group_state=e
> xpanded&selectedJob=115843014

Now almost all failures on the Fennec is timeout, that's why it spends so much time.
The duration of pass Mdm1 only takes under 10 mins, maybe we can reduce the timeout time on Fennec...?

Anyway, I'll try to reduce the fail rate before asking another request.

Thanks!

Comment 10

[Bob Clary [:bc] (inactive)]

 Looking at the retriggers I did on your try runs, you are pretty close to making it under 10%.

Comment 11

[Alastor Wu [:alwu]]

(In reply to Bob Clary [:bc:] from comment #10)
>  Looking at the retriggers I did on your try runs, you are pretty close to
> making it under 10%.

I saw some tests have relatively high fail rate than others, let me try to disable them and see the result.

Comment 12

[Alastor Wu [:alwu]]

In this result [1], we found the crash in JellyBeanAsyncCodec$CallbackSender.notifyError, and it might be the root cause which cause the Android 4.2 & 5.0 failure.

After offline discussion with John, he will take a look that issue.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=6b94808793ba87b38b3a4505e28fd2665de15932&selectedJob=116280017

Comment 13

[Alastor Wu [:alwu]]

Now mdm1 would permanent fail due to the crash of HLS tests, we need to fix it in bug1387963.

Comment 14

[Alastor Wu [:alwu]]

The try result + disable HLS tests
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e38412ddc86c19d95f78c36f84508ea1557557da&selectedJob=121351810

Comment 15

[Alastor Wu [:alwu]]

The result in comment14 looks good, we can consider to enable mdm1 after fixing bug1387702.

Comment 16

[Alastor Wu [:alwu]]

Hi, BC,
This is the try result [1] from yesterday's build, and it seems we've achieved the fail rate under the 10%.
Could we re-enable mdm1 now? How do you think?
Thanks!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=75d08540553f3db769b93085e348ad1933eee0e7

Comment 17

[Bob Clary [:bc] (inactive)]

opt looks clean across the board. debug is definitely out for android 4.2 (nexus 4) and 7.1 (pixel). I'll look into the load and where we can run these. Thanks!

Comment 18

[Alastor Wu [:alwu]]

FYI, :jolin has a patch to fix the fail on 4.2, and I think he will file bug and fix it soon.

Comment 19

[John Lin [:jhlin][:jolin]]

(In reply to Alastor Wu [:alwu][please needinfo me][GMT+8] from comment #18)
> FYI, :jolin has a patch to fix the fail on 4.2, and I think he will file bug
> and fix it soon.

 Unfortunately my theory about 4.2 timeouts is wrong. Need more time to investigate the real cause.

Comment 20

[Blake Wu [:bwu][:blakewu]]

If possible, can we enable tests first without being blocked by 4.2? We could enable it on 4.2 later.

Comment 21

[Bob Clary [:bc] (inactive)]

Yes, my plan is to enable everything I can support minus debug 4.2 and debug 7.1 until we get a handle on those failures.

Comment 22

[Alastor Wu [:alwu]]

Hi, BC,
How was the status of enabling test? Is everything good?
Thanks!

Comment 23

[Bob Clary [:bc] (inactive)]

Attachment: bug-1375389-v1.patch

This will enable Mdm1 except on Nexus 4 (I don't have enough devices) and Pixel debug. r=self. I'll land this later today.

Comment 24

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/75d0f76f10626646ffb45799f56da8c3e2eea9cb
deployed 2017-08-21 09:30 PT. Marking this fixed. I'll follow up if there are any issues.

Comment 25

[Alastor Wu [:alwu]]

(In reply to Bob Clary [:bc:] from comment #24)
> https://github.com/mozilla/autophone/commit/
> 75d0f76f10626646ffb45799f56da8c3e2eea9cb
> deployed 2017-08-21 09:30 PT. Marking this fixed. I'll follow up if there
> are any issues.

Hooray! Super thank for your help :)

Comment 26

[Bob Clary [:bc] (inactive)]

I shouldn't have enabled Mdm1 on beta or release. I'll work up a patch later today.

Comment 27

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/d7e3e06f02aa5f8be12f7a3a648f8710f5b5bade

r=self.