1150579 - Turn on Mulet reftests on TaskCluster prod

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Runtime, defect)

Comment 1

[:gerard-majax]

Attachment: Switch Mulet reftests ON

Comment 2

[James Lal [:lightsofapollo]]

Comment on attachment 8587493 [details] [diff] [review]
Switch Mulet reftests ON

formalizing the r+ I already gave on you irc..

Comment 3

[:gerard-majax]

This try has a lot of retriggers to check how usable it is: https://treeherder.allizom.org/#/jobs?repo=try&revision=7b52379388d1

Comment 4

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #3)
> This try has a lot of retriggers to check how usable it is:
> https://treeherder.allizom.org/#/jobs?repo=try&revision=7b52379388d1

Reported failures on R1, R3, R5 and R6 are nearly all instances of bug 1140394.

There is one real intermittent on R1, but the rate is low enough I think.

Comment 5

[:gerard-majax]

Ryan, I'll let you judge whether the impact of bug 1140394 described above will make your daily job bad enough or not. If you think it's bad enough, I'll investigate and fix this before we turn on those tests.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

I think the biggest issue with bug 1140394 is that it occurs in a basically unstarrable way. I'd be more inclined to tolerate it if we at least had a way to make the failures show up as suggestable on Treeherder.

Comment 7

[:gerard-majax]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #6)
> I think the biggest issue with bug 1140394 is that it occurs in a basically
> unstarrable way. I'd be more inclined to tolerate it if we at least had a
> way to make the failures show up as suggestable on Treeherder.

Ok, this is something I missed but I understand easily :). According to Ted in bug 1140394 comment 8, we are being blocked by resolution of bug 798300.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Ugh :(

Comment 9

[:gerard-majax]

Ryan, after re-enabling many of the disabled reftests and finding a workaround for bug 1140394, I get this: https://treeherder.allizom.org/#/jobs?repo=try&revision=8d65013d65a1

It seems pretty stable green for me, and the two only oranges are bug 1153584. Those may have the same unstarrable issue as you feared about, but it seems to be only tied to high load. Plus, in bug 1153584 I'm asking if we could convert those 5xx errors into "blue" infra failures.

My current plan is:
 - land reftests changes from bug 1153574
 - land mozharness stdout workaround for bug 1140394

And then we could turn on the switch.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

sgtm, thanks for tracking this all down :)

Comment 11

[:gerard-majax]

So the deps got fixed. Let's do a last round of retriggers.

Comment 12

[:gerard-majax]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=106f30d1e4fd
https://treeherder.allizom.org/#/jobs?repo=try&revision=106f30d1e4fd

Comment 13

[:gerard-majax]

Attachment: Switch Mulet reftests ON

Rebasing on top of current master

Comment 14

[:gerard-majax]

Attachment: Switch Mulet reftests ON

Comment 15

[:gerard-majax]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e7662ad3aaac
https://treeherder.allizom.org/#/jobs?repo=try&revision=e7662ad3aaac

Comment 16

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #12)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=106f30d1e4fd
> https://treeherder.allizom.org/#/jobs?repo=try&revision=106f30d1e4fd

This try was missing the fix for bug 1140394: https://hg.mozilla.org/try/file/c3ab479717c2/layout/tools/reftest/b2g_desktop.py#l163

Comment 17

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #15)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=e7662ad3aaac
> https://treeherder.allizom.org/#/jobs?repo=try&revision=e7662ad3aaac

This try does include the fix for bug 1130494: https://hg.mozilla.org/try/file/54a151be2924/layout/tools/reftest/b2g_desktop.py#l163

Comment 18

[:gerard-majax]

One must read bug 1140494 and not bug 1130494 in the above comment 17.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/b2g-inbound/rev/06783f130730

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/06783f130730