1112779 - Turn off Mulet mochitests on all trees except Try and Cedar

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Mochitest-2 and 4 were enabled in production on August 1. From the start, they were permafail and hidden immediately (see bug 1047571 and bug 1047572). In the following 4.5 months, literally nothing has happened in the bugs on file.

Similarly, reftests were enabled for Try only in early October and immediately hidden (see bug 1080123). Same story.

Instead of wasting resources indefinitely, we should just shut them off until someone's going to step up to actually get them fit to run in production.

Comment 1

[Armen [:armenzg]]

For the record, mochitests are also not running the right thing (bug 1072443).

Comment 2

[Ryan VanderMeulen [:RyanVM]]

I'm honestly considering just completely disabling them in that case.

Comment 3

[Jonathan Griffin (:jgriffin)]

It's not the case that they provide no value.  Rather, they're running in the parent process, and we'd prefer to have them running in a child process, like we do with b2gdesktop.  That said, it's very unclear what priority Mulet has right now; there's not been enough developer engagement to actually replace b2gdesktop, which was the original goal.

Comment 4

[Alexandre Poirot [:ochameau]]

We still have some goals to replace b2g desktop with mulet during Q1.
We have been focusing on making gaia tests to work against it.
Until we have all tests suites up and running on CI, we can't officialy replace b2g desktop.
And until we replace it we won't get a lot of traction, so it is mostly me and gerard-majax trying to address whatever if broken.
The good news is that I heard from James that most tests and especially gaia one are already running on taskcluster!

Having said that, I'm not against disabling them as soon as it doesn't prevent us reenabling them once we are ready to switch to mulet (and fixed the failures)! Will it also prevent us from running them on Try?

Comment 5

[Alexandre Poirot [:ochameau]]

Also, reftest is different than mochitests, gerard-majax has been working on greening them a lot recently.
Hopefully that's close to be green now.

Comment 6

[Alexandre Poirot [:ochameau]]

Reftest are only enable on Try and Cedar, right?

Comment 7

[Armen [:armenzg]]

Attachment: Disable mulet mochitest 2 & 4 except on Try/Cedar

Comment 8

[Armen [:armenzg]]

Attachment: differences.txt

Builders differing after this patch.

We only run reftests on Try and Cedar.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8538766 [details]
differences.txt

Per the IRC discussion w/ ochameau, we want to turn off *all* mochitests, not just 2 & 4.

Comment 10

[Armen [:armenzg]]

Attachment: [configs] Disable mulet mochitests except on Try/Cedar

Comment 11

[Armen [:armenzg]]

Attachment: differences.txt

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8538806 [details]
differences.txt

Looks the same as before.

Comment 13

[Armen [:armenzg]]

Comment on attachment 8538805 [details] [diff] [review]
[configs] Disable mulet mochitests except on Try/Cedar

My bad about attaching the wrong differences file. I will attach it again once I'm back to my Linux machine.

Comment 14

[Armen [:armenzg]]

Attachment: differences.txt

Comment 15

[Armen [:armenzg]]

I believe this is live for the last 8 hours.
http://hg.mozilla.org/build/buildbot-configs/graph

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8540312 [details]
differences.txt

Love it.