Closed Bug 1144080 Opened 5 years ago Closed 5 years ago

Mulet reftests and mochitests temp disable for getting TaskCluster into production

Categories

(Firefox OS Graveyard :: Runtime, defect)

x86_64
Linux
defect
Not set

Tracking

(firefox39 fixed)

RESOLVED FIXED
2.2 S8 (20mar)
Tracking Status
firefox39 --- fixed

People

(Reporter: gerard-majax, Assigned: gerard-majax)

References

Details

(Whiteboard: [systemsfe])

Attachments

(3 files, 6 obsolete files)

I'm not a big fan of this, but since we have try not running mulet mochitests and reftests anymore, we rely solely on taskcluter getting into production. There are still tons of mochitests and reftests failures that seems to be taskcluster-specifics. Let's disable those for now, get taskcluster in production, and re-enable after infrastructure has been properly fixed.

Follow-up to re-enable is bug 1144079.
Convention: tests that were already disabled for investigation on try infrastructure will be commented with "MULET: Bug 1144079 - Re-enable Mulet mochitests and reftests taskcluster-specific disables.". Tests disabled specifically for taskcluster will be commented: "TC: Bug 1144079 - Re-enable Mulet mochitests and reftests taskcluster-specific disables.".
Attached file Script for disabling reftests (obsolete) —
Attachment #8578640 - Attachment is obsolete: true
Attached file Script for disabling reftests (obsolete) —
(In reply to Alexandre LISSY :gerard-majax from comment #6)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d30af3c7068c
> https://treeherder.allizom.org/#/jobs?repo=try&revision=d30af3c7068c
> 
> This list is starting to get quite huge, makes me sad.

I even see failures for things that are skipped !
Flags: needinfo?(jlal)
(In reply to Alexandre LISSY :gerard-majax from comment #7)
> (In reply to Alexandre LISSY :gerard-majax from comment #6)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=d30af3c7068c
> > https://treeherder.allizom.org/#/jobs?repo=try&revision=d30af3c7068c
> > 
> > This list is starting to get quite huge, makes me sad.
> 
> I even see failures for things that are skipped !

For instance, tc-R2 => https://hg.mozilla.org/try/rev/4394e170dc33#l6.18
>  I even see failures for things that are skipped !

I don't have much context into why the reftest skip syntax is not working (and/or mulet is not getting recognized) did you did into this at all?
Flags: needinfo?(jlal)
(In reply to Alexandre LISSY :gerard-majax from comment #11)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b62a61d1d475
> https://treeherder.allizom.org/#/jobs?repo=try&revision=b62a61d1d475

Still some skipped tests that are returning UNEXPECTED-FAIL. :(.
(In reply to Alexandre LISSY :gerard-majax from comment #13)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=03c0a9335636
> https://treeherder.allizom.org/#/jobs?repo=try&revision=03c0a9335636

All R1, R2 and R3 failures reported are on tests that should have been skipped on Mulet: they all have 'skip-if(Mulet)'.

According to the logs, the sandbox does have "Mulet" to true.

Changing skip-if(Mulet) to put it at the end of the line on all cases of R1, R2 and R3 that were failing in the previous attempt:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a72542465865
https://treeherder.allizom.org/#/jobs?repo=try&revision=a72542465865
Attachment #8578646 - Attachment is obsolete: true
(In reply to Alexandre LISSY :gerard-majax from comment #14)
> (In reply to Alexandre LISSY :gerard-majax from comment #13)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=03c0a9335636
> > https://treeherder.allizom.org/#/jobs?repo=try&revision=03c0a9335636
> 
> All R1, R2 and R3 failures reported are on tests that should have been
> skipped on Mulet: they all have 'skip-if(Mulet)'.
> 
> According to the logs, the sandbox does have "Mulet" to true.
> 
> Changing skip-if(Mulet) to put it at the end of the line on all cases of R1,
> R2 and R3 that were failing in the previous attempt:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=a72542465865
> https://treeherder.allizom.org/#/jobs?repo=try&revision=a72542465865

Ok, R2 and R3 are green now. Still something to do with R1, though.
(In reply to Alexandre LISSY :gerard-majax from comment #15)
> Should improve tc-R6:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fa77e43da64
> https://treeherder.allizom.org/#/jobs?repo=try&revision=1fa77e43da64

New stuff to deal with on tc-R6.
(In reply to Alexandre LISSY :gerard-majax from comment #20)
> More skip-if(Mulet) on tc-R5, and addressing previous tc-R1 and tc-R6:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=74f718dfb201
> https://treeherder.allizom.org/#/jobs?repo=try&revision=74f718dfb201

tc-R1, tc-R2, tc-R3, tc-R4, and tc-R6 are green. A couple more failures in tc-R5 but the list is shrinking.
(In reply to Alexandre LISSY :gerard-majax from comment #23)
> More less tc-R5:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=4f526f19ea0d
> https://treeherder.allizom.org/#/jobs?repo=try&revision=4f526f19ea0d

Now just one tc-R5 failure.
(In reply to Alexandre LISSY :gerard-majax from comment #29)
> Should deal with all the hg failures:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab5f114735db
> https://treeherder.allizom.org/#/jobs?repo=try&revision=ab5f114735db

and finally, tc-R5 green.
(In reply to Alexandre LISSY :gerard-majax from comment #31)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d68bbbce3df2
> https://treeherder.allziom.org/#/jobs?repo=try&revision=d68bbbce3df2

https://treeherder.allizom.org/#/jobs?repo=try&revision=d68bbbce3df2

and that's a green for:
 tc-M1, tc-M2, tc-M3, tc-M4, tc-M5, tc-M6
 tc-R1, tc-R2, tc-R3, tc-R4, tc-R5, tc-R6
Attachment #8579931 - Flags: review?(ahalberstadt)
Comment on attachment 8579932 [details] [diff] [review]
Disable reftests on Mulet for TaskCluster r=...

This one is very big, but it's because of very slight differences of rendering between try and taskcluster. The goal is to land as is, because we do not have anymore mulet on try so we need it on taskcluster asap. :lightsofapollo is looking into the differences to be able to re-enable those soon.
Attachment #8579932 - Flags: review?(ahalberstadt)
Attachment #8579931 - Attachment is obsolete: true
Attachment #8579931 - Flags: review?(ahalberstadt)
Attachment #8579932 - Attachment is obsolete: true
Attachment #8579932 - Flags: review?(ahalberstadt)
Attachment #8580023 - Flags: review?(ahalberstadt)
Attachment #8580024 - Flags: review?(ahalberstadt)
(In reply to Alexandre LISSY :gerard-majax from comment #38)
> Should improve the few intermittents on Mochitests and Reftests:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=3127e5230fe6
> https://treeherder.allizom.org/#/jobs?repo=try&revision=3127e5230fe6

Quite some green :).
Attachment #8580023 - Flags: review?(ahalberstadt) → review+
Attachment #8580024 - Flags: review?(ahalberstadt) → review+
Attachment #8580023 - Attachment is obsolete: true
Attachment #8580024 - Attachment is obsolete: true
Comment on attachment 8580556 [details] [diff] [review]
Disable reftests on Mulet for TaskCluster r=ahal

Carrying r+, rebased
Attachment #8580556 - Flags: review+
Comment on attachment 8580557 [details] [diff] [review]
Disable mochitests on Mulet for TaskCluster r=ahal

Carrying r+, rebased
Attachment #8580557 - Flags: review+
Keywords: checkin-needed
Whiteboard: [systemsfe]
Blocks: 1153574
You need to log in before you can comment on or make changes to this bug.