Closed Bug 1144080 Opened 9 years ago Closed 9 years ago

Mulet reftests and mochitests temp disable for getting TaskCluster into production

Categories

(Firefox OS Graveyard :: Runtime, defect)

Product:
Firefox OS Graveyard
Component:
Runtime
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox39 fixed)

Status:
RESOLVED FIXED
Milestone:
2.2 S8 (20mar)
Tracking Flags:
Tracking Status
firefox39 --- fixed

People

(Reporter: gerard-majax, Assigned: gerard-majax)

References

Details

(Whiteboard: [systemsfe])

Attachments

(3 files, 6 obsolete files)

Script for disabling reftests
568 bytes, application/x-sh
Details
Disable reftests on Mulet for TaskCluster r=ahal
248.13 KB, patch
gerard-majax
: review+
Details | Diff | Splinter Review
Disable mochitests on Mulet for TaskCluster r=ahal
9.11 KB, patch
gerard-majax
: review+
Details | Diff | Splinter Review 
I'm not a big fan of this, but since we have try not running mulet mochitests and reftests anymore, we rely solely on taskcluter getting into production. There are still tons of mochitests and reftests failures that seems to be taskcluster-specifics. Let's disable those for now, get taskcluster in production, and re-enable after infrastructure has been properly fixed.

Follow-up to re-enable is bug 1144079.
Convention: tests that were already disabled for investigation on try infrastructure will be commented with "MULET: Bug 1144079 - Re-enable Mulet mochitests and reftests taskcluster-specific disables.". Tests disabled specifically for taskcluster will be commented: "TC: Bug 1144079 - Re-enable Mulet mochitests and reftests taskcluster-specific disables.".
Attached file Script for disabling reftests (obsolete) — 

    
  

        Attachment #8578640 -
        Attachment is obsolete: true

    
    

    
  

      
      
      
      

        Attached file
          
        Script for disabling reftests (obsolete)
        — 
      

    


  

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #6)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d30af3c7068c
> https://treeherder.allizom.org/#/jobs?repo=try&revision=d30af3c7068c
> 
> This list is starting to get quite huge, makes me sad.

I even see failures for things that are skipped !
Flags: needinfo?(jlal)

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #7)
> (In reply to Alexandre LISSY :gerard-majax from comment #6)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=d30af3c7068c
> > https://treeherder.allizom.org/#/jobs?repo=try&revision=d30af3c7068c
> > 
> > This list is starting to get quite huge, makes me sad.
> 
> I even see failures for things that are skipped !

For instance, tc-R2 => https://hg.mozilla.org/try/rev/4394e170dc33#l6.18

    
    

    
  
>  I even see failures for things that are skipped !

I don't have much context into why the reftest skip syntax is not working (and/or mulet is not getting recognized) did you did into this at all?
Flags: needinfo?(jlal)

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #11)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b62a61d1d475
> https://treeherder.allizom.org/#/jobs?repo=try&revision=b62a61d1d475

Still some skipped tests that are returning UNEXPECTED-FAIL. :(.

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #13)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=03c0a9335636
> https://treeherder.allizom.org/#/jobs?repo=try&revision=03c0a9335636

All R1, R2 and R3 failures reported are on tests that should have been skipped on Mulet: they all have 'skip-if(Mulet)'.

According to the logs, the sandbox does have "Mulet" to true.

Changing skip-if(Mulet) to put it at the end of the line on all cases of R1, R2 and R3 that were failing in the previous attempt:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a72542465865
https://treeherder.allizom.org/#/jobs?repo=try&revision=a72542465865

    
    

    
  


  

        Attachment #8578646 -
        Attachment is obsolete: true

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #14)
> (In reply to Alexandre LISSY :gerard-majax from comment #13)
> > https://treeherder.mozilla.org/#/jobs?repo=try&revision=03c0a9335636
> > https://treeherder.allizom.org/#/jobs?repo=try&revision=03c0a9335636
> 
> All R1, R2 and R3 failures reported are on tests that should have been
> skipped on Mulet: they all have 'skip-if(Mulet)'.
> 
> According to the logs, the sandbox does have "Mulet" to true.
> 
> Changing skip-if(Mulet) to put it at the end of the line on all cases of R1,
> R2 and R3 that were failing in the previous attempt:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=a72542465865
> https://treeherder.allizom.org/#/jobs?repo=try&revision=a72542465865

Ok, R2 and R3 are green now. Still something to do with R1, though.

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #15)
> Should improve tc-R6:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fa77e43da64
> https://treeherder.allizom.org/#/jobs?repo=try&revision=1fa77e43da64

New stuff to deal with on tc-R6.

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #20)
> More skip-if(Mulet) on tc-R5, and addressing previous tc-R1 and tc-R6:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=74f718dfb201
> https://treeherder.allizom.org/#/jobs?repo=try&revision=74f718dfb201

tc-R1, tc-R2, tc-R3, tc-R4, and tc-R6 are green. A couple more failures in tc-R5 but the list is shrinking.

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #23)
> More less tc-R5:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=4f526f19ea0d
> https://treeherder.allizom.org/#/jobs?repo=try&revision=4f526f19ea0d

Now just one tc-R5 failure.

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #29)
> Should deal with all the hg failures:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab5f114735db
> https://treeherder.allizom.org/#/jobs?repo=try&revision=ab5f114735db

and finally, tc-R5 green.

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #31)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d68bbbce3df2
> https://treeherder.allziom.org/#/jobs?repo=try&revision=d68bbbce3df2

https://treeherder.allizom.org/#/jobs?repo=try&revision=d68bbbce3df2

and that's a green for:
 tc-M1, tc-M2, tc-M3, tc-M4, tc-M5, tc-M6
 tc-R1, tc-R2, tc-R3, tc-R4, tc-R5, tc-R6

    
    

    
  


  

    
  

        Attachment #8579931 -
        Flags: review?(ahalberstadt)

    
    

    
  
Comment on attachment 8579932 [details] [diff] [review]
Disable reftests on Mulet for TaskCluster r=...

This one is very big, but it's because of very slight differences of rendering between try and taskcluster. The goal is to land as is, because we do not have anymore mulet on try so we need it on taskcluster asap. :lightsofapollo is looking into the differences to be able to re-enable those soon.

        Attachment #8579932 -
        Flags: review?(ahalberstadt)

    
  

        Attachment #8579931 -
        Attachment is obsolete: true

        Attachment #8579931 -
        Flags: review?(ahalberstadt)

    
  

        Attachment #8579932 -
        Attachment is obsolete: true

        Attachment #8579932 -
        Flags: review?(ahalberstadt)

    
    

    
  


  

    
  

        Attachment #8580023 -
        Flags: review?(ahalberstadt)

    
  

        Attachment #8580024 -
        Flags: review?(ahalberstadt)

    
    

    
  
(In reply to Alexandre LISSY :gerard-majax from comment #38)
> Should improve the few intermittents on Mochitests and Reftests:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=3127e5230fe6
> https://treeherder.allizom.org/#/jobs?repo=try&revision=3127e5230fe6

Quite some green :).

        Attachment #8580023 -
        Flags: review?(ahalberstadt) → review+

        Attachment #8580024 -
        Flags: review?(ahalberstadt) → review+

    
  

        Attachment #8580023 -
        Attachment is obsolete: true

    
  

        Attachment #8580024 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 8580556 [details] [diff] [review]
Disable reftests on Mulet for TaskCluster r=ahal

Carrying r+, rebased

        Attachment #8580556 -
        Flags: review+

    
    

    
  
Comment on attachment 8580557 [details] [diff] [review]
Disable mochitests on Mulet for TaskCluster r=ahal

Carrying r+, rebased

        Attachment #8580557 -
        Flags: review+

    
  
Keywords: checkin-needed
Whiteboard: [systemsfe]

    
  
Blocks: 1153574

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: