Closed Bug 1228918 Opened 9 years ago Closed 8 years ago

Enable reftest on emu-x86-kk, Treeherder Production

Categories

(Firefox Build System :: Task Configuration, task, P1)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
mozilla47

People

(Reporter: bmo, Assigned: bmo)

References

Details

(Whiteboard: [EMU] [CI])

Attachments

(2 files, 2 obsolete files)

MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production
58 bytes, text/x-review-board-request
garndt
: review+
Details
MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test.
58 bytes, text/x-review-board-request
garndt
: review+
Details 
Working bug to enable Reftest on emulator-x86-kk.
Initial patch to enable reftest on emu-x86-kk.
Assignee: nobody → aschen
Depends on: 1162375
Blocks: emu-x86-kk-tests
Whiteboard: [EMU] [CI]

    
    

    
  
Comment on attachment 8705578 [details]
MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30081/diff/1-2/

        Attachment #8705578 -
        Attachment description: MozReview Request: Bug 1228918 - Mesa3D fallback to use softpipe renderer. → MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test.

    
    

    
  
grandt, can you help to review this patch ?

The 1st patch is to enable reftest on B2G-emulator-arm/x86-kk opt build, we used to enable it but running into tons of oranges(Bug1162375). Spent some time to figure out the cause and found that it's related mesa 3D library(8.0.4) installed in docker tester image "taskcluster/tester:0.4.5". The 2nd patch tried to switch mesa driver from llvmpipe to softpipe, which can get most of TC passed[1].
For those oranges(6 remaining), will be followed up via bug 1162375 and target to fix in next run.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=eba2ecf0ece7

    
  

        Attachment #8705577 -
        Flags: review?(garndt)

    
  

        Attachment #8705578 -
        Flags: review?(garndt)

    
  
Status: NEW → ASSIGNED

    
    

    
  
Comment on attachment 8705577 [details]
MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production

https://reviewboard.mozilla.org/r/30079/#review27275

::: testing/taskcluster/tasks/branches/base_jobs.yml:263
(Diff revision 1)
> +      tasks/builds/b2g_emulator_kk_opt.yml:

So it looks like there are a few tests that are orange right now but will be showing up as tier 1 on production treeherder.  Could we get some feedback from a sheriff on how to proceed?  From my understanding this could cause tree closure.  Perhaps the sheriffs are ok with starring them under some bug, but I would like to be safe first.

        Attachment #8705577 -
        Flags: review?(garndt)

    
    

    
  
Comment on attachment 8705578 [details]
MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test.

https://reviewboard.mozilla.org/r/30081/#review27277

        Attachment #8705578 -
        Flags: review?(garndt) → review+

    
    

    
  
(In reply to Greg Arndt [:garndt] from comment #32)
> So it looks like there are a few tests that are orange right now but will be
> showing up as tier 1 on production treeherder.  Could we get some feedback
> from a sheriff on how to proceed?  From my understanding this could cause
> tree closure.  Perhaps the sheriffs are ok with starring them under some
> bug, but I would like to be safe first.

Thanks, garndt. I'll have sheriff's feedback to ensure those oranges would be hidden on staging/production treeherder.

    
    

    
  
Hi Astley,

its no problem to hide the tests, we would need to know which one (like all or like R5, R19 etc).

cc'ing Wes since he also does sheriffing

    
    

    
  
(In reply to Carsten Book [:Tomcat] from comment #35)
> Hi Astley,
> 
> its no problem to hide the tests, we would need to know which one (like all
> or like R5, R19 etc).
> 
> cc'ing Wes since he also does sheriffing
Tomcat, thanks for help.
Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40)
Please also let me know how to proceed afterward.

    
    

    
  
(In reply to Astley Chen [:astley] from comment #36)
> Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40)
> Please also let me know how to proceed afterward.

Hey Astley,

i created now a filter that hides this tests and also filed a bug to get them unhidden when the tests are fixed (bug 1239951)

    
    

    
  
https://reviewboard.mozilla.org/r/30079/#review27275

> So it looks like there are a few tests that are orange right now but will be showing up as tier 1 on production treeherder.  Could we get some feedback from a sheriff on how to proceed?  From my understanding this could cause tree closure.  Perhaps the sheriffs are ok with starring them under some bug, but I would like to be safe first.

By comment#37, sheriff had created a filter to hide known oranges. I'd trigger a new try run and see if everything is in good shape prior to patch landed. Thanks.

    
  
Priority: -- → P1

    
    

    
  
(In reply to Carsten Book [:Tomcat] from comment #37)
> (In reply to Astley Chen [:astley] from comment #36)
> > Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40)
> > Please also let me know how to proceed afterward.
> 
> Hey Astley,
> 
> i created now a filter that hides this tests and also filed a bug to get
> them unhidden when the tests are fixed (bug 1239951)

Hi tomcat, from the several try results, I found that oranges appeared to be intermittent within below chunks: 5,8,9,13,14,21,22,23,26,29,31,32,33,34,38,39,40.
To ensure production tests wont be impacted by intermittent oranges, I will suggest to hide ALL reftest chunks on production and leave those oranges open on staging for try.
Can you help to do that ? Thanks.

Hi garndt, since now we have sheriffs assist to hide the chunks, can you help to review again ? Thanks.

    
  

        Attachment #8705577 -
        Flags: review?(garndt)

    
    

    
  
Comment on attachment 8705577 [details]
MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30079/diff/1-2/

    
    

    
  
Comment on attachment 8705578 [details]
MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30081/diff/2-3/

    
    

    
  
Comment on attachment 8705577 [details]
MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production

https://reviewboard.mozilla.org/r/30079/#review28801

Just a comment about how the symbols show up on TH and about the oranges.  Looks good.

::: testing/taskcluster/tasks/tests/b2g_emulator_reftest.yml:35
(Diff revision 2)
> +    treeherderEnv:

with adding this to production as a tier 1 job, do you still want the group symbol to be tc-R?

I think this would show up as:
tc-R(R1 R2 R3 ...)

Maybe it could be R(1 2 3 4)

Also looking at treeherder it seems that these are orange? does the other commit fix those?  If not, having these show up on production as a tier 1 job probably will be an issue.  Just something to keep in mind :)

        Attachment #8705577 -
        Flags: review?(garndt) → review+

    
    

    
  
Ah, disregard my comment about the oranges, I noticed the comments above about having them hidden.

    
    

    
  
(In reply to Astley Chen [:astley] from comment #41)
> (In reply to Carsten Book [:Tomcat] from comment #37)
> > (In reply to Astley Chen [:astley] from comment #36)
> > > Scope of hidden chunks in reftest are: R(5), tc-R(9,14,21,22,23,40)
> > > Please also let me know how to proceed afterward.
> > 
> > Hey Astley,
> > 
> > i created now a filter that hides this tests and also filed a bug to get
> > them unhidden when the tests are fixed (bug 1239951)
> 
> Hi tomcat, from the several try results, I found that oranges appeared to be
> intermittent within below chunks:
> 5,8,9,13,14,21,22,23,26,29,31,32,33,34,38,39,40.
> To ensure production tests wont be impacted by intermittent oranges, I will
> suggest to hide ALL reftest chunks on production and leave those oranges
> open on staging for try.
> Can you help to do that ? Thanks.
> 

should be done now!

    
    

    
  
https://reviewboard.mozilla.org/r/30079/#review28801

> with adding this to production as a tier 1 job, do you still want the group symbol to be tc-R?
> 
> I think this would show up as:
> tc-R(R1 R2 R3 ...)
> 
> Maybe it could be R(1 2 3 4)
> 
> Also looking at treeherder it seems that these are orange? does the other commit fix those?  If not, having these show up on production as a tier 1 job probably will be an issue.  Just something to keep in mind :)

The group symbol "tc-R" is used to distinguish general reftest and Mulet Reftest(group symbol "R") and it seems a convention used in somewhere.
So far I have no idea on a better naming and no plan to fix it in this bug since the intention of this bug is to enable reftest on B2G KK emulator.
Thanks for pointing out of this, I'll be back to check in bug1162375.

    
    

    
  
Comment on attachment 8693469 [details] [diff] [review]
Enable-reftest-on-emu-x86-kk-Treeherder-production.patch

move to review board

        Attachment #8693469 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 8699301 [details] [diff] [review]
Enable reftest on emu-arm-kk for test

moved to review board.

        Attachment #8699301 -
        Attachment is obsolete: true

    
  
Keywords: checkin-needed

    
    

    
  
failed to apply:

apply changeset? [ynmpcq?]: y
applying 7ac662108697
patching file testing/taskcluster/tasks/branches/base_jobs.yml
Hunk #1 FAILED at 238
1 out of 1 hunks FAILED -- saving rejects to file testing/taskcluster/tasks/branches/base_jobs.yml.rej
patch failed to apply
abort: fix up the merge and run hg transplant --continue
Flags: needinfo?(aschen)
Keywords: checkin-needed

    
    

    
  
Comment on attachment 8705577 [details]
MozReview Request: Bug 1228918 - Enable reftest on emu-{arm,x86}-kk, Treeherder Production

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30079/diff/2-3/

    
    

    
  
Comment on attachment 8705578 [details]
MozReview Request: Bug 1228918 - Fallback Mesa3D Gallium driver from llvmpipe to softpipe. Mesa 8.0.4 llvmpipe can introduce precision issues on reftest on B2G KK emu test.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/30081/diff/3-4/

    
    

    
  
(In reply to Carsten Book [:Tomcat] from comment #53)
> failed to apply:
> 
> apply changeset? [ynmpcq?]: y
> applying 7ac662108697
> patching file testing/taskcluster/tasks/branches/base_jobs.yml
> Hunk #1 FAILED at 238
> 1 out of 1 hunks FAILED -- saving rejects to file
> testing/taskcluster/tasks/branches/base_jobs.yml.rej
> patch failed to apply
> abort: fix up the merge and run hg transplant --continue

rebased and require garndt to review again. thanks.
Flags: needinfo?(aschen) → needinfo?(garndt)

    
    

    
  
Reviewd.
Flags: needinfo?(garndt)

    
  
Keywords: checkin-needed

    
    

    
  
tomcat, can you help again ? thanks a lot.

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/7926adaf2ba7
https://hg.mozilla.org/mozilla-central/rev/2f1fb9cc37d9
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47

    
  
Product: TaskCluster → Firefox Build System

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: