1301260 - Win7 reftests, mochitest-gpu and mochitest-gl-* aren't running on try

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

On trunk, some retriggering showed that opt Win7 e10s Ru is hitting bug 1267106 more than 50% of the time, and PGO is hitting it 28% of the time, so I was getting ready to hide it.

First I needed to see whether it also needed to be hidden on release branches, so I looked at aurora, where it is 100% green, still running on hardware.

That made me look at Try, to see if maybe we were running it both ways there, and I could compare. We aren't running it at all on Try, which is a major party foul.

I remember there being questions about bug 1267106 around the time of the switch to VMs, but... was it only on VM, and green on hardware, or 50% on VM and <1% on hardware?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

these were migrated based on bug 1298417.  Given the data at the time, the only failures we saw were on win7-vm opt and the failure rate (10%) was <= that of hardware for win7-ix at the time.  

Based on comments from orangefactor robot in bug 1267106 (comments 26-29, you can see that the failure rate is the same if not less for win7-hw vs win7-vm).  In fact on September 4th, we still had higher win7-ix failure rates than our current failure rate of win7-ix.

If we want to move this one job (win7-vm/opt/ru-e10s) back to hardware we can- I would like to understand why we think this is worse than it was on hardware- I could be seeing the data incorrectly.

Comment 2

[Phil Ringnalda (:philor)]

Indeed, looks like someone fixed it for Win8 at the end of June, broke it badly for Win7 at the end of July, and the migration didn't particularly change its brokenness.

I only looked at a dozen or so -u all jobs that ran on Windows on try, but I sure didn't see any sign that it's running there, though.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add reftest-no-accel to try server for win7-vm

Builders added:
+ Windows 7 VM 32-bit try debug test reftest-no-accel-e10s
+ Windows 7 VM 32-bit try opt test reftest-no-accel-e10s

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I will be on PTO tomorrow starting in a couple hours- please feel free to land this and deploy as you see fit- otherwise I will pick this up on Monday :)

Comment 5

[Phil Ringnalda (:philor)]

Isn't the problem that that whole chunk is broken by something like the fact that "if 'win7_vm' in win32" is true for both win7_vm and win7_vm_gfx? We are also not scheduling mochitest-gl or mochitest-gpu on try.

Comment 6

[Phil Ringnalda (:philor)]

Nor are we scheduling any of the other reftests, reftest-with-accel, reftest-e10s-with-accel, or reftest-not-e10s-no-accel, on VMs which is where we run them off of try.

Though as https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=f6b1dd1b069b709eba7e0235b98dfd24511ea9f6&tochange=18d51f3de31f41270589e66af0f00f07278b1d94&group_state=expanded which I did to compare what was missing shows, in a brain-twisting way we *do* schedule them for -u all[Windows 7] even though we don't for -u all.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

good catch :philor.  It seems that in addition to this patch up for review we will need to have fix to try server parser to run windows 7 vm stuff by default.  right now there are only 5 jobs which are run on HW, I think we could run everything by default on try and make a small except for those few jobs.

:catlee, how can we resolve the issue :philor mentions in comment 6?

Comment 8

[Chris AtLee [:catlee]]

(In reply to Phil Ringnalda (:philor) from comment #5)
> Isn't the problem that that whole chunk is broken by something like the fact
> that "if 'win7_vm' in win32" is true for both win7_vm and win7_vm_gfx? We
> are also not scheduling mochitest-gl or mochitest-gpu on try.

hmm, if that were true, we wouldn't be running *any* win7 VM (non-GFX) tests on try by default?

Comment 9

[Phil Ringnalda (:philor)]

We wouldn't be? I'm just looking at that one if statement, the one in the patch, the one which removes everything which starts with reftest, starts with mochitest-gl or starts with mochitest-gpu, which is precisely the three sets of tests all of which run on win7_vm_gfx and none of which are being scheduled on try.

Comment 10

[Phil Ringnalda (:philor)]

s/in the patch/in the context of the patch/

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 8789494 [details] [diff] [review]
add reftest-no-accel to try server for win7-vm

Review of attachment 8789494 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +3473,5 @@
>          for test_type in ('opt_unittest_suites', 'debug_unittest_suites'):
>              for t in win32['win7_vm'][test_type][:]:
>                  suite_name, suite_config = t
> +                if suite_name.startswith('reftest-no-accel'):
> +                    continue

can you make the next line explicit as to what suites it's removing instead of skipping over the reftest-no-accel tests here?

Comment 12

[Chris AtLee [:catlee]]

(In reply to Phil Ringnalda (:philor) from comment #10)
> s/in the patch/in the context of the patch/

ah, sorry, I thought you were referring to how trychooser may be working (or not).

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add reftest-no-accel to try server for win7-vm (2.0)

updated with comments and removal of gfx versions as per :philor's catch :)

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

builder diff with above patch:
Builders added:
+ Windows 7 VM 32-bit try debug test reftest-no-accel-e10s
+ Windows 7 VM 32-bit try opt test reftest-no-accel-e10s
Builders removed
- Windows 7 VM-GFX 32-bit try debug test reftest-no-accel
- Windows 7 VM-GFX 32-bit try debug test reftest-no-accel-e10s
- Windows 7 VM-GFX 32-bit try opt test reftest-no-accel
- Windows 7 VM-GFX 32-bit try opt test reftest-no-accel-e10s

Comment 15

[Phil Ringnalda (:philor)]

On mozilla-inbound, we run Win7 reftest-no-accel on gfx, note the slavename in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=2fc46e5f15dc4492d25c516a12cea1f4af89df97&filter-searchStr=windows%207%2032-bit%20reftest-no-accel&selectedJob=35678216

Comment 16

[Phil Ringnalda (:philor)]

Though if you do want to move it from running on win7_vm_gfx on every tree to running on win7_vm on every tree, that would be a fine use for this bug, and I can just start over with a new one for the way that none of the gfx suites run properly on try.

Comment 17

[Phil Ringnalda (:philor)]

Oy. Two fresh new bugs: bug 1302259 for considering whether or not you want to move reftest-no-accel off the GFX VMs, bug 1302256 for the way we moved the GFX jobs into production, but forgot to remove try_by_default: False.

Comment 18

[Phil Ringnalda (:philor)]

Comment on attachment 8790429 [details] [diff] [review]
add reftest-no-accel to try server for win7-vm (2.0)

But either way, this isn't what you want to do, only moving Try.