:arr, can you help get this into the right place to verify we have setup these machines properly?
Arr is currently PTO for the next few days. @alin, @andrei: do you have any idea?
:aobreja is looking at this today to see if there's anything we can assist with before arr's return. Removing :aselagea's NI.
> I suspect these are related to the fact that there is a need to do something > in the bios for the video card which is unique to windows 10. Did some investigation and came to conclusion that this issue is not generated by video card of some settings in bios since the e10 mochitest jobs finish successfully most of the time and sometime failed,even on the same machine. By checking  and filter after "failed" jobs or "e10n mochitest" or "name of the failed job" we see that some failed jobs have green status for other machines,machines that have the same configuration.Also by checking  you can see that there are lots of webgl tests that are green.In fact there are very few that have failed. I don't think this is a bios setting or a machine issue since for the other machines these tests are running well,this sounds more like exception issue.I'm pretty sure that if we let for few days these test to ran the same test will pass on a machine where it failed before. https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix?numbuilds=500 https://treeherder.mozilla.org/#/jobs?repo=try&revision=aba49bf51ce76fb1e625752a04f2af25e5f4d0f7&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=131158264
thanks for looking into this- it likely could be related to the tests- I just found it odd that the failures were in a small range of machine numbers- let me collect more failures. looking at machine stats, I see consistent failures for gpu and webgl jobs: 75 has a green instance https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-076?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-077?numbuilds=500 (all 'gpu' jobs are red) https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-078?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-079?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-080?numbuilds=500 ... https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-090?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-091?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-092?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-093?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-094?numbuilds=500 95 had no instances 96 had green instances and on a machine outside of the range I don't see failures on gpu or gl- jobs: https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-103?numbuilds=500 https://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix-105?numbuilds=500 I really think this is something related to machines- I have seen this before and the patterns are really odd- but I am doing many retriggers to see if there are anynew patterns.
this was also seen by :philor in reference to failures seen on mozilla-beta in bug 1400099. I believe this is hardware related and would like to see these machines reimaged.
Disabled t-w1064-ix-076 through t-w1064-ix-095.
Also caused reftest failures, bug 1400004 / https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1400004
Did some investigation why only this range cause problems and it seems that machines between [076-095] had the onboard graphics enabled and also they ran at lower resolution. I fixed that by disable onboard graphics and verify that all machines are on the right resolution and enable them back .The problem should be fixed since now these machines are configured the same way as the other that don't cause issue.
The problem is solved,I don't see any other failing webgl tests since the change. I'll mark this bug as resolved ,if anything change please feel free to re-open the bug.
Moving this in our courtyard and giving credit to :aobreja for fixing this.
3 failures in 943 pushes (0.003 failures/push) were associated with this bug in the last 7 days. Repository breakdown: * try: 3 Platform breakdown: * windows10-64: 3 For more details, see: https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1400071&startday=2017-09-18&endday=2017-09-24&tree=all