786052 - run tests and talos on win7-32 to verify multi-node setup

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Armen [:armenzg]]

I will add the details once I am up-to-date with what is needed.

I think; this is going to become my highest priority.

P1 and assigned to me.

Comment 1

[Armen [:armenzg]]

FTR I can RDP into w732-ix-test1.winbuild.scl1.mozilla.com
The name of the slave on slavealloc is w732-ix-test1.

I managed for it to take jobs form buildbot:
http://dev-master01.build.scl1.mozilla.com:8041/buildslaves/w732-ix-test1

Tomorrow I will iron out the remaining issues towards a successful.

This is the documentation on how I have setup this slave so far:
https://etherpad.mozilla.org/armenzg-win7

Comment 2

[Armen [:armenzg]]

We should soon be seeing some successful unit tests and talos jobs for w732-ix-test1 on http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

Things that I have setup differently are (all documented on the etherpad):
* Installed a higher version of Apache
* Installed Mozilla build 1.6.1 instead of 1.4
* Installed python 2.5 separately
* Installed pywin32 extensions for 2.5 and 2.7
* Installed simplejson for Python 2.5

Comment 3

[Armen [:armenzg]]

It seems we need CoreUtils because we run "nohup" for talos.
http://ftp.gnu.org/gnu/coreutils/coreutils-8.19.tar.xz

* "if exist t-* nohup rm -vrf t-*"
catlee: "for some reason the rm command was being killed by the hup signal"
catlee: 'if exist' is a windows batch thing

Comment 4

[Armen [:armenzg]]

> It seems we need CoreUtils because we run "nohup" for talos.
> http://ftp.gnu.org/gnu/coreutils/coreutils-8.19.tar.xz

The previous URL is incorrect. What I did is to unzip CoreUtils's bin directory to C:\CoreUtils\bin.

Downloaded from:
http://gnuwin32.sourceforge.net/downlinks/coreutils-bin-zip.php which redirects to:
http://superb-dca2.dl.sourceforge.net/project/gnuwin32/coreutils/5.3.0/coreutils-5.3.0-bin.zip

diff -r 6185e055e02e env.py
--- a/env.py    Wed Sep 12 19:20:34 2012 -0400
+++ b/env.py    Thu Sep 13 09:05:23 2012 -0700
@@ -142,7 +141,8 @@
             'C:\\cygwin\\bin;' + \
             'C:\\WINDOWS\\System32;' + \
             'C:\\program files\\gnuwin32\\bin;' + \
-            'C:\\WINDOWS;'
+            'C:\\WINDOWS;' + \
+            'C:\\CoreUtils\\bin;'
 }
 
 MozillaEnvironments['win32-perf-unittest'] = {
@@ -165,6 +167,8 @@
             'C:\\WINDOWS;'
 }
 
+MozillaEnvironments['win32-perf']['PATH'] += MozillaEnvironments['win32-perf-unittest']['PATH']
+
 MozillaEnvironments['win64-perf'] = {
     "MOZ_CRASHREPORTER_NO_REPORT": '1',
     "MOZ_NO_REMOTE": '1',

Comment 5

[Armen [:armenzg]]

Attachment: allow w732-ix-test1 to post to graphs.allizom.org

I want to add a slave temporarily to graphs.allizom.org to ensure that it can post to it like any other slave.

I assume that OS_id would be 26.

What do you think?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 660905 [details] [diff] [review]
allow w732-ix-test1 to post to graphs.allizom.org

Review of attachment 660905 [details] [diff] [review]:
-----------------------------------------------------------------

::: sql/data.sql
@@ +526,5 @@
>  insert into machines values (NULL,12,0,"2.26","talos-r3-w7-061",1,unix_timestamp());
>  insert into machines values (NULL,12,0,"2.26","talos-r3-w7-062",1,unix_timestamp());
>  insert into machines values (NULL,12,0,"2.26","talos-r3-w7-063",1,unix_timestamp());
>  insert into machines values (NULL,12,0,"2.26","talos-r3-w7-064",1,unix_timestamp());
> +insert into machines values (NULL,26,0,"2.67","w732-ix-test1",1,unix_timestamp());

the name doesn't match the others, maybe:
talos-r4-w7ix-001

Comment 7

[Armen [:armenzg]]

dustin, can you please land this for me?
insert into os_list values (NULL, "WINNT 6.1 (ix)");
insert into machines values (NULL,26,0,"2.67","w732-ix-test1",1,unix_timestamp());

jmaher agrees with me that the naming of the slave can be determined later.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

those need to be applied to:
graphs.allizom.org (staging)
graphs.mozilla.org (production)

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

done on both prod and staging.

Comment 10

[Armen [:armenzg]]

Thanks dustin!

Comment 11

[Armen [:armenzg]]

Comment on attachment 660905 [details] [diff] [review]
allow w732-ix-test1 to post to graphs.allizom.org

I landed this change to the default branch.
http://hg.mozilla.org/graphs/rev/bd054a44f50a

Is there any other branches I should land it into?
http://hg.mozilla.org/graphs/graph

(added rhelmer & jeads in case one of them knows)

Comment 12

[Armen [:armenzg]]

Adding hwine, arr and moconnor to let them know that the win7 ix node is now running:
* debug unit tests
* opt unit tests

I hope to have talos running correctly either tonight or tomorrow morning.
I will ask someone from the A-team to give me some feedback on failing results.

Up to which point should I get this image to before we can say "we have reached the point that we can say we can support Windows 7 32-bit on this specific hardware"? aka move on to another Windows OS.

Comment 13

[Hal Wine [:hwine] use NI!]

To the point we're confident that any failing results (unit or talos) are not related to the hardware.

Comment 14

[Armen [:armenzg]]

I can't run talos.
There's something weird with the webserver.

I get 403 Forbidden all the time.

From httpd.conf (running Apache 2.2.22 instead of 2.2.15)
DocumentRoot "C:\talos-slave\talos-data\talos"

I run chmod -v -R a+rwx talos but I don't see the permission change (even though the output says it did).
drwxr-xr-x 18 cltbld Administrators    20480 Sep 14 07:49 talos

I tried both chmod from CoreUtils and from mozilla-build (both have the same version 5.97).

Any thoughts?

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Check the Apache config?  And look at the error logs?

Comment 16

[Armen [:armenzg]]

Checking the error logs help me discover that <Directory> was not set to DocumentRoot, hence, being too strict on the permissions. Thanks dustin!

Comment 17

[Armen [:armenzg]]

Attachment: adjustments to screen resolution script

It seems that having a different SP also means a slight different version.

I also addressed that for some odd reason the frequency on this slave is 1. This is probably because the setting on the control panel is "default of device" rather than being explicit about 60Hz (not that I had that option).

This patch also works with our standard win7 slaves.

Comment 18

[Chris Cooper [:coop] (he/him)]

Comment on attachment 662538 [details] [diff] [review]
adjustments to screen resolution script

Review of attachment 662538 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/support/mouse_and_screen_resolution.py
@@ +116,5 @@
>      return {"x": GetSystemMetrics (0), "y": GetSystemMetrics (1)}
>  
> +def queryScreenFrequency():
> +    p=win32api.EnumDisplaySettings(None, 0)
> +    return p.DisplayFrequency

You should probably check to make sure p exists here (and has DisplayFrequency). r+ with that change.

Also (not required for landing, just as a thought), do we want to default to 60 if we can't look up the frequency, or do we want to fail out?

Comment 19

[Armen [:armenzg]]

Attachment: adjustments to screen resolution script (take 2)

We want to fail is we can't determine the frequency.

Using value 0 was incorrect. The value of win32con.ENUM_CURRENT_SETTINGS is -1.
This patch works on win7 minis.

Comment 20

[Armen [:armenzg]]

Comment on attachment 662693 [details] [diff] [review]
adjustments to screen resolution script (take 2)

Carried forward the r+ and live now:
http://hg.mozilla.org/build/tools/rev/511fe98c5d63

Comment 21

[Armen [:armenzg]]

I'm going to be looking into these:
####################
827 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/content/canvas/test/chrome/test_webgl_debug_renderer_info.html | an unexpected uncaught JS exception reported through window.onerror - NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMHTMLCanvasElement.getContext] at chrome://mochitests/content/chrome/content/canvas/test/chrome/test_webgl_debug_renderer_info.html:35
####################
15125 ERROR TEST-UNEXPECTED-FAIL | /tests/gfx/tests/test_acceleration.html | Acceleration enabled on Windows XP or newer - didn't expect 0, but got it
15126 ERROR TEST-UNEXPECTED-FAIL | /tests/gfx/tests/test_acceleration.html | Direct2D enabled on Windows Vista or newer
15127 ERROR TEST-UNEXPECTED-FAIL | /tests/gfx/tests/test_acceleration.html | DirectWrite enabled on Windows Vista or newer
####################

They might ANGLE libraries missing and/or older graphic drivers and/or DirectX missing:
https://wiki.mozilla.org/ReferencePlatforms/Test/Win7#ANGLE_libraries
https://wiki.mozilla.org/ReferencePlatforms/Test/Win7#NVidia_drivers_update_.28Version:_260.99.3B_Date:_2010.10.25.29
https://wiki.mozilla.org/ReferencePlatforms/Test/Win7#DirectX_runtime_.28June_2010.29

Comment 22

[Joel Maher ( :jmaher ) (UTC -8)]

webgl tests require specific graphics cards/drivers.  I learned about this while doing the EC2 stuff last week.  If these are VMs that would be an easy cause, if these are real OS's, it could be the graphics card is too weak.

For the EC2 stuff an ideal situation is to create a 'hw' test that runs all the different harnesses but with the limited set of tests (webgl related) which depend on hardware.  This way on non acceptable graphics hardware we can run everything else.

Comment 23

[Armen [:armenzg]]

I installed the "NVidia drivers update (Version: 260.99; Date: 2010.10.25)", increased the screen resolution to 1280x1024 (currently the max. screen resolution but good enough [1]) and re-run reftests.

In the previous run we got green but USE_WIDGET_LAYERS disabled [2]
On the second run we got USE_WIDGET_LAYERS disabled as well and that is because we reduce our screen resolution back to 1024x768 [4] which is expected since bug 702504 is not yet fixed.

I've re-triggered all of my oranges to see which ones clear up with this.

[1]
https://bugzilla.mozilla.org/show_bug.cgi?id=712630#c119

[2]
REFTEST INFO | WARNING: USE_WIDGET_LAYERS disabled
REFTEST INFO | drawWindow flags = DRAWWINDOW_DRAW_CARET | DRAWWINDOW_DRAW_VIEW; window size = 816,780; test browser size = 800,1000

[3]
REFTEST INFO | WARNING: USE_WIDGET_LAYERS disabled
REFTEST INFO | drawWindow flags = DRAWWINDOW_DRAW_CARET | DRAWWINDOW_DRAW_VIEW; window size = 816,780; test browser size = 800,1000

[4]
http://hg.mozilla.org/mozilla-central/file/default/testing/machine-configuration.json

Comment 24

[Armen [:armenzg]]

I deployed the following:
* Install Angle libraries:
* Install DirectX
* Deploy Mozilla Maintenance Service
but I still get a bunch of test failures.

For instance this:

TEST-INFO | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0110_general.js | running test ...
TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\test_0110_general.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>

TEST-INFO | (xpcshell/head.js) | test 1 pending

TEST-INFO | (xpcshell/head.js) | test 2 pending
TEST-INFO | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/head_update.js | [setupUpdaterTest : 1882] testing successful removal of the directory used to apply the mar file

TEST-PASS | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/head_update.js | [setupUpdaterTest : 1884] false == false
TEST-INFO | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/head_update.js | [runUpdate : 1401] Running the updater: C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\0110_mar\updater.exe C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\0110_mar C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\0110_applyToDir 0 C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\0110_applyToDir\a\b C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test\unit\0110_applyToDir\a\b\callback_app.exe ./ callback.log Test Arg 2 Test Arg 3
TEST-INFO | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0110_general.js | [run_test : 257] testing updater binary process exitValue for success when applying a complete mar

TEST-UNEXPECTED-FAIL | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0110_general.js | 1 == 0 - See following stack:
JS frame :: C:\talos-slave\test\build\xpcshell\head.js :: do_throw :: line 451
JS frame :: C:\talos-slave\test\build\xpcshell\head.js :: _do_check_eq :: line 545
JS frame :: C:\talos-slave\test\build\xpcshell\head.js :: do_check_eq :: line 566
JS frame :: C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test/unit/test_0110_general.js :: run_test :: line 259
JS frame :: C:\talos-slave\test\build\xpcshell\head.js :: _execute_test :: line 315
JS frame :: -e :: <TOP_LEVEL> :: line 1
native frame :: <unknown filename> :: <TOP_LEVEL> :: line 0

TEST-INFO | (xpcshell/head.js) | exiting test
<<<<<<<

Comment 25

[Armen [:armenzg]]

Or these:
827 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/content/canvas/test/chrome/test_webgl_debug_renderer_info.html | an unexpected uncaught JS exception reported through window.onerror - NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMHTMLCanvasElement.getContext] at chrome://mochitests/content/chrome/content/canvas/test/chrome/test_webgl_debug_renderer_info.html:35

760 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/plugins/test/test_painting.html | painted after invalidate (expected 2 independent paints, expected 2 logged paints, got 3 actual paints)

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

bjacob: what is the magic command to run to see what video driver we are using?  it would be nice to get that information in here.

Comment 27

[Benoit Jacob [:bjacob] (mostly away)]

The simplest is to go to about:support and look at the graphics section. On windows you can also run dxdiag.

Comment 28

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Armen Zambrano G. [:armenzg] from comment #25)
> Or these:
> 827 ERROR TEST-UNEXPECTED-FAIL |
> chrome://mochitests/content/chrome/content/canvas/test/chrome/
> test_webgl_debug_renderer_info.html | an unexpected uncaught JS exception
> reported through window.onerror - NS_ERROR_FAILURE: Component returned
> failure code: 0x80004005 (NS_ERROR_FAILURE)
> [nsIDOMHTMLCanvasElement.getContext] at
> chrome://mochitests/content/chrome/content/canvas/test/chrome/
> test_webgl_debug_renderer_info.html:35

Hm, line 35 is:
  var gl = canvas.getContext("experimental-webgl");

So it failed to get a WebGL context. Is this configuration blacklisted? Does the WebGL mochitest (content/canvas/test/webgl, in mochitest-1) run on this machine?

Comment 29

[Armen [:armenzg]]

I will check the other questions on Monday (about:support/gfx).

(In reply to Benoit Jacob [:bjacob] from comment #28)
> (In reply to Armen Zambrano G. [:armenzg] from comment #25)
> > Or these:
> > 827 ERROR TEST-UNEXPECTED-FAIL |
> > chrome://mochitests/content/chrome/content/canvas/test/chrome/
> > test_webgl_debug_renderer_info.html | an unexpected uncaught JS exception
> > reported through window.onerror - NS_ERROR_FAILURE: Component returned
> > failure code: 0x80004005 (NS_ERROR_FAILURE)
> > [nsIDOMHTMLCanvasElement.getContext] at
> > chrome://mochitests/content/chrome/content/canvas/test/chrome/
> > test_webgl_debug_renderer_info.html:35
> 
> Hm, line 35 is:
>   var gl = canvas.getContext("experimental-webgl");
> 
> So it failed to get a WebGL context. Is this configuration blacklisted? Does
> the WebGL mochitest (content/canvas/test/webgl, in mochitest-1) run on this
> machine?
It does not. See below.


Current status:
* debug/opt m1: 
48626 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Can't create a WebGL context

* debug/opt m4:
15125 ERROR TEST-UNEXPECTED-FAIL | /tests/gfx/tests/test_acceleration.html | Acceleration enabled on Windows XP or newer - didn't expect 0, but got it
15126 ERROR TEST-UNEXPECTED-FAIL | /tests/gfx/tests/test_acceleration.html | Direct2D enabled on Windows Vista or newer
15127 ERROR TEST-UNEXPECTED-FAIL | /tests/gfx/tests/test_acceleration.html | DirectWrite enabled on Windows Vista or newer

* debug/opt m-chrome:
827 ERROR TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/content/canvas/test/chrome/test_webgl_debug_renderer_info.html | an unexpected uncaught JS exception reported through window.onerror - NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMHTMLCanvasElement.getContext] at chrome://mochitests/content/chrome/content/canvas/test/chrome/test_webgl_debug_renderer_info.html:35

* talos xperf:
FAIL: Unable to proceed with missing counter 'tp5n_main_startup_fileio'

Logs can be found in here:
http://people.mozilla.com/~armenzg/b786052.win7-ix/debug.m1.txt
http://people.mozilla.com/~armenzg/b786052.win7-ix/debug.m4.txt
http://people.mozilla.com/~armenzg/b786052.win7-ix/debug.moc-chrome.txt
http://people.mozilla.com/~armenzg/b786052.win7-ix/opt.m1.txt
http://people.mozilla.com/~armenzg/b786052.win7-ix/opt.m4.txt
http://people.mozilla.com/~armenzg/b786052.win7-ix/opt.moc-chrome.txt
http://people.mozilla.com/~armenzg/b786052.win7-ix/talos.xperf.txt

Comment 30

[Benoit Jacob [:bjacob] (mostly away)]

OK, so we have three options at this point:

 1. either fix the problem that this machine doesn't get GPU accelerated features. Maybe a driver issue?

 2. or record these tests as expected-to-fail on this configuration

 3. or skip these tests on this configuration

Is this configuration one of the new virtualized/cloud test slaves that we are considering and for which we already know that they can't get GPU accelerated features? If yes, then 3. is the best option there. There is no point in running GPU-dependent tests on this configurations.

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

To my knowledge there are no plans to run tests on virtualized/cloud slaves.  We are purchasing new physical hardware with consumer-grade graphics cards expressly to continue to test GPU-accelerated graphics.

Comment 32

[Chris AtLee [:catlee]]

We're looking into moving some linux tests (that don't require a GPU) over to cloud slaves.

Comment 33

[Armen [:armenzg]]

(In reply to Benoit Jacob [:bjacob] from comment #30)
> OK, so we have three options at this point:
> 
>  1. either fix the problem that this machine doesn't get GPU accelerated
> features. Maybe a driver issue?
> 
>  2. or record these tests as expected-to-fail on this configuration
> 
>  3. or skip these tests on this configuration
> 
> Is this configuration one of the new virtualized/cloud test slaves that we
> are considering and for which we already know that they can't get GPU
> accelerated features? If yes, then 3. is the best option there. There is no
> point in running GPU-dependent tests on this configurations.

bjacob, this is a hardware machine with consumer-grade graphics cards.
I updated the drivers to 260.99.3B_Date:_2010.10.25.29

I will attach the dxdiag information.

Comment 34

[Armen [:armenzg]]

Attachment: dxdiag

Comment 35

[Armen [:armenzg]]

Attachment: Using VGA Adapter rather than NVIDIA card

We have noticed that we're running off the standard VGA adapter rather than the GeForce card.

I'm trying to figure how to get out of this hole:
1) disable standard card and hopefully start using GeForce (I need OOB access - wip)
2) get someone to the colo to figure it out (we might need a dongle)

Comment 36

[Amy Rich [:arr] [:arich]]

FYI, if you disable the standard card, you disable IPMI (that works over the standard card).

Comment 37

[Armen [:armenzg]]

MaRu has switched the machine to use NVIDIA card.
I will put the machine through staging.

Comment 38

[Armen [:armenzg]]

It seems that the new configuration is working and we will have new failures.
I will run the full stack of tests and report all of the issues at once.

For instance for opt m1:
108 ERROR TEST-UNEXPECTED-FAIL | /tests/Harness_sanity/test_sanitySimpletest.html | waitForClipboard should work - got false, expected true
2350 ERROR TEST-UNEXPECTED-FAIL | /tests/browser/base/content/test/test_contextmenu.html | paste for command cmd_copyLink - got , expected http://mozilla.com/
2552 ERROR TEST-UNEXPECTED-FAIL | /tests/browser/base/content/test/test_contextmenu.html | checking item #24 (context-selectall) enabled state - got false, expected true
2623 ERROR TEST-UNEXPECTED-FAIL | /tests/browser/base/content/test/test_contextmenu.html | checking item #10 (context-selectall) enabled state - got false, expected true
2685 ERROR TEST-UNEXPECTED-FAIL | /tests/browser/base/content/test/test_contextmenu.html | checking item #8 (context-selectall) enabled state - got false, expected true
9316 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug166235.html | an unexpected uncaught JS exception reported through window.onerror - NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsITransferable.getTransferData] at chrome://specialpowers/content/specialpowersAPI.js:118
37099 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_copypaste.html | an unexpected uncaught JS exception reported through window.onerror - NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsITransferable.getTransferData] at http://mochi.test:8888/tests/content/base/test/test_copypaste.html:100
124319 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | The audioAvailable event's time attribute was invalid.
124320 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check amount of signal data processed - got 103424, expected 133120
124321 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check signal present - got false, expected true
124322 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check mute fragments present - got false, expected true

VS
48626 ERROR TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Can't create a WebGL context

Comment 39

[Phil Ringnalda (:philor)]

I don't have a feeling about the clipboard-related ones, but test_a4_tone.html is the lack of a sound card.

Comment 40

[Joel Maher ( :jmaher ) (UTC -8)]

I get the same errors while testing on linux in ec2

Comment 41

[Armen [:armenzg]]

oh dear...
now... should I be looking into asking IT to add sounds cards into the machines? or get approval to skip those tests?

Comment 42

[Amy Rich [:arr] [:arich]]

Audio was never one of the requirements that was specified by releng when we looked at replacement hardware.

Comment 43

[Joel Maher ( :jmaher ) (UTC -8)]

There are only a few audio related tests, and those tests specifically are very timing dependent.  We need to determine the root cause and if we need to run these specific tests on this hardware reference set.

Comment 44

[Amy Rich [:arr] [:arich]]

FYI, each node has one PCI-E slot, and we're using it for graphics cards.

http://www.ixsystems.com/servers/ix/2u-servers/ix-22x4-multi-node-hpc-server.html has the specs (and that link was included for review in the original bug that talked about ordering these).

Comment 45

[Armen [:armenzg]]

Adding the writer and reviewers of the test:
http://hg.mozilla.org/mozilla-central/annotate/8776d96f0099/content/media/test/test_a4_tone.html

We're setting up some machines with Windows 7 32-bit to run unit tests.

We would like to know what is it needed for these test failures to work?
(see comment 38)

Thanks!

Comment 46

[Amy Rich [:arr] [:arich]]

jmaher: where else would we run these tests?  joduinn's plan was to use this exact same hardware for all talos desktop testing except for OS X (aka windows and linux).

Comment 47

[Armen [:armenzg]]

Could a dummy sound driver be added as a solution?

Comment 48

[Amy Rich [:arr] [:arich]]

Beyond the real of my expertise...  maru et.al., could something like virtual audio cable (https://en.wikipedia.org/wiki/Virtual_Audio_Cable) work (suggestion courtesy of cshields)?

Comment 49

[Matthew Larrain[:MaRu]]

Virtual audio cable is probably your best bet.

Comment 50

[Benoit Jacob [:bjacob] (mostly away)]

Comment on attachment 678292 [details]
dxdiag

Sorry, what would you like me to do about this file?

This dxdiag output shows that this machine has an extremely outdated GPU,
          Card name: Standard VGA Graphics Adapter
       Manufacturer: (Standard display types)
          Chip type: Matrox Graphics Inc.

There is no chance to run any GPU-accelerated feature on this hardware.

Comment 51

[Matthew Larrain[:MaRu]]

bjacob there is a GT430 card from nvidia in that machine as well as the onboard card. We want to know if we can run this system with the onboard as the primary card and have the tests run against the gt430 as a secondard display card.

Comment 52

[Armen [:armenzg]]

I run a complete set of tests (talos jobs and xpcshell will have to be re-triggered) and here's our current summary.

For mochitest-1 (opt & debug) I will give a shot to Virtual Audio Cable and see what happens.
For mochitest-browser-chrome I will attempt to be connected to the machine while the job is running to see what is going on. 

* Rev3 WINNT 6.1 mozilla-central debug test mochitest-1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1352344409.1352346483.31768.gz&fulltext=1
124440 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | The audioAvailable event's time attribute was invalid.
124441 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check amount of signal data processed - got 103424, expected 133120
124442 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check signal present - got false, expected true
124443 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check mute fragments present - got false, expected true

* Rev3 WINNT 6.1 mozilla-central opt test mochitest-1
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1352324854.1352325868.6862.gz&fulltext=1
124420 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | The audioAvailable event's time attribute was invalid.
124421 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check amount of signal data processed - got 103424, expected 133120
124422 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check signal present - got false, expected true
124423 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_a4_tone.html | Check mute fragments present - got false, expected true

* Rev3 WINNT 6.1 mozilla-central opt test mochitest-browser-chrome
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1352316693.1352318020.28651.gz&fulltext=1
TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/base/content/test/browser_bug409481.js | copy link command - Got , expected http://www.example.com/ctest
TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/devtools/commandline/test/browser_cmd_screenshot.js | Screenshot got created and copied
TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/devtools/commandline/test/browser_cmd_screenshot.js | Screenshot present in clipboard in private browsing
TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/devtools/commandline/test/browser_cmd_screenshot.js | Screenshot taken in private browsing mode is not present outside of it in the clipboard
TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/devtools/webconsole/test/browser_webconsole_bug_587617_output_copy.js | Timed out while polling clipboard for pasted data.

Comment 53

[Armen [:armenzg]]

The mochitest-1 failures for test_a4_tone.html cleared up after installing Virtual Audio Cable.

I'm re-triggering now the talos jobs that needed to be re-run + debug xpcshell.

Comment 54

[Armen [:armenzg]]

This has been great.
All of the opt, talos and debug jobs have run green to the exception to talos xperf.
Talos xperf should be solved after re-adding the talosslave scheduled task with highest privilege (unfortunately xperf needs high privileges to run).

Comment 55

[Armen [:armenzg]]

The only orange that we have (which I won't even bother to look much into since no one cares [1]) is a debug jetpack job:
http://people.mozilla.com/~armenzg/b786052.win7-ix/ix.win7.debug.jetpack.html

Unless I hear objections from anyone we're done in here and from my standpoint this hardware with this graphic card does the job.

I'm going to run a one full set again but only intermittent oranges could show up:
http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/w732-ix-test1?numbuilds=100
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest



[1] https://tbpl.mozilla.org/?noignore=1&jobname=test%20jetpack

Comment 56

[Justin Wood (:Callek)]

(In reply to Amy Rich [:arich] [:arr] from comment #48)
> Beyond the real of my expertise...  maru et.al., could something like
> virtual audio cable (https://en.wikipedia.org/wiki/Virtual_Audio_Cable) work
> (suggestion courtesy of cshields)?

yes Armen, as an FYI this was in use on the w2k3 ref image, https://wiki.mozilla.org/ReferencePlatforms/Win32#Install_the_virtual_audio_driver

And has been a part of what SeaMonkey has been using for related audio tests for a while, and things seem to work well.

I should note that, at least with the free version, there is a generic audio sound prepending to the audio stream when invoked. I did not find any issues in SeaMonkey testing with that yet, but worth knowing about.

Comment 57

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to Matthew Larrain[:MaRu] from comment #51)
> bjacob there is a GT430 card from nvidia in that machine as well as the
> onboard card. We want to know if we can run this system with the onboard as
> the primary card and have the tests run against the gt430 as a secondard
> display card.

Any reason to not just run the system with the gt430 as primary?  Unless the gt430 can somehow drive a virtual second display or something and we can make sure firefox starts up on it and that the hw accel stuff chooses that display card, there's no way to just run tests against the secondary gpu.

Comment 58

[Amy Rich [:arr] [:arich]]

The OOB display uses the primary graphics card, so if you disable that you lose OOB video (which is used to deploy the machine and debug issues).

Comment 59

[Dustin J. Mitchell [:dustin] (he/him)]

:vlad, if you know of a simple, automated way to configure *Windows* to use that display as primary, without disabling onboard in the BIOS (which affects OOB), then that may be a good solution.  We know of no such way at this point.

Comment 60

[Armen [:armenzg]]

Attachment: [wip] win7 configs

This patch is not good since all of the IX machines should be in another platform of themselves to not mix results with the Rev3 machines.

Nevertheless, this did the job on staging.

Comment 61

[Armen [:armenzg]]

Attachment: [wip] win7 custom

Probably the ix machines should need their own separate env.py entry.