549120 - Automated reftests need to be run with direct2d enabled

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Jeff Muizelaar [:jrmuizel]]

I'm not exactly sure what's all needed to get this working. Is it possible for our VMs to emulate gfx hw like the desktop vmware products do? Otherwise we might be able to use one of the software direct3d implementations.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

It looks like Windows Server 2003 does not support Direct2D, and all reftests currently run on our build slaves using that OS, so we can't support this right now.

We did recently add Windows 7 talos machines, which carry the WINNT 6.1 label on tbox and replaced the Vista machines. And we are heading towards running all the test jobs on the mini talos farm, so Direct2D could be tested once bug 548768 is resolved. 

Would you want to reftest with and without Direct2D ? We'd need to know if that was going to be handled in the testsuite or if extra arguments would be needed when calling it.

(In reply to comment #0)
> I'm not exactly sure what's all needed to get this working. 

It'd be very helpful if developers err on the side of providing more information when filing these requests, since they're going to know much more about <new-feature> than we are.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #1)
> It looks like Windows Server 2003 does not support Direct2D, and all reftests
> currently run on our build slaves using that OS, so we can't support this right
> now.
> 
> We did recently add Windows 7 talos machines, which carry the WINNT 6.1 label
> on tbox and replaced the Vista machines. And we are heading towards running all
> the test jobs on the mini talos farm, so Direct2D could be tested once bug
> 548768 is resolved. 
> 
> Would you want to reftest with and without Direct2D ? We'd need to know if that
> was going to be handled in the testsuite or if extra arguments would be needed
> when calling it.

Yes, we'd like to reftest with and without Direct2D. This could be done with a pref change, but I think it would be ideal to have reftests/unittests run on machines that support direct2d and machines that don't.

> It'd be very helpful if developers err on the side of providing more
> information when filing these requests, since they're going to know much more
> about <new-feature> than we are.

Here's some more information:
- Direct2D can run on either Vista or Windows 7
- We could run the direct2d builds against real graphics cards. It's possible that virtualized ones would work, but that might make things trickier. What are our options here?
- If we can't run against real graphics cards, we can probably run using a software rasterizer (either the reference rasterizer or the WARP one). Using the software rasterizer might be a good idea anyways because it will mean we're less likely to be plagued by bugs in the 3d drivers. And if there are bugs in the software rasterizer, reproducing them will be much easier than if we used actual hardware.

Comment 3

[Jeff Muizelaar [:jrmuizel]]

When can we expect to get tests run on Vista or Win7?

Comment 4

[Damon Sicore (:damons)]

Can we get an answer to the question in Comment 3, please?

Comment 5

[Justin Fitzhugh]

According to dsicore - this is needed in q2.  John, can you verify this is on the list for q2?

Comment 6

[Aki Sasaki (not active)]

Bug 548768 (mentioned in comment 1) is a q1 goal that is finishing up, and involves unit tests on Win7 if I'm not mistaken.  If the tests in question are already running, that bug will take us most if not all the way there.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #3)
> When can we expect to get tests run on Vista or Win7?

This is being worked on (as a Q1 goal) in bug#548768, hence the bugdep that nthomas added. Exact details in the bug, but we are trying really hard to hit that goal.

Note: We no longer run Vista on our infrastructure, and only run on WinXP and Win7. Does direct2d apply to WinXP?


> (In reply to comment #1)
> > Would you want to reftest with and without Direct2D ? We'd need to know if that
> > was going to be handled in the testsuite or if extra arguments would be needed
> > when calling it.
> 
> Yes, we'd like to reftest with and without Direct2D. This could be done with a
> pref change, but I think it would be ideal to have reftests/unittests run on
> machines that support direct2d and machines that don't.

We run all tests on pools of identical machines; it scales better and helps avoid intermittent test failures. It will take us much longer to spin up a new separate pool of differently configured hardware just for this test suite. We can do that if it is *needed*, but if a pref change does work, that would be much quicker.

Comment 8

[Joe Drew (not getting mail)]

We don't need different machines - the 2.26 GHz Talos minis are just fine.

Comment 9

[Rob Arnold [:robarnold]]

(In reply to comment #2)
> - Direct2D can run on either Vista or Windows 7
I just wanted to add that Direct2D should also run on Windows Server 2008 SP2 w/Platform Update and Windows Server 2008 R2. I don't know if that makes it easier since 2008 R2 should be a long-supported version like 2003 was.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

> > (In reply to comment #1)
> > Yes, we'd like to reftest with and without Direct2D. This could be done with a
> > pref change, but I think it would be ideal to have reftests/unittests run on
> > machines that support direct2d and machines that don't.
> 
> We run all tests on pools of identical machines; it scales better and helps
> avoid intermittent test failures. It will take us much longer to spin up a new
> separate pool of differently configured hardware just for this test suite. We
> can do that if it is *needed*, but if a pref change does work, that would be
> much quicker.

(In reply to comment #8)
> We don't need different machines - the 2.26 GHz Talos minis are just fine.

Using the shared pool of identical machines is good. What bug# is tracking adding the pref to the start of reftest suite?

Comment 11

[Mike Shaver (:shaver emeritus)]

What's the latest status here?  Hardware acceleration is heating up as a competitive element, and we have a pretty strong story once we are comfortable shipping it on by default.

Jeff: I think you need to answer John's question in comment 10!

Comment 12

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #10)
> Using the shared pool of identical machines is good. What bug# is tracking
> adding the pref to the start of reftest suite?

This one :)

We just need to run the reftests as follows:

python runreftest.py --setpref="gfx.font_rendering.directwrite.enabled=true"
--setpref="mozilla.widget.render-mode=6" ../../../layout/reftests/reftest.list

Comment 13

[Jeff Muizelaar [:jrmuizel]]

Can we get an ETA on this?

Comment 14

[Armen [:armenzg]]

Changing the dependency to the tracking bug specific for running Windows unit tests on the minis.

The project is now on my hands (just being handed off) and it is one of my high priority goals for this quarter.

I can't give you a good ETA yet.

I can start running this gfx reftest on staging in the next week or two and you could start having look at it.

I expect to have Win7 and WinXP slaves running unit tests on 1st-2nd week of June.

If we can run this reftest on XP that will make things faster (since we can deploy packages on XP in an automated way). If not I will give higher priority to getting unit tests running on Win7 and get this out first.


Jeff how long does this reftest take to run? I want to know to see if I can get it in with mochitest-other OR have it as a separate test suite (this adds a little more CPU overhead to set up).

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Armen, this is something we have to run on the Windows 7 slaves (see comments above).

Comment 16

[Armen [:armenzg]]

(In reply to comment #15)
> Armen, this is something we have to run on the Windows 7 slaves (see comments
> above).

Then the highest priority will be given to it. Thanks.

Comment 17

[Armen [:armenzg]]

Attachment: enable the d2d reftest [just for staging purposes]

I am running now on staging the d2d reftests.

This will show up in:
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest
with the builder name being:
"WINNT 6.1 mozilla-central opt test reftest-d2d"

Jeff this log makes more sense to what we saw together.
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1274210842.1274211780.31221.gz

I will be running on staging the whole suite of unit tests and only one talos test (so at least I run some talos jobs). If the staging slaves are significantly back-logged please let me know and I will reduce the set of unit tests so we can catch up.

I will check in the morning how things are going. For now I will continue my work on deploying what I need to run the Windows unit tests on production.

Comment 18

[Jeff Muizelaar [:jrmuizel]]

We seem to be crashing when running the following, however we don't a proper crash stack. In fact, something seems to be broken with the harness:

REFTEST INFO | Loading http://localhost:4444/1274211382075/55/font-face/prop-order-over-rule-order-1a.html
REFTEST INFO | Loading http://localhost:4444/1274211382075/55/font-face/prop-order-over-rule-order-2a.html
!!! error running onStopped callback: TypeError: callback is not a function
INFO | automation.py | Application ran for: 0:05:59.162000
INFO | automation.py | Reading PID log: c:\users\cltbld\appdata\local\temp\tmpda_xtlpidlog
==> process 968 launched child process 3468
INFO | automation.py | Checking for orphan process with PID: 3468
Traceback (most recent call last):
  File "reftest/runreftest.py", line 267, in <module>
    main()
  File "reftest/runreftest.py", line 264, in main
    sys.exit(reftest.runTests(args[0], options))
  File "reftest/runreftest.py", line 161, in runTests
    timeout=options.timeout + 30.0)
  File "c:\talos-slave\mozilla-central-win7-opt-u-reftest-d2d\build\reftest\automation.py", line 742, in runApp
    self.checkForZombies(processLog)
  File "c:\talos-slave\mozilla-central-win7-opt-u-reftest-d2d\build\reftest\automation.py", line 675, in checkForZombies
    if self.isPidAlive(processPID):
  File "c:\talos-slave\mozilla-central-win7-opt-u-reftest-d2d\build\reftest\automation.py", line 512, in isPidAlive
    ctypes.windll.kernel32.GetExitCodeProcess(pHandle, self.ctypes.byref(pExitCode))
AttributeError: 'Automation' object has no attribute 'ctypes'

Comment 19

[Armen [:armenzg]]

I am waiting on IT to kick the machine that was running this test (the other machine was borrowed for another experiment). I will have this test running again before the EOW.

(In reply to comment #18)
> We seem to be crashing when running the following, however we don't a proper
> crash stack. In fact, something seems to be broken with the harness:
>
Anything that you want me to try/do?

Comment 20

[Jeff Muizelaar [:jrmuizel]]

> (In reply to comment #18)
> > We seem to be crashing when running the following, however we don't a proper
> > crash stack. In fact, something seems to be broken with the harness:
> >
> Anything that you want me to try/do?

Looks like this problem was caused by bug 543825

Comment 21

[Armen [:armenzg]]

This is new to me:
WindowsError: [Error 22] The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail
Unknown Error: command finished with exit code: 1

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1274393027.03.1274394104.24392.gz

Any ideas?

Comment 22

[Bas Schouten (:bas.schouten)]

(In reply to comment #21)
> This is new to me:
> WindowsError: [Error 22] The application has failed to start because its
> side-by-side configuration is incorrect. Please see the application event log
> or use the command-line sxstrace.exe tool for more detail
> Unknown Error: command finished with exit code: 1
> 
> http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1274393027.03.1274394104.24392.gz
> 
> Any ideas?
I'd recommend looking in the Application Event Log! ;)

Comment 23

[Armen [:armenzg]]

I think I have found it by myself :)
Fore reference:
"Event viewer->Windows Logs->Application" and I found an error of SideBySide

Bas, Jeff anything you want me to try? Do you want access to the machine to try fixing it?

Log Name:      Application
Source:        SideBySide
Date:          5/21/2010 6:00:00 AM
Event ID:      33
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      talos-r3-w7-001
Description:
Activation context generation failed for "c:\talos-slave\mozilla-central-win7-debug-u-reftest-d2d\build\firefox\firefox.exe". Dependent Assembly Microsoft.VC80.DebugCRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.0.50727.762" could not be found. Please use sxstrace.exe for detailed diagnosis.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="SideBySide" />
    <EventID Qualifiers="49409">33</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-05-21T13:00:00.000000000Z" />
    <EventRecordID>102343</EventRecordID>
    <Channel>Application</Channel>
    <Computer>talos-r3-w7-001</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Microsoft.VC80.DebugCRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.0.50727.762"</Data>
...
    <Data>c:\talos-slave\mozilla-central-win7-debug-u-reftest-d2d\build\firefox\firefox.exe</Data>
...
  </EventData>
</Event>

Comment 24

[James Ross]

It needs the debug MSVC runtime as it's a debug build; http://blogs.msdn.com/jigarme/archive/2008/05/08/vc-debug-c-runtime-files.aspx is probably a good bit of information on how to get them from a dev box (with the right MSVC version installed).

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Sounds like bug 562459 to me.

Comment 26

[Armen [:armenzg]]

(In reply to comment #25)
> Sounds like bug 562459 to me.

It sounds to me as well.

Jeff AFAIK we can't run unit tests against debug builds on the testing machines without installing Visual Studio. I might have to install to get us by just for reftests d2d.

Comment 27

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

jrmuizel, armenzg:

Does this test-suite-run-with-d2d-params work ok on an opt build?

(comments#21-26 seem to be all about debug builds. I know debug builds useful, and we want to test debug builds also. However, I first want to step back and verify if the current problem is in the toolchain with debug builds, or something else)

Comment 28

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #27)
> jrmuizel, armenzg:
> 
> Does this test-suite-run-with-d2d-params work ok on an opt build?

Yes. However, the opt test runs don't seem to be running anymore?

Comment 29

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #28)
> (In reply to comment #27)
> > jrmuizel, armenzg:
> > 
> > Does this test-suite-run-with-d2d-params work ok on an opt build?
> 
> Yes. However, the opt test runs don't seem to be running anymore?

We'll get those re-started this week or early next week, depending on some unrelated downtime work tomorrow.

After that, we can then loop back to what exactly is needed for running the test suite on debug builds.

Comment 30

[Armen [:armenzg]]

I will update the bug as soon as I have it running again.
Do not consider it running until then.
I should have it back running before the end of tomorrow.

Comment 31

[Armen [:armenzg]]

I forgot to update my last comment.

I have re-enabled this on staging.

I have a couple of patches up for review in bug 549458.

We have to deploy MozillaBuild this week and land those two patches and we should have this running on production (only for normal builds; no debug unit tests yet).

Comment 32

[Armen [:armenzg]]

I have enabled the reftests-d2d test suite this morning.

Can we close this bug and file following bugs if anything raises up?

AFAIK the only failing test in:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1277390627.1277391635.27033.gz
is REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/mozilla-central-win7-opt-u-reftest-d2d/build/reftest/tests/layout/reftests/text/cgj-01.html
which seems to be code related rather than releng related (IIUC).

Comment 33

[Armen [:armenzg]]

This is running on production.

Reopen if there is anything missing.

Comment 34

[Jeff Muizelaar [:jrmuizel]]

I don't see any results on http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest. I'm also suspicious of only one reftest failing. There should be more than that.

Comment 35

[Armen [:armenzg]]

I should have been more explicit wrt to what production means.

You can see the test in:
http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox
called as:
"Rev3 WINNT 6.1 mozilla-central opt test reftest-d2d"

I have just unhidden it since we had few green runs.
Could you review one of the green runs and tell me if you feel comfortable?

This is the command that is run:
> C:\Windows\system32\cmd.exe /c python reftest/runreftest.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols --setpref="gfx.font_rendering.directwrite.enabled=true" --setpref="mozilla.widget.render-mode=6" reftest/tests/layout/reftests/reftest.list

Comment 36

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

jrmuizel: from today's platform meeting, it seems like there is something wrong here - the opt direct2d tests are running green when you expect them to be failing?

Comment 37

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #36)
> jrmuizel: from today's platform meeting, it seems like there is something wrong
> here - the opt direct2d tests are running green when you expect them to be
> failing?

That's correct. It seems like we're not actually testing direct2d in this case for some reason.

Comment 38

[Armen [:armenzg]]

I have loaned a machine to Jeff and we reproduced the oranges manually. I will be debugging this in the morning.

Comment 39

[Armen [:armenzg]]

Reporting back.

Jeff was able to run the box-shadown tests and it reports the 3 known test failures.

This morning I tried to run the whole test suite and it crashed as the tinderbox logs are currently showing. The problem that the crash it is being shown on tinderbox as GREEN.

Why does it show it as GREEN? Because a controlled crash happens and it does not return an exit code that should turn the build orange/red. Nevertheless, that should not distract us on why are the box-shadow tests passing when they should not.

My first theory was that because it crashes it stops running any more unit tests and therefore it never reaches the box-shadow unit tests. The problem is that I see the tests being run and passing:

> TEST-START | file:///c:/talos-slave/mozilla-central-win7-opt-u-reftest-d2d/build/reftest/tests/layout/reftests/box-shadow/boxshadow-onecorner.html
> REFTEST TEST-START | file:///c:/talos-slave/mozilla-central-win7-opt-u-reftest-d2d/build/reftest/tests/layout/reftests/box-shadow/boxshadow-onecorner-ref.html
> REFTEST TEST-PASS | file:///c:/talos-slave/mozilla-central-win7-opt-u-reftest-d2d/build/reftest/tests/layout/reftests/box-shadow/boxshadow-onecorner.html |

When I look at a "WINNT 5.2 mozilla-central opt test reftest" I see that 4681 (4451/0/230) tests have been run which matches to the number of reftests run on reftest-d2d and reftest for Win7 (4445/0/236).

This means that the boxshadow tests are being run but are not failing as they should.



Jeff could you please keep on poking at the machine and see if you can come up with any ideas? I will be back on Monday to keep on trying.

If you want to reproduce the crash run the svg tests and it should crash:
> > C:\Windows\system32\cmd.exe /c python reftest/runreftest.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols --setpref="gfx.font_rendering.directwrite.enabled=true" --setpref="mozilla.widget.render-mode=6" reftest/tests/layout/reftests/svg/reftest.list
Here is also the MINIDUMP variable to set:
> set MINIDUMP_STACKWALK=c:\talos-slave\mozilla-central-win7-opt-u-reftest-d2d\tools\breakpad\win32\minidump_stackwalk.exe

Comment 40

[Armen [:armenzg]]

Attachment: [buildbotcustom] fix the reftests-d2d tests

This fixes the reftests for Direct2d.
Unfortunately the double quotes were messing up the user.js file with this:
> user_pref(""gfx.font_rendering.directwrite.enabled", true");
> user_pref(""mozilla.widget.render-mode", 6");

This should be fixed in the next reconfigure of the testing masters.

Comment 41

[Armen [:armenzg]]

The box-shadow tests are now failing as they should have:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1278952160.1278953225.1644.gz&fulltext=1

AFAIU this bug is FIXED (wrt running it correctly). There a bunch of oranges which I will have to check and file if needed.

Bas or Jeff could you please confirm that the tests you want are matching to watch you expect?

> REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/mozilla-central-win7-opt-u-reftest-d2d/build/reftest/tests/layout/reftests/box-shadow/boxshadow-rounded-spread.html |
> REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/mozilla-central-win7-opt-u-reftest-d2d/build/reftest/tests/layout/reftests/box-shadow/boxshadow-onecorner.html |

Comment 42

[Armen [:armenzg]]

I have filed bug 578110 to track all-known oranges after enabling properly the test suite.

Let's keep this bug closed and file a release engineering bug if any of the tracked oranges requires our intervention.

Comment 43

[[e-mail] (:bhearsum)]

Comment on attachment 456872 [details] [diff] [review]
[buildbotcustom] fix the reftests-d2d tests

Looks like this has already been landed.

Comment 44

[Armen [:armenzg]]

And the landing:
http://hg.mozilla.org/build/buildbotcustom/rev/026384ea664d

Comment 45

[Armen [:armenzg]]

BTW not sure if this worked before but now I am able to see the reftests D2D on TBPL on the noignore page.
http://tests.themasta.com/tinderboxpushlog/?tree=Firefox&noignore=1

Comment 46

[Bas Schouten (:bas.schouten)]

(In reply to comment #45)
> BTW not sure if this worked before but now I am able to see the reftests D2D on
> TBPL on the noignore page.
> http://tests.themasta.com/tinderboxpushlog/?tree=Firefox&noignore=1

Also, all the normal Win7 tests are now ran with D2D as well :)