Closed Bug 1370636 Opened 4 years ago Closed 3 years ago

Enable Wd job on Windows platforms


(Testing :: geckodriver, enhancement, P1)

Version 3


(firefox64 fixed)

Tracking Status
firefox64 --- fixed


(Reporter: ato, Assigned: whimboo)


(Blocks 2 open bugs)



(3 files, 4 obsolete files)

We currently compile geckodriver on Windows, but we are not running any tests.  We need to enable the Wd job on Windows platforms.
Depends on: 1368264
Via bug 1367477 I added the trychooser entry for wdspec tests, which should be available soon.
Depends on: 1367477
Oh, it should have been more an info than a dependency. Reverting it.
No longer depends on: 1367477
Priority: -- → P1
Blocks: 1417649 made sure the geckodriver binary was
included in the correct resource file on macOS.  To enable running
the Wd job (WPT WebDriver tests) on Windows on try we need to
achieve the same effect there.

nfroyd, do you have any pointers what files we need to tickle for
Flags: needinfo?(nfroyd)
Probably browser/installer/; searching for "MOZ_UPDATER" or "MOZ_CRASHREPORTER" or "pingsender" should provide enough examples to cargo-cult from?
Flags: needinfo?(nfroyd)
Priority: P1 → P3
I do see the geckodriver executable as being part of the archive. So this should be basically all what we need to get the Wd jobs running on Windows. I will have a look if we can make that happen given that it drives me nuts we only run the tests on Linux.
I got the job created via Taskcluster on Windows, and also updated the mozharness configs for the correct location of the geckodriver.exe. But starting geckodriver.exe fails because VCRUNTIME140.DLL is missing on the AMI.

Rob or Peete could one of you have a look at this?
Assignee: nobody → hskupin
Flags: needinfo?(rthijssen)
Flags: needinfo?(pmoore)
Priority: P3 → P1
I triggered some more jobs. Maybe this only affects Windows 7 but not Windows 10 AMIs.
there are a few instances of vcruntime140.dll available on the gecko-1-b-win2012 worker type.

here's a task that lists them all (using `C: && cd C:\ && dir vcruntime140.dll /s`):
Flags: needinfo?(rthijssen)
Rob, that's a build machine but not a test machine. In my case the worker type is gecko-t-win7-32.
Flags: needinfo?(rthijssen)
reran on win 7 & 10.
looks like it's not available on the win 7 workers but it is on the win 10
this is how we install it on windows 2012:

we could try something similar on windows 7 but i have no idea if it will work.

note that the 32 bit installer (vcredist_vs2015_x86) creates: C:\Windows\SysWOW64\vcruntime140.dll
while the 64 bit installer (vcredist_vs2015_x64) creates: C:\Windows\System32\vcruntime140.dll

which suggests to me that there will be hurdles installing this on a 32 bit system.
Flags: needinfo?(rthijssen)
Please note that the same problem is also visible on Windows 10.
Flags: needinfo?(pmoore)
testing vcredist install on beta workers (gecko-t-win7-32-beta, gecko-t-win10-64-beta)
Depends on: 1460042
Now it works better but we get dozen of `Test harness output was not a valid structured log message` failures:

> 08:49:21 CRITICAL - platform win32 -- Python 2.7.14, pytest-unknown, py-1.5.2, -- Z:\task_1525854760\build\venv\Scripts\python.exe

I assume this is pytest, which is just logging to stdout via print?

But I do not understand the following:

> 08:49:21 CRITICAL - tests/web-platform/tests/infrastructure/webdriver/tests/ {"source": "web-platform-tests", "thread": "Thread-TestrunnerManager-1", "time": 1525855761591, "action": "log", "message": "Got command: u'log'", "level": "DEBUG", "pid": 4512}

This is coming from TestrunnerManager and fails for the message `Got command:` which is logged via:

> self.logger.debug("Got command: %r" % command)

Maybe it is because the message doesn't contain a log level?

> {"source": "web-platform-tests", "test": "/infrastructure/webdriver/tests/", "thread": "Thread-TestrunnerManager-1", "time": 1525855761129, "action": "test_start", "pid": 4512}

James could you please help?
Flags: needinfo?(james)
Note that the reference without the log level comes from the raw.log while the reference with the level is from the log viewer.
Btw. the only reference I was able to find for such a failure message is in the `StructuredOutputParser` of mozharness and it seems that the action as contained in the log line is not valid:

For the above cases it would mean "log" and "test_start", which seem to be fine? Also why does that only happen on Windows but not Linux?
We should be wrapping the output from pytest so that it ends up in the logger rather than going directly to stdout. I guess there is a Windows/Linux difference here; one thought that comes to mind is that it could be caused by the difference between multiprocessing on the two platforms (on Linux Python attempts to share memory in a copy-on-write sense, on Windows it sensibly doesn't).
Flags: needinfo?(james)
Attachment #8974032 - Attachment is obsolete: true
Attachment #8974033 - Attachment is obsolete: true
MozReview-Commit-ID: BlFMMXt8WBU
MozReview-Commit-ID: FgwitlstFP7
No longer blocks: 1417649
James, as promised here is a current wdspec test job which shows the structured log problem:

It would be great if we could have a look at this today.
Flags: needinfo?(james)
Depends on: 1495007
Blocks: 1495513
Blocks: 1495521
With bug  1495007 fixed we are now able to enable the Wd test jobs on Windows. Here is a try build from James with the patch on the other patch applied:

The two tests which are currently failing will be disabled on Windows.

Also once the tests are running on Windows, we will go ahead and also enable headless tests on bug 1493907.
Ok, so I pushed another try build:

Sadly I'm not able to disable those affected tests. James, mind having a look at those disable statements of the manifest files? I wonder why it doesn't work:

I actually found this logic in some other manifest files so I wonder if that is Wd only, or doesn't work at all.

Also please ignore the trailing space after the sub test name in the first manifest. It's not the problem here given the other test is also not getting disabled. Thanks.
The patch makes sure that the correct path to the geckodriver binary
is used for the web-platform-tests script.
We solved the manifest question via IRC. It's all working now:
Flags: needinfo?(james)
Comment on attachment 9013594 [details]
Bug 1370636 - [wdspec] Enable Wd jobs on Windows platforms. r?jgraham

James Graham [:jgraham] has approved the revision.
Attachment #9013594 - Flags: review+
Attachment #9006455 - Attachment is obsolete: true
Attachment #9006456 - Attachment is obsolete: true
Comment on attachment 9013592 [details]
Bug 1370636 - [wdspec] Update mozharness configuration for geckodriver on Windows. r?jgraham

James Graham [:jgraham] has approved the revision.
Attachment #9013592 - Flags: review+
Comment on attachment 9013593 [details]
Bug 1370636 - [wdspec] Temporarily disable failing tests on Windows. r?jgraham

James Graham [:jgraham] has approved the revision.
Attachment #9013593 - Flags: review+
Pushed by
[wdspec] Update mozharness configuration for geckodriver on Windows. r=jgraham
Pushed by
[wdspec] Temporarily disable failing tests on Windows. r=jgraham
Pushed by
[wdspec] Enable Wd jobs on Windows platforms. r=jgraham
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
You need to log in before you can comment on or make changes to this bug.