Closed Bug 1370636 Opened 7 years ago Closed 6 years ago

Enable Wd job on Windows platforms

Categories

(Testing :: geckodriver, enhancement, P1)

Product:
Testing
Component:
geckodriver
Version:
Version 3
Platform:
All
Windows
Type:
enhancement

Tracking

(firefox64 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla64
Tracking Flags:
Tracking Status
firefox64 --- fixed

People

(Reporter: ato, Assigned: whimboo)

References

(Blocks 1 open bug)

Details

Attachments

(3 files, 4 obsolete files)

Bug 1370636 - [wdspec] Update mozharness configuration for geckodriver on Windows. r?jgraham
46 bytes, text/x-phabricator-request
jgraham
: review+
Details | Review
Bug 1370636 - [wdspec] Temporarily disable failing tests on Windows. r?jgraham
46 bytes, text/x-phabricator-request
jgraham
: review+
Details | Review
Bug 1370636 - [wdspec] Enable Wd jobs on Windows platforms. r?jgraham
46 bytes, text/x-phabricator-request
jgraham
: review+
Details | Review 
We currently compile geckodriver on Windows, but we are not running any tests.  We need to enable the Wd job on Windows platforms.
Depends on: 1368264
Via bug 1367477 I added the trychooser entry for wdspec tests, which should be available soon.
Depends on: 1367477
Oh, it should have been more an info than a dependency. Reverting it.
No longer depends on: 1367477
Priority: -- → P1
Blocks: 1417649
https://bugzil.la/1374263 made sure the geckodriver binary was
included in the correct resource file on macOS.  To enable running
the Wd job (WPT WebDriver tests) on Windows on try we need to
achieve the same effect there.

nfroyd, do you have any pointers what files we need to tickle for
this?
Flags: needinfo?(nfroyd)
Probably browser/installer/package-manifest.in; searching for "MOZ_UPDATER" or "MOZ_CRASHREPORTER" or "pingsender" should provide enough examples to cargo-cult from?
Flags: needinfo?(nfroyd)
Priority: P1 → P3
I do see the geckodriver executable as being part of the target.common.tests.zip archive. So this should be basically all what we need to get the Wd jobs running on Windows. I will have a look if we can make that happen given that it drives me nuts we only run the tests on Linux.
I got the job created via Taskcluster on Windows, and also updated the mozharness configs for the correct location of the geckodriver.exe. But starting geckodriver.exe fails because VCRUNTIME140.DLL is missing on the AMI.

Rob or Peete could one of you have a look at this?
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Flags: needinfo?(rthijssen)
Flags: needinfo?(pmoore)
Priority: P3 → P1
I triggered some more jobs. Maybe this only affects Windows 7 but not Windows 10 AMIs.
there are a few instances of vcruntime140.dll available on the gecko-1-b-win2012 worker type.

here's a task that lists them all (using `C: && cd C:\ && dir vcruntime140.dll /s`):

https://tools.taskcluster.net/groups/U2ePZVLXTDWZowA6YE2NcA

https://taskcluster-artifacts.net/U2ePZVLXTDWZowA6YE2NcA/0/public/logs/live_backing.log
Flags: needinfo?(rthijssen)
Rob, that's a build machine but not a test machine. In my case the worker type is gecko-t-win7-32.
Flags: needinfo?(rthijssen)
reran on win 7 & 10.
looks like it's not available on the win 7 workers but it is on the win 10
this is how we install it on windows 2012:
https://github.com/mozilla-releng/OpenCloudConfig/blob/8a69ab1/userdata/Manifest/gecko-1-b-win2012.json#L388-L406

we could try something similar on windows 7 but i have no idea if it will work.

note that the 32 bit installer (vcredist_vs2015_x86) creates: C:\Windows\SysWOW64\vcruntime140.dll
while the 64 bit installer (vcredist_vs2015_x64) creates: C:\Windows\System32\vcruntime140.dll

which suggests to me that there will be hurdles installing this on a 32 bit system.
Flags: needinfo?(rthijssen)
Please note that the same problem is also visible on Windows 10.
Flags: needinfo?(pmoore)
testing vcredist install on beta workers (gecko-t-win7-32-beta, gecko-t-win10-64-beta)
https://github.com/mozilla-releng/OpenCloudConfig/commit/dbb53cca3bb19e3328b447c43c5215c27d366408
Depends on: 1460042
Now it works better but we get dozen of `Test harness output was not a valid structured log message` failures:

https://treeherder.mozilla.org/logviewer.html#?job_id=177621642&repo=try

> 08:49:21 CRITICAL - platform win32 -- Python 2.7.14, pytest-unknown, py-1.5.2, pluggy-0.5.3.dev -- Z:\task_1525854760\build\venv\Scripts\python.exe

I assume this is pytest, which is just logging to stdout via print?

But I do not understand the following:

> 08:49:21 CRITICAL - tests/web-platform/tests/infrastructure/webdriver/tests/test_load_file.py::test_load {"source": "web-platform-tests", "thread": "Thread-TestrunnerManager-1", "time": 1525855761591, "action": "log", "message": "Got command: u'log'", "level": "DEBUG", "pid": 4512}

This is coming from TestrunnerManager and fails for the message `Got command:` which is logged via:

> self.logger.debug("Got command: %r" % command)

Maybe it is because the message doesn't contain a log level?

> {"source": "web-platform-tests", "test": "/infrastructure/webdriver/tests/test_load_file.py", "thread": "Thread-TestrunnerManager-1", "time": 1525855761129, "action": "test_start", "pid": 4512}

James could you please help?
Flags: needinfo?(james)
Note that the reference without the log level comes from the raw.log while the reference with the level is from the log viewer.
Btw. the only reference I was able to find for such a failure message is in the `StructuredOutputParser` of mozharness and it seems that the action as contained in the log line is not valid:

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/structuredlog.py?q=%22Test+harness+output+was+not+a+valid+structured+log+message%22&redirect_type=single#70-72

For the above cases it would mean "log" and "test_start", which seem to be fine? Also why does that only happen on Windows but not Linux?
We should be wrapping the output from pytest so that it ends up in the logger rather than going directly to stdout. I guess there is a Windows/Linux difference here; one thought that comes to mind is that it could be caused by the difference between multiprocessing on the two platforms (on Linux Python attempts to share memory in a copy-on-write sense, on Windows it sensibly doesn't).
Flags: needinfo?(james)
Attachment #8974032 - Attachment is obsolete: true
Attachment #8974033 - Attachment is obsolete: true
MozReview-Commit-ID: BlFMMXt8WBU
MozReview-Commit-ID: FgwitlstFP7
Blocks: 1489130
No longer blocks: 1417649
James, as promised here is a current wdspec test job which shows the structured log problem:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c017fb9ec65c1b21aee317945eb76be5a23ddcc7&selectedJob=201636819

It would be great if we could have a look at this today.
Flags: needinfo?(james)
Depends on: 1495007
Blocks: 1495513
Blocks: 1495521
With bug  1495007 fixed we are now able to enable the Wd test jobs on Windows. Here is a try build from James with the patch on the other patch applied:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c9cdf6610527685eb6e07e6bbcf8dff5354807d

The two tests which are currently failing will be disabled on Windows.

Also once the tests are running on Windows, we will go ahead and also enable headless tests on bug 1493907.
Blocks: 1493907
Blocks: 1493948
Ok, so I pushed another try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=499458b7919c489039f0515dc25dd18c9277ce70&selectedJob=202696459

Sadly I'm not able to disable those affected tests. James, mind having a look at those disable statements of the manifest files? I wonder why it doesn't work: https://hg.mozilla.org/try/rev/a9ae1b9c5abd18d6f9a339d2333815682fca2c08

I actually found this logic in some other manifest files so I wonder if that is Wd only, or doesn't work at all.

Also please ignore the trailing space after the sub test name in the first manifest. It's not the problem here given the other test is also not getting disabled. Thanks.
The patch makes sure that the correct path to the geckodriver binary
is used for the web-platform-tests script.
We solved the manifest question via IRC. It's all working now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b34cd477af88c4b69ebe8023cd68a2af453c1ca
Flags: needinfo?(james)
Comment on attachment 9013594 [details]
Bug 1370636 - [wdspec] Enable Wd jobs on Windows platforms. r?jgraham

James Graham [:jgraham] has approved the revision.
Attachment #9013594 - Flags: review+
Attachment #9006455 - Attachment is obsolete: true
Attachment #9006456 - Attachment is obsolete: true
Comment on attachment 9013592 [details]
Bug 1370636 - [wdspec] Update mozharness configuration for geckodriver on Windows. r?jgraham

James Graham [:jgraham] has approved the revision.
Attachment #9013592 - Flags: review+
Comment on attachment 9013593 [details]
Bug 1370636 - [wdspec] Temporarily disable failing tests on Windows. r?jgraham

James Graham [:jgraham] has approved the revision.
Attachment #9013593 - Flags: review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ec024b2fc13a
[wdspec] Update mozharness configuration for geckodriver on Windows. r=jgraham
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9e11b103f7ec
[wdspec] Temporarily disable failing tests on Windows. r=jgraham
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b9a19eb6f805
[wdspec] Enable Wd jobs on Windows platforms. r=jgraham
https://hg.mozilla.org/mozilla-central/rev/ec024b2fc13a
https://hg.mozilla.org/mozilla-central/rev/9e11b103f7ec
https://hg.mozilla.org/mozilla-central/rev/b9a19eb6f805
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox64: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: