Closed Bug 1414401 Opened 7 years ago Closed 7 years ago

Enable building on Linux i686

Categories

(Testing :: geckodriver, enhancement)

Product:
Testing
Component:
geckodriver
Version:
Version 3
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox59 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla59
Tracking Flags:
Tracking Status
firefox59 --- fixed

People

(Reporter: ato, Assigned: ato)

References

Details

Attachments

(3 files)

Bug 1414401 - Enable geckodriver on Linux i686
59 bytes, text/x-review-board-request
whimboo
: review+
ted
: review+
Details
Bug 1414401 - Schedule Wd tests on linux32
59 bytes, text/x-review-board-request
jgraham
: review+
Details
Bug 1414401 - Disable key_shortcuts.py on Linux i686
59 bytes, text/x-review-board-request
ato
: review+
Details 
We also want to compile geckodriver on Linux i686, and indeed all
of our target platforms.  In the past we had some issues to do with
cross-compiling Rust code on Taskcluster [1], but they appear to no
longer be present [2].

  [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1367519
  [2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b34de9371a1a1f731396b00362ee2f458cec0ed7&group_state=expanded
Assignee: nobody → ato
Status: NEW → ASSIGNED
We should probably also enable the Wd job on Linux i686.  How do I do that?
Flags: needinfo?(james)
(In reply to Andreas Tolfsen ‹:ato› from comment #2)
> We should probably also enable the Wd job on Linux i686.  How do I do that?

See: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml

CC'ing Joel for info.
Comment on attachment 8925139 [details]
Bug 1414401 - Enable geckodriver on Linux i686

https://reviewboard.mozilla.org/r/196390/#review201858

r=me with at least wdspec tests added.
Attachment #8925139 - Flags: review?(hskupin) → review+
diff --git a/taskcluster/ci/test/test-sets.yml b/taskcluster/ci/test/test-sets.yml
index 52ace398d2b1..8bb5475321d3 100644
--- a/taskcluster/ci/test/test-sets.yml
+++ b/taskcluster/ci/test/test-sets.yml
@@ -296,6 +296,7 @@ linux32-tests:
     - reftest-no-accel
     - web-platform-tests
     - web-platform-tests-reftests
+    - web-platform-tests-wdspec
     - xpcshell
 
 linux32-opt-tests:
Flags: needinfo?(james)
Comment on attachment 8925139 [details]
Bug 1414401 - Enable geckodriver on Linux i686

https://reviewboard.mozilla.org/r/196390/#review202398
Attachment #8925139 - Flags: review?(ted) → review+
In the latest fixup I have enabled the Wd job for linux32, but due
to the 57 CI freeze I can’t push this changeset immediately.  I
understand the easiest thing to do is just wait for the change
freeze end.
Comment on attachment 8926032 [details]
Bug 1414401 - Schedule Wd tests on linux32

https://reviewboard.mozilla.org/r/197260/#review202410
Attachment #8926032 - Flags: review?(james) → review+
I think that's stupid. This rides the trains and is no more dangerous than any other random change that isn't part of the freeze.
I would just email the mailing list and if there are no concerns in 24 hours that cannot be resolved, we wait- otherwise this is landed in <48 hours.
Given that this is the process, it is easier to just wait landing
this a week or two than having to send an email to a mailing list
and get approval.
try run looks good.  I spoke to jgraham and I don’t really see
any reason to enable the Wd job on linux32 debug.  We never use
debug when diagnosing problems, and we already run them on Linux x64
debug.
(In reply to Andreas Tolfsen ‹:ato› from comment #15)
> any reason to enable the Wd job on linux32 debug.  We never use
> debug when diagnosing problems, and we already run them on Linux x64

Who is 'we' here? When I'm diagnosing test failures I always look out for debug builds first, because those indeed give a way better insight of what might have gone wrong with Firefox. Those will also show assertions, which indicate possible crashes in opt builds. So I would strongly vote for enabling opt and debug.
(In reply to Henrik Skupin (:whimboo) from comment #16)

> (In reply to Andreas Tolfsen ‹:ato› from comment #15)
> 
> > any reason to enable the Wd job on linux32 debug.  We never use
> > debug when diagnosing problems, and we already run them on Linux
> > x64
> 
> Who is 'we' here? When I'm diagnosing test failures I always
> look out for debug builds first, because those indeed give a way
> better insight of what might have gone wrong with Firefox. Those
> will also show assertions, which indicate possible crashes in opt
> builds. So I would strongly vote for enabling opt and debug.

Debug builds are useful when diagnosing C and C++ problems, and
almost never useful for diagnosing Marionette problems because
it is written in JS.  Keep in mind that Wd tests geckodriver and
Marionette; not Firefox.
(In reply to Andreas Tolfsen ‹:ato› from comment #17)
> Debug builds are useful when diagnosing C and C++ problems, and
> almost never useful for diagnosing Marionette problems because
> it is written in JS.  Keep in mind that Wd tests geckodriver and
> Marionette; not Firefox.

This is not fully true. Keep in mind that Marionette unit tests also only test Marionette client and server.

Every command as run via Marionette can cause an unexpected behavior of Firefox. Not all the failures are obvious when running an opt build of Firefox. Some would need a debug build, as I explained above.

Please also note that I usually go through all intermittent failures first, and know what I'm talking about. If you really want to force not running any debug build, I would suggest you take over that work for a while. At least for those cases when a problem is not obvious because no debug build exist and shows this problem.
There are no objections so let's go ahead and land this.
(In reply to Henrik Skupin (:whimboo) from comment #18)

> (In reply to Andreas Tolfsen ‹:ato› from comment #17)
> 
> Every command as run via Marionette can cause an unexpected
> behavior of Firefox. Not all the failures are obvious when running
> an opt build of Firefox. Some would need a debug build, as I
> explained above.
> 
> Please also note that I usually go through all intermittent
> failures first, and know what I'm talking about. If you really
> want to force not running any debug build, I would suggest you
> take over that work for a while. At least for those cases when a
> problem is not obvious because no debug build exist and shows this
> problem.

This level of hostility is unnecessary: I am not trying to force
anything.  If you find running Wd on linux32 debug useful, we should
of course do that.

This will add roughly 30 minutes extra runtime with the current test
suite size, which I believe is acceptable.  Fortunately jgraham
already included this in his intent email.
Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/34708561a45c
Enable geckodriver on Linux i686 r=ted,whimboo
https://hg.mozilla.org/integration/autoland/rev/331c5f0d914e
Schedule Wd tests on linux32 r=jgraham
Backed out for almost perma failing webdriver's /webdriver/tests/actions/key_shortcuts.py on Linux 32-bit:

https://hg.mozilla.org/integration/autoland/rev/06c7f50b73aead72a059557d20acfc0c666d6b37

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=331c5f0d914ed183b52e5dcee72f8f31281cdde0&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=retry
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=143813937&repo=autoland

[task 2017-11-10T18:36:27.304Z] 18:36:27     INFO - TEST-START | /webdriver/tests/contexts/maximize_window.py
[task 2017-11-10T18:36:27.305Z] 18:36:27     INFO - TEST-SKIP | /webdriver/tests/contexts/maximize_window.py | took 0ms
[task 2017-11-10T18:36:27.323Z] 18:36:27     INFO - Starting runner
[task 2017-11-10T18:36:27.328Z] 18:36:27     INFO - Starting http server on 127.0.0.1:8000
[task 2017-11-10T18:36:27.329Z] 18:36:27     INFO - Starting http server on 127.0.0.1:8001
[task 2017-11-10T18:36:27.330Z] 18:36:27     INFO - Starting http server on 127.0.0.1:8443
[task 2017-11-10T18:36:27.348Z] 18:36:27     INFO - PID 850 | 1510338987345	geckodriver	INFO	geckodriver 0.19.1 ( '331c5f0d914e'  '2017-11-07 10:48 -0800')
[task 2017-11-10T18:36:27.352Z] 18:36:27     INFO - PID 850 | 1510338987349	geckodriver	INFO	Listening on 127.0.0.1:4444
[task 2017-11-10T18:36:27.852Z] 18:36:27     INFO - WebDriver HTTP server listening at http://127.0.0.1:4444/
[task 2017-11-10T18:36:27.853Z] 18:36:27     INFO - TEST-START | /webdriver/tests/actions/key_shortcuts.py
[task 2017-11-10T18:36:28.178Z] 18:36:28     INFO - STDOUT: ============================= test session starts ==============================
[task 2017-11-10T18:36:28.179Z] 18:36:28     INFO - STDOUT: platform linux2 -- Python 2.7.12, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /builds/worker/workspace/build/venv/bin/python
[task 2017-11-10T18:36:28.180Z] 18:36:28     INFO - STDOUT: rootdir: /builds/worker/workspace/build/tests/web-platform/tests/webdriver/tests/actions, inifile: 
[task 2017-11-10T18:36:28.181Z] 18:36:28     INFO - STDOUT: collecting ... 
[task 2017-11-10T18:36:28.197Z] 18:36:28     INFO - STDOUT: collected 3 items
[task 2017-11-10T18:36:28.197Z] 18:36:28     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/actions/key_shortcuts.py::test_mod_a_and_backspace_deletes_all_text 
[task 2017-11-10T18:36:28.198Z] 18:36:28     INFO - PID 850 | 1510338988194	mozrunner::runner	INFO	Running command: "/builds/worker/workspace/build/application/firefox/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.CcAw6f0HOq1q"
[task 2017-11-10T18:36:55.183Z] 18:36:55     INFO - PID 850 | 1510339015179	Marionette	INFO	Enabled via --marionette
[task 2017-11-10T18:36:56.128Z] 18:36:56     INFO - PID 850 | GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.
[task 2017-11-10T18:36:57.366Z] 18:36:57     INFO - PID 850 | 1510339017360	Marionette	INFO	Listening on port 2828
[task 2017-11-10T18:36:57.564Z] 18:36:57     INFO - PID 850 | 1510339017557	Marionette	DEBUG	Register listener.js for window 2147483649
[task 2017-11-10T18:36:57.661Z] 18:36:57     INFO - PID 850 | 1510339017648	Marionette	DEBUG	Received DOM event "beforeunload" for "about:blank"
[task 2017-11-10T18:36:57.699Z] 18:36:57     INFO - PID 850 | 1510339017688	Marionette	DEBUG	Received DOM event "pagehide" for "about:blank"
[task 2017-11-10T18:36:57.716Z] 18:36:57     INFO - PID 850 | 1510339017701	Marionette	DEBUG	Received DOM event "DOMContentLoaded" for "http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html"
[task 2017-11-10T18:36:57.733Z] 18:36:57     INFO - PID 850 | 1510339017720	Marionette	DEBUG	Received DOM event "pageshow" for "http://web-platform.test:8000/webdriver/tests/actions/support/test_actions_wdspec.html"
[task 2017-11-10T18:36:57.870Z] 18:36:57     INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/actions/key_shortcuts.py | expected OK
[task 2017-11-10T18:36:57.871Z] 18:36:57     INFO - TEST-INFO took 30017ms
Flags: needinfo?(ato)
For some reason key_shortcuts.py, which was landed between the time
this test was written and when it landed on m-c, takes a long time
on linux32.
Flags: needinfo?(ato)
Attachment #8927614 - Flags: review?(ato)
Comment on attachment 8927614 [details]
Bug 1414401 - Disable key_shortcuts.py on Linux i686

https://reviewboard.mozilla.org/r/198906/#review203926
Attachment #8927614 - Flags: review?(ato) → review+
Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cb2f1791b209
Enable geckodriver on Linux i686 r=ted,whimboo
https://hg.mozilla.org/integration/autoland/rev/b445d7d2eadc
Schedule Wd tests on linux32 r=jgraham
https://hg.mozilla.org/integration/autoland/rev/8d17c033d79b
Increase key shortcuts timeout. r=ato
Backed out for failing Linux opt  /webdriver/tests/actions/key_shortcuts.py

https://hg.mozilla.org/integration/autoland/rev/cdcac626d2bad3fefa894517c9d6f80982f79eb1

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=8d17c033d79b2a1669f150ed2044f6bd5bebb2d6&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=retry

Failure log:
https://treeherder.mozilla.org/logviewer.html#?job_id=144071650&repo=autoland&lineNumber=1223


[task 2017-11-12T09:48:05.200Z] 09:48:05     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/actions/key_shortcuts.py::test_mod_a_and_backspace_deletes_all_text 
[task 2017-11-12T09:48:05.205Z] 09:48:05     INFO - PID 853 | 1510480085198	mozrunner::runner	INFO	Running command: "/builds/worker/workspace/build/application/firefox/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.kZLkg95VbI8r"
[task 2017-11-12T09:48:33.012Z] 09:48:33     INFO - PID 853 | 1510480113004	Marionette	INFO	Enabled via --marionette
[task 2017-11-12T09:48:33.788Z] 09:48:33     INFO - PID 853 | GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.
[task 2017-11-12T09:48:34.699Z] 09:48:34     INFO - PID 853 | 1510480114696	Marionette	INFO	Listening on port 2828
[task 2017-11-12T09:48:34.845Z] 09:48:34     INFO - TEST-UNEXPECTED-TIMEOUT | /webdriver/tests/actions/key_shortcuts.py | expected OK
[task 2017-11-12T09:48:34.845Z] 09:48:34     INFO - TEST-INFO took 30021ms
Flags: needinfo?(ato)
Not sure why /webdriver/tests/actions/key_shortcuts.py keeps timing
out when I send this patch off to autoland.  I’ve triggered a
fresh try run [1] to see if really only happens on autoland.

From the log above [2] it looks like it takes nearly 30 seconds to
start Firefox and before Marionettes tarts listening, at which point
the test is interrupted by the test runner because it took too long.

  [1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=8b37076f6c45
  [2] https://treeherder.mozilla.org/logviewer.html#?job_id=144071650&repo=autoland&lineNumber=1223
Flags: needinfo?(ato)
Maybe if you cannot get it investigated, would changing the mozharness configs help by adding `--webdriver-args=-vv`? It would be good to see debug info from Firefox.
(In reply to Andreas Tolfsen ‹:ato› from comment #33)

> Not sure why /webdriver/tests/actions/key_shortcuts.py keeps
> timing out when I send this patch off to autoland.  I’ve
> triggered a fresh try run [1] to see if really only happens on
> autoland.

Good news is that it appears to reproduce on try.  Bad news is that
I don’t know what is causing it yet.

(In reply to Henrik Skupin (:whimboo) from comment #34)

> Maybe if you cannot get it investigated, would changing the
> mozharness configs help by adding `--webdriver-args=-vv`? It would
> be good to see debug info from Firefox.

Yes, that seems like a good idea.  Can you point me to the file I
need to change?
Flags: needinfo?(hskupin)
(In reply to Andreas Tolfsen ‹:ato› from comment #35)
> Yes, that seems like a good idea.  Can you point me to the file I
> need to change?

Seems like there is no option included for this in the mozharness script, so you will have to add it directly to that file:

https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/web_platform_tests.py?q=file%3Aweb_platform_tests.py&redirect_type=single#223
Flags: needinfo?(hskupin)
I looked at the usage statistics of geckodriver on Linux i686 and
it’s around 1%.  Instead of figuring out the root cause here I am
going to try disabling the test because I can’t justify spending
any more time on this.

If the same problem persists when I enable the Wd job on macOS we
will know it is a more systemic issue that needs looking into.
Pushed by atolfsen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/40eff3779e41
Enable geckodriver on Linux i686 r=ted,whimboo
https://hg.mozilla.org/integration/autoland/rev/ea595e5494e1
Schedule Wd tests on linux32 r=jgraham
https://hg.mozilla.org/integration/autoland/rev/33f727c87866
Disable key_shortcuts.py on Linux i686 r=ato
Blocks: 1417649
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: