1439370 - Make our Selenium tests check for JS exceptions

Status: RESOLVED INVALID

(Tree Management :: Treeherder: Frontend, enhancement, P3)

Description

[Ed Morley [:emorley]]

Currently if any unhandled JS exceptions occur during our Selenium tests, they don't cause the test to fail.

This means we can miss exceptions in loaded but untested page components.

I think Will looked into this in the past and found it to be non-trivial, however it seems there may now be better support:
https://watirmelon.blog/2016/06/29/using-webdriver-to-automatically-check-for-javascript-errors-on-every-page-2016-edition/

Dave, do you have any recommendations as to how we should do this? :-)

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The referenced article is using logging preferences, which are not part of the WebDriver specification, and therefore not available to Firefox via Marionette. We already have bug 1339614 for adding the ability to retrieve the browser logs via Marionette, however we'd also need to add a way for the Selenium client(s) to request them.

Comment 2

[Ed Morley [:emorley]]

Ah thank you for the explanation. So it sounds like this isn't easily do-able at present, unless we switch to running our tests in Chrome?

Comment 3

[Ed Morley [:emorley]]

It's worth noting that Jest picks up any console errors that occur whilst the tests ran, so we will at least have coverage of this from the unit test side. Perhaps another reason for leaning towards more new unit tests and not expanding our Selenium test coverage any further in the short-medium term (given we're currently very much weighted towards E2E tests at the moment).

Comment 4

[Ed Morley [:emorley]]

I noticed that geckodriver 0.24.0 added support for devtools.console.stdout.content, which I believe means console logging could be sent to stdout. Whilst this wouldn't make the jobs fail, it would at least make it easier to spot errors in test output.

Is this something that pytest-selenium could/should support through a built-in option, or do you see it as more appropriate for us to manually set devtools.console.stdout.content?

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ed Morley [:emorley] from comment #4)

I noticed that geckodriver 0.24.0 added support for devtools.console.stdout.content, which I believe means console logging could be sent to stdout. Whilst this wouldn't make the jobs fail, it would at least make it easier to spot errors in test output.

Is this something that pytest-selenium could/should support through a built-in option, or do you see it as more appropriate for us to manually set devtools.console.stdout.content?

pytest-selenium supports setting preferences using a firefox_options fixture, such as:

import pytest
@pytest.fixture
def firefox_options(firefox_options):
    firefox_options.set_preference('devtools.console.stdout.content', True)
    return firefox_options

This could be set in conftest.py to apply to all tests. I just tested extending the driver_log fixture for a crude check for errors, and it seems to work:

@pytest.fixture
def driver_log(driver_log):
    yield driver_log
    with open(driver_log) as f:
        js_errors = [l for l in f.readlines() if "JavaScript error" in l]
    assert js_errors == []
 return firefox_options

Comment 6

[Ed Morley [:emorley]]

Hi thank you for those - very helpful!

Trying them out I get test failures from browser chrome exceptions, eg:

JavaScript error: jar:file:///home/vagrant/firefox/omni.ja!/components/nsUrlClassifierListManager.js, line 597: TypeError: this.tablesData[table] is undefined

JavaScript error: resource://gre/modules/osfile/osfile_async_front.jsm, line 404: Error: OS.File has been shut down....://gre/modules/osfile/osfile_async_front.jsm, line 404: Error: OS.File has been shut down. Rejecting post to remove

These aren't due to the preference change (since that pref is supposed to only affect content not chrome, and they still occur if I don't use the modified firefox_options fixture), but have presumably always been in the driver log.

Other than manually filtering out any exceptions that include jar:file:/// or resource://, do you have any thoughts on how best to suppress these?

Also, I'm presuming they are bugs in the browser/geckodriver that should be reported?

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I wonder if geckodriver supports redirecting the content logs elsewhere, so we can check them independently of the chrome output. I haven't been involved in this area for a while. Henrik, do you know if this is possible?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Using devtools.console.stdout.content is all you can do at the moment. As Ed already mentioned any other chrome related JS error/warning is not related to that setting, and is present in each log regardless other settings. Hiding those chrome related output is part of bug 1399441.

Comment 9

[Sarah Clements [:sclements]]

We've moved away from selenium - see bug 1576966.