1715900 - Investigate adding a test-suite which we can use to verify if interventions are still needed (or break)

Status: RESOLVED FIXED

(Web Compatibility :: Interventions, defect)

Description

[Thomas Wisniewski [:twisniewski]]

To reduce burden on QA, we would like to add a test-suite for the webcompat addon, which can essentially check which interventions are still required, and which may have changed (whether they may have been fixed, obsoleted due to the site changing, or even worsened).

After some preliminary investigations I have found that a Marionette-driven test-suite should be feasible in the short term. It does not have to actually run in CI, but rather would be run by someone on the web compat team at least once a release cycle.

This patch I will attach to this bug is an example of how the beginnings of such a suite might look. As the code would be in moz-central, it would be relatively straightforward for testers to make an artifact build and run the suite with mach marionette-test --headless browser/extensions/webcompat/tests/marionette/ or even run specific tests. As we already have some tests in-tree, I feel it would be good to keep them all in-tree if possible.

Note that there are some open questions. Most pertinently:

1. can we improve how easy it is to run these, perhaps allowing folks to trigger it on a one-time basis to run on automation?
2. how will we handle secret information (since it clearly cannot go in moz-central)
3. can we extend this method to handle odd cases such as sites requiring VPNs?

Right now however, it at least shows that this method is feasible without being too demanding on the folks creating or running the tests. It seems like that it would also be straightforward for us to port the tests to a different test harness than Marionette, should we wish to do so later.

Right now only two simple interventions are tested in the patches, but looking over the Marionette driver API, it seems likely that we can test all of our current interventions once question 2 above is addressed.

Comment 1

[Thomas Wisniewski [:twisniewski]]

Attachment: WIP: Bug 1715900 - Add a preliminary in-tree test suite to verify webcompat interventions

Comment 2

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Make wpt webdriver library available to mach commands

Comment 3

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Add mach test-interventions command for testing webcompat interventions

This adds pytest-based infrastructure for running webcompat
interventions tests. The tests are written in Python using WebDriver
to control a Firefox instance, either with the webcompat interventions
enabled or disabled.

Each test is a function with a name like test_foo. The convention is
one test directory per intervention bug, and a test file matching the
test_*.py pattern. Each test function must be marked as either being
for the configuration with interventions enabled or with them
disabled, using the decorators @pytest.mark.with_interventions or
@pytest.mark.without_interventions. The directory naming convention
makes it possible to run the tests for a single bug, and a --bug
command line argument is provided for this.

Currently the WebDriver library used is that from
web-platform-tests. However this is light on functionality and under
documented, so we should probably look at vendoring selenium or some
other full featured library.

There are also various other missing features e.g. we don't do
anything with the result status, and don't write the results out into
any kind of machine-readable form.

Comment 4

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Add tier 3 interventions job on taskcluster

Comment 5

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Add some sample interventions tests

Comment 6

[James Graham [:jgraham]]

I've added a prototype based on pytest, using the web-platform-tests WebDriver library. I don't think that's the right approach in the long term; we probably want to figure out how to vendor selenium or similar, since that will already have useful helper functions and so on. It's also set up to run on try; there's an example push at [1].

To run the tests locally one uses ./mach test-interventions. To run on try use mach try fuzzy -q webcompat (but we could also add a preset). Note the job is Tier 3 so not visible by default.

[1] https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=HduzEBpqTMmQ95M5KiEaEQ.0&tier=1%2C2%2C3&revision=b9185fe0658521dbdbe72f17a8e5910932100e4e

Comment 7

[Thomas Wisniewski [:twisniewski]]

Attachment: WIP: Bug 1715900 - Add ability to specify logins in a JSON file and add a sample test using logins

Depends on D118499

Comment 8

[James Graham [:jgraham]]

Attachment: Bug 1715900 - Run all with[out] interventions tests in a single session

Invoke pytest twice, once with interventions enabled and once with it
disabled. This allows us to run all the tests for each invocation in a
single browser session, rather than restarting the browser between
each test. This makes things much faster.

Comment 9

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Remove use of functools.cache for Python 3.6+ support

Comment 10

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Convert tests to use Selenium

This is better documented than the in-tree WebDriver library and
probably easier for people to use.

Selenium is installed for PyPI; it's assumed this is OK since we
aren't intending to run the tests on integration branches in the near
future, and site changes are a bigger cause of instability than PyPI
downtime.

This also removes some of the helpers that are provided as APIs in
Selenium and rewrites a couple of others to be clearer. In particular
the [try_]find_element[s] are replaced by find_element[s], which takes
a Selector object which is (currently) either Css, Xpath, or
Text. This allows defining a selector in one place and reusing it, and
avoids having an API that allows passing in multiple
selectors. Instead of the try_ variants we allow specifying a default
value if no element is found (although maybe this could be simplified further).

Comment 11

[James Graham [:jgraham]]

Attachment: WIP: Bug 1715900 - Fix bug argument

Comment 12

[Thomas Wisniewski [:twisniewski]]

Attachment: Bug 1715900 - Bump urllib3 to version 1.26.0, boto3 to 1.16.63, and botocore to 1.19.63; r?mhentges

Comment 13

[Thomas Wisniewski [:twisniewski]]

Attachment: Bug 1715900 - Add a mach test-interventions command for testing webcompat interventions; r?jgraham,mhentges

Depends on D138383

Comment 14

[Thomas Wisniewski [:twisniewski]]

Attachment: Bug 1715900 - Add initial tests (and helpers) for the mach test-interventions command; r?jgraham

Depends on D138384

Comment 15

[Thomas Wisniewski [:twisniewski]]

Attachment: Bug 1715900 - Add initial tests (and helpers) for the mach test-interventions command; r?jgraham

Depends on D138384

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11c95163a2f6
Bump urllib3 to version 1.26.0; r=mhentges
https://hg.mozilla.org/integration/autoland/rev/2355ea418b97
Add a mach test-interventions command for testing webcompat interventions; r=jgraham,mhentges
https://hg.mozilla.org/integration/autoland/rev/c1bad2dbd960
Add initial tests (and helpers) for the mach test-interventions command; r=jgraham

Comment 17

[Atila Butkovits]

Backed out for causing Python failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/9c10853c6a5216191d6657e7422671c92286d3df

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=B5xwP8sdSCmsy9hElBUS-A.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=c1bad2dbd960df00dd43fec22b2f39ebf3fd3454

Failure log:

• https://treeherder.mozilla.org/logviewer?job_id=367597453&repo=autoland&lineNumber=619
• https://treeherder.mozilla.org/logviewer?job_id=367600951&repo=autoland&lineNumber=384

Comment 18

[Thomas Wisniewski [:twisniewski]]

@mhentges, it seems like we have a requirements conflict:

From log 1: [task 2022-02-11T18:26:17.133Z] python/mach/mach/test/test_site_compatibility.py::test_sites_compatible TEST-UNEXPECTED-FAIL
From log 2: [task 2022-02-11T18:46:25.758Z] botocore 1.18.6 has requirement urllib3<1.26,>=1.20; python_version != "3.4", but you have urllib3 1.26.0.

What should we do about this?

Comment 19

[Mitchell Hentges [:mhentges] 🦀]

botocore 1.18.6 has requirement urllib3<1.26,>=1.20; python_version != "3.4", but you have urllib3 1.26.0.

Looks like changing tools/moztreedocs/requirements.in such that:

• boto3==1.16.63
• botocore==1.19.63

Should resolve this issue.
Note that when re-generating requirements.txt, you'll want to use Python 3.9 on Linux.
If necessary, I can do that locally and send you a patch, but I'd prefer if you're able to do it locally.

---

python/mach/mach/test/test_site_compatibility.py::test_sites_compatible TEST-UNEXPECTED-FAIL

Hmm, it looks like the issue is that our docker images for python tests use Python 3.6.9, but selenium>=4.0.0 requires Python 3.7+.
There's an issue here to track updating the CI python version.
In the meantime, I'd recommend going back to selenium==3.141.0, if possible.

Comment 20

[Thomas Wisniewski [:twisniewski]]

https://treeherder.mozilla.org/jobs?repo=try&revision=5b1b6f79c71e7583a233f5ef83f1cf93f6129671

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8db101a97937
Bump urllib3 to version 1.26.0, boto3 to 1.16.63, and botocore to 1.19.63; r=mhentges
https://hg.mozilla.org/integration/autoland/rev/d2bc4e004628
Add a mach test-interventions command for testing webcompat interventions; r=jgraham,mhentges
https://hg.mozilla.org/integration/autoland/rev/2deb7e3906b5
Add initial tests (and helpers) for the mach test-interventions command; r=jgraham

Comment 22

[Thomas Wisniewski [:twisniewski]]

Just a few notes on how I updated urllib and boto vendoring, in case they help others down the line;

1. set up python 3.6.11 via pyenv on my Big Sur Intel Macbook (using homebrew).
   • brew install pyenv
   • pyenv install 3.6.11
   • pyenv global 3.6.11
   • add some lines [1] to my ~/.zshrc
   • open a new terminal window, confirm the correct python is active with python -v.
2. update the urllib requirement:
   • updated third_party/python/requirements.in (urllib==1.26)
   • mach vendor python
   • hg diff to confirm the results
3. set up Python 3.9.10, as above in step 1.
4. to update the boto(tools) requirement:
   • updated tools/moztreedocs/requirements.in (boto3==1.16.63 and botocore==1.19.63)
   • cd moz-central/tools/moztreedocs
   • mach python -m piptools compile --generate-hashes --output-file=requirements.txt requirements.in
   • hg diff to confirm requirements.txt was correctly changed

[1] changes to .zshrc:

export PATH="$HOME/.pyenv/bin:$PATH"
export PATH="/usr/local/bin:$PATH"

export LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include -I/usr/local/opt/bzip2/include"

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init --path)"
  eval "$(pyenv virtualenv-init -)"
fi

Comment 23

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/8db101a97937
https://hg.mozilla.org/mozilla-central/rev/d2bc4e004628
https://hg.mozilla.org/mozilla-central/rev/2deb7e3906b5