1438469 - Intermittent Selenium test_filter_jobs_by_failure_result "AssertionError: assert '' == 'exception'"

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

I've seen this a handful of times on Travis:


$ pytest tests/selenium/ --driver Firefox
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
driver: Firefox
sensitiveurl: mozilla\.org
Django settings: tests.settings (from ini file)
rootdir: /home/travis/build/mozilla/treeherder, inifile: setup.cfg
plugins: variables-1.7.1, selenium-1.11.4, django-3.1.2, base-url-1.4.1
collected 20 items                                                             
tests/selenium/test_filter_jobs_by_failure_result.py F..                 [ 15%]
tests/selenium/test_filter_jobs_by_keywords.py .                         [ 20%]
tests/selenium/test_filter_jobs_by_result_status.py .......              [ 55%]
tests/selenium/test_perfherder.py .                                      [ 60%]
tests/selenium/test_range.py ..                                          [ 70%]
tests/selenium/test_reset_filters.py .                                   [ 75%]
tests/selenium/test_select_unclassified_job.py ..                        [ 85%]
tests/selenium/test_switch_app.py .                                      [ 90%]
tests/selenium/test_switch_repo.py .                                     [ 95%]
tests/selenium/test_view_single_result.py .                              [100%]
=================================== FAILURES ===================================
________________ test_filter_jobs_by_failure_result[testfailed] ________________
base_url = 'http://localhost:38851'
selenium = <selenium.webdriver.firefox.webdriver.WebDriver (session="70922604-c981-48bf-8f95-7276b8237f83")>
test_jobs = None, result = 'exception'
    @pytest.mark.parametrize('result', RESULTS)
    def test_filter_jobs_by_failure_result(base_url, selenium, test_jobs, result):
        page = Treeherder(selenium, base_url).open()
        assert len(page.all_jobs) == len(RESULTS)
        with page.filters_menu() as filters:
            for result in RESULTS:
                getattr(filters, 'toggle_{}_jobs'.format(result))()
        assert len(page.all_jobs) == 0
        with page.filters_menu() as filters:
            getattr(filters, 'toggle_{}_jobs'.format(result))()
        assert len(page.all_jobs) == 1
        page.all_jobs[0].click()
>       assert page.info_panel.job_details.result == result
E       AssertionError: assert '' == 'exception'
E         + exception
tests/selenium/test_filter_jobs_by_failure_result.py:28: AssertionError
------------------------------- pytest-selenium --------------------------------
URL: http://localhost:38851/#/jobs?repo=test_treeherder_jobs&filter-resultStatus=success&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&filter-resultStatus=exception&selectedJob=3
Driver log: /tmp/pytest-of-travis/pytest-0/test_filter_jobs_by_failure_re0/driver.log
WARNING: Failed to gather log types: Message: GET /session/70922604-c981-48bf-8f95-7276b8237f83/log/types did not match a known command
=============================== warnings summary ===============================
tests/selenium/test_filter_jobs_by_failure_result.py::test_filter_jobs_by_failure_result[testfailed]
  /home/travis/venv/lib/python2.7/site-packages/django/db/backends/mysql/base.py:101: Warning: (124L, 'InnoDB rebuilding table to add column FTS_DOC_ID')
    return self.cursor.execute(query, args)
-- Docs: http://doc.pytest.org/en/latest/warnings.html
============== 1 failed, 19 passed, 1 warnings in 137.49 seconds ===============

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Access to the HTML report (with screenshot) would be useful here. The traceback suggests that the job remaining after the filter is applied has a result of "". I suspect this is perhaps a timing issue, and we might need to improve the wait after clicking the job. One suggestion might be to install the pytest-repeat plugin, and run:

> pytest --driver Firefox -x --count 100 --pdb tests/selenium/test_filter_jobs_by_failure_result.py::test_filter_jobs_by_failure_result

This will repeat the tests in that file up to 100 times, but if a failure is encountered will enter the Python debugger. This can be useful to identify if it's a timing issue or something else.

If you'd like to set up HTML reports in Travis you can use https://docs.travis-ci.com/user/uploading-artifacts/ to upload them to S3. These would include the traceback, driver logs, HTML source and a screenshot for all failing tests.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

If you just want to run the 'testfailed' parameter, then you can filter on keyword. I'd also suggest using --verbose for more informative console output:

> pytest --verbose --driver Firefox -x --count 100 --pdb -k testfailed tests/selenium/test_filter_jobs_by_failure_result.py::test_filter_jobs_by_failure_result

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've replicated this, and it does appear to be a timing issue. I'll see if I can improve the wait.

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3231

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/f8e9c6f72569fbba1ef1d31215b4345886ac9250
Bug 1438469 - Improve InfoPanel.is_open to include check for job result