1437951 - [ops infra socorro] passing end to end tests

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Miles Crabill [:miles]]

The e2e test suite in the Socorro repo should pass when run against -new-stage and -new-prod, or at least have the same results as the current infra.

This bug covers bringing the ops infra deployments up to speed.

Comment 1

[Miles Crabill [:miles]]

see also: bug 1429546

Comment 2

[Chris Hartjes [:grumpy][:chartjes]]

All test runs done with Socorro e2e test suite running using Firefox 57

Test runs -- current stage

[gw1] [  3%] PASSED tests/test_api.py::TestAPI::test_platforms
[gw2] [  6%] PASSED tests/test_api.py::TestAPI::test_crontabber
[gw3] [  9%] PASSED tests/test_api.py::TestAPI::test_supersearch_fields
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
[gw3] [ 12%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
[gw1] [ 15%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
[gw2] [ 18%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
[gw1] [ 21%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
[gw3] [ 24%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
[gw2] [ 27%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
[gw1] [ 30%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
[gw0] [ 33%] PASSED tests/test_api.py::TestAPI::test_public_api_navigation
tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
[gw2] [ 36%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
[gw3] [ 39%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
[gw0] [ 42%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
[gw1] [ 45%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
[gw2] [ 48%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
[gw3] [ 51%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
[gw0] [ 54%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
[gw1] [ 57%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
[gw2] [ 60%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
[gw3] [ 63%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
[gw1] [ 66%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw0] [ 69%] PASSED tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw2] [ 72%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw3] [ 75%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
tests/test_search.py::TestSuperSearch::test_search_change_column
[gw1] [ 78%] PASSED tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw2] [ 81%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw0] [ 84%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw3] [ 87%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_column
tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature
tests/test_search.py::TestSuperSearch::test_search_with_one_line
[gw1] [ 90%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_one_line
[gw0] [ 93%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
[gw3] [ 96%] PASSED tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
[gw2] [100%] FAILED tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature

Current Docker-based stage at https://socorro-webapp.stage.mozaws.net


[gw2] [  3%] PASSED tests/test_api.py::TestAPI::test_crontabber
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
[gw1] [  6%] PASSED tests/test_api.py::TestAPI::test_platforms
[gw3] [  9%] PASSED tests/test_api.py::TestAPI::test_supersearch_fields
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
[gw0] [ 12%] FAILED tests/test_api.py::TestAPI::test_public_api_navigation
tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
[gw0] [ 15%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
[gw1] [ 18%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
[gw3] [ 21%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
[gw2] [ 24%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
[gw1] [ 27%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
[gw2] [ 30%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
[gw0] [ 33%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
[gw3] [ 36%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
[gw1] [ 39%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
[gw2] [ 42%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
[gw0] [ 45%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
[gw3] [ 48%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
[gw1] [ 51%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
[gw2] [ 54%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
[gw0] [ 57%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
[gw3] [ 60%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
[gw1] [ 63%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
[gw0] [ 66%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw2] [ 69%] PASSED tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw3] [ 72%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw0] [ 75%] PASSED tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
tests/test_search.py::TestSuperSearch::test_search_with_one_line
[gw1] [ 78%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
[gw2] [ 81%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw3] [ 84%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw0] [ 87%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_one_line
tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature
tests/test_search.py::TestSuperSearch::test_search_change_column
tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
[gw2] [ 90%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
[gw0] [ 93%] PASSED tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
[gw3] [ 96%] FAILED tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature
[gw1] [100%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_column

Run against production

[gw1] [  3%] PASSED tests/test_api.py::TestAPI::test_platforms
[gw2] [  6%] PASSED tests/test_api.py::TestAPI::test_crontabber
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
[gw0] [  9%] PASSED tests/test_api.py::TestAPI::test_supersearch_fields
tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
[gw0] [ 12%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
[gw1] [ 15%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
[gw2] [ 18%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
[gw0] [ 21%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
[gw3] [ 24%] PASSED tests/test_api.py::TestAPI::test_public_api_navigation
tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
[gw1] [ 27%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
[gw2] [ 30%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
[gw3] [ 33%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
[gw1] [ 36%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
[gw0] [ 39%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
[gw2] [ 42%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
[gw3] [ 45%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
[gw1] [ 48%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
[gw0] [ 51%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
[gw2] [ 54%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
[gw3] [ 57%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
[gw0] [ 60%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
[gw1] [ 63%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
[gw3] [ 66%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw2] [ 69%] PASSED tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw0] [ 72%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw1] [ 75%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
tests/test_search.py::TestSuperSearch::test_search_change_column
[gw3] [ 78%] PASSED tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw2] [ 81%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw0] [ 84%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw1] [ 87%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_column
tests/test_search.py::TestSuperSearch::test_search_with_one_line
tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature
tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
[gw1] [ 90%] PASSED tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
[gw3] [ 93%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_one_line
[gw2] [ 96%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
[gw0] [100%] FAILED tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature

Comment 3

[Chris Hartjes [:grumpy][:chartjes]]

Run against Dockerized-production

 pytest --base-url "https://socorro-webapp.prod.mozaws.net"                                                                                          
=================================================================== test session starts ====================================================================
platform darwin -- Python 2.7.14, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /Users/chartjes/.local/share/virtualenvs/e2e-tests-Zqekq6Nj/bin/python2.7
cachedir: .cache
driver: Firefox
sensitiveurl: mozilla\.org
metadata: {'Python': '2.7.14', 'Driver': 'Firefox', 'Capabilities': {}, 'Base URL': 'https://socorro-webapp.prod.mozaws.net', 'Platform': 'Darwin-17.4.0-x86_64-i386-64bit', 'Plugins': {'variables': '1.7.0', 'selenium': '1.11.3', 'xdist': '1.21.0', 'mozlog': '3.7', 'html': '1.16.1', 'forked': '0.2', 'base-url': '1.4.1', 'metadata': '1.5.1'}, 'Packages': {'py': '1.5.2', 'pytest': '3.3.2', 'pluggy': '0.6.0'}}
baseurl: https://socorro-webapp.prod.mozaws.net
rootdir: /Users/chartjes/chartjes/socorro/e2e-tests, inifile: setup.cfg
plugins: xdist-1.21.0, variables-1.7.0, selenium-1.11.3, metadata-1.5.1, html-1.16.1, forked-0.2, base-url-1.4.1, mozlog-3.7
[gw0] darwin Python 2.7.14 cwd: /Users/chartjes/chartjes/socorro/e2e-tests
[gw1] darwin Python 2.7.14 cwd: /Users/chartjes/chartjes/socorro/e2e-tests
[gw2] darwin Python 2.7.14 cwd: /Users/chartjes/chartjes/socorro/e2e-tests
[gw3] darwin Python 2.7.14 cwd: /Users/chartjes/chartjes/socorro/e2e-tests
[gw0] Python 2.7.14 (default, Jan  6 2018, 12:16:16)  -- [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
[gw1] Python 2.7.14 (default, Jan  6 2018, 12:16:16)  -- [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
[gw2] Python 2.7.14 (default, Jan  6 2018, 12:16:16)  -- [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
[gw3] Python 2.7.14 (default, Jan  6 2018, 12:16:16)  -- [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
gw0 [33] / gw1 [33] / gw2 [33] / gw3 [33]
scheduling tests via LoadScheduling

tests/test_api.py::TestAPI::test_supersearch_fields
tests/test_api.py::TestAPI::test_public_api_navigation
tests/test_api.py::TestAPI::test_platforms
tests/test_api.py::TestAPI::test_crontabber
[gw1] [  3%] PASSED tests/test_api.py::TestAPI::test_platforms
[gw2] [  6%] PASSED tests/test_api.py::TestAPI::test_crontabber
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
[gw3] [  9%] PASSED tests/test_api.py::TestAPI::test_supersearch_fields
tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
[gw3] [ 12%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_exploitable_crash_report_is_not_displayed_for_logged_out_users
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
[gw1] [ 15%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
[gw2] [ 18%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
[gw0] [ 21%] PASSED tests/test_api.py::TestAPI::test_public_api_navigation
tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
[gw3] [ 24%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Thunderbird]
[gw1] [ 27%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
[gw2] [ 30%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
[gw0] [ 33%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_bugzilla_link_contain_current_site
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
[gw1] [ 36%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_reports_form_has_same_product[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
[gw3] [ 39%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
[gw2] [ 42%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_current_version_selected_in_top_crashers_header[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
[gw0] [ 45%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_all_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
[gw1] [ 48%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_plugin_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
[gw3] [ 51%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crasher_filter_browser_return_results
tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
[gw2] [ 54%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Firefox]
tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
[gw0] [ 57%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[Thunderbird]
tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
[gw1] [ 60%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[FennecAndroid]
tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
[gw3] [ 63%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_top_crashers_reports_links_work[SeaMonkey]
tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
[gw0] [ 66%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_7_days_is_selected_default_for_nightlies
tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw2] [ 69%] PASSED tests/test_crash_reports.py::TestCrashReports::test_top_crasher_reports_tab_has_uuid_report
tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw1] [ 72%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_browser_reports_have_browser_icon
tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw3] [ 75%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_only_plugin_reports_have_plugin_icon
tests/test_search.py::TestSuperSearch::test_search_change_column
[gw0] [ 78%] PASSED tests/test_search.py::TestSuperSearch::test_super_search_page_is_loaded
[gw2] [ 81%] PASSED tests/test_crash_reports.py::TestCrashReports::test_that_lowest_version_topcrashers_do_not_return_errors
[gw1] [ 84%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_facet
[gw3] [ 87%] PASSED tests/test_search.py::TestSuperSearch::test_search_change_column
tests/test_search.py::TestSuperSearch::test_search_with_one_line
tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature
tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
[gw0] [ 90%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_one_line
[gw2] [ 93%] PASSED tests/test_search.py::TestSuperSearch::test_search_with_multiple_lines
[gw3] [ 96%] PASSED tests/test_search.py::TestSearchForSpecificResults::test_selecting_one_version_doesnt_show_other_versions
[gw1] [100%] FAILED tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature

Comment 4

[Miles Crabill [:miles]]

Looks like every environment fails: tests/test_search.py::TestSearchForSpecificResults::test_search_for_valid_signature
and -new-stage fails: tests/test_api.py::TestAPI::test_public_api_navigation

Pretty good results!

Comment 5

[Miles Crabill [:miles]]

It looks like these test run are run locally in a virtualenv and not via the new fancy Docker container.
I tried to run them myself in the container, but ran into issues that you didn't have in your runs.

Here is the log from my docker session. I snipped out some dives into Python code corresponding to the stack traces.

Have you tested the dockerized e2e tests successfully?

Comment 6

[Chris Hartjes [:grumpy][:chartjes]]

I don't run them using the Docker container -- I run them manually on my laptop inside a virtualenv.

I believe that stephend has verified that the tests pass in Docker but building the image fails for me. I've attached the log file.

Comment 7

[Chris Hartjes [:grumpy][:chartjes]]

Attachment: Failure to build Docker container containing tests

Comment 8

[Miles Crabill [:miles]]

Attachment: miles' docker e2e traceback

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Miles and I synched up in #breakpad, and we got identical results, running with -n 1, which limits pytest-xdist's concurrency/parallel-Firefox-build running to 1.

This helps mitigate some of the similar session-based issues (around geckodriver/IPC/Marionette), such as bug 1433503 and bug 1365914.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: socorro-docker-output

Console output of:

docker run -it socorro-tests pytest -n 1 --base-url https://socorro-webapp.stage.mozaws.net

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Where's this bug at? Is there outstanding work that needs to be done still?

Comment 12

[Miles Crabill [:miles]]

I think we're satisfied with the results of the e2e tests against the new infrastructure. The last time I tried, they match the results of the existing infrastructure almost entirely, so we're good here.

Please reopen if that isn't the case.

Comment 13

[Stephen Donner [:stephend] Not actively reading bugmail]

I know this bug is resolved fixed, but for completeness:

I reran the prod tests (freshly checked out from master) with both -n=1 and without setting concurrency, explicitly (which defaulted us to use n=auto, and thus, at least 4+ concurrent instances running the tests.

These latest results mirror comment 9's results, which is great:

======================== 1 failed, 32 passed in 281.94 seconds ========================
Stephens-MacBook-Pro:e2e-tests stephendonner$ docker run -t socorro-tests pytest -n=1 \
                                              --base-url=https://crash-stats.mozilla.com

vs.

=========================== 3 failed, 29 passed, 3 error in 95.54 seconds ==============
Stephens-MacBook-Pro:e2e-tests stephendonner$ docker run -t socorro-tests pytest \ 
                                              --base-url=https://crash-stats.mozilla.com

Will and I will continue to work together to figure how to stabilize/make them more atomic/run them as part of the overall build/release process, etc., in follow-up bugs/PRs.