Closed Bug 1696344 Opened 4 years ago Closed 4 years ago

Intermittent 0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_framework_gatherers_urls[perfdocs] TEST-UNEXPECTED-FAIL

Categories

(Testing :: Performance, defect, P5)

Product:
Testing
Component:
Performance
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Filed by: nerli [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=331993478&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bTwqvEVLQEiJelFm02gN-Q/runs/0/artifacts/public/logs/live_backing.log

[task 2021-03-04T04:23:25.098Z]  0:30.50 /builds/worker/checkouts/gecko/tools/lint/test/test_perfdocs.py
[task 2021-03-04T04:23:25.109Z]  0:30.50 ============================= test session starts ==============================
[task 2021-03-04T04:23:25.109Z]  0:30.50 platform linux -- Python 3.6.9, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/python-test/bin/python
[task 2021-03-04T04:23:25.110Z]  0:30.50 rootdir: /builds/worker/checkouts/gecko, inifile: /builds/worker/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2021-03-04T04:23:25.110Z]  0:30.50 collecting ... collected 19 items
[task 2021-03-04T04:23:25.110Z]  0:30.50 
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_start_and_fail[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_start_and_pass[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_bad_paths[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_gatherer_fetch_perfdocs_tree[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_gatherer_get_test_list[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verification[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_validate_yaml_pass[perfdocs] PASSED
[task 2021-03-04T04:23:25.110Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_invalid_yaml[perfdocs] PASSED
[task 2021-03-04T04:23:25.116Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_validate_rst_pass[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_invalid_rst[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_validate_descriptions_pass[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_not_existing_suite_in_test_list[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_not_existing_tests_in_suites[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_missing_contents_in_suite[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_invalid_dir[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_verifier_file_invalidation[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_framework_gatherers[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_framework_gatherers_urls[perfdocs] TEST-UNEXPECTED-FAIL
[task 2021-03-04T04:23:25.117Z]  0:30.50 tools/lint/test/test_perfdocs.py::test_perfdocs_logger_failure[perfdocs] PASSED
[task 2021-03-04T04:23:25.117Z]  0:30.50 
[task 2021-03-04T04:23:25.117Z]  0:30.50 =================================== FAILURES ===================================
[task 2021-03-04T04:23:25.118Z]  0:30.50 _______________ test_perfdocs_framework_gatherers_urls[perfdocs] _______________
[task 2021-03-04T04:23:25.118Z]  0:30.50 
[task 2021-03-04T04:23:25.118Z]  0:30.50 logger = <MagicMock name='PerfDocLogger' id='140591720465296'>
[task 2021-03-04T04:23:25.118Z]  0:30.50 structured_logger = <mozlog.structuredlog.StructuredLogger object at 0x7fde0f88deb8>
[task 2021-03-04T04:23:25.118Z]  0:30.50 perfdocs_sample = {'config': '/tmp/tmprpsb7hg7/perfdocs/config.yml', 'config_2': '/tmp/tmprpsb7hg7/perfdocs/config_2.yml', 'example1_man...ptor/suite/raptor_example1.ini', 'example2_manifest': '/tmp/tmprpsb7hg7/raptor/another_suite/raptor_example2.ini', ...}
[task 2021-03-04T04:23:25.118Z]  0:30.50 
[task 2021-03-04T04:23:25.119Z]  0:30.51     @mock.patch("perfdocs.logger.PerfDocLogger")
[task 2021-03-04T04:23:25.119Z]  0:30.51     def test_perfdocs_framework_gatherers_urls(logger, structured_logger, perfdocs_sample):
[task 2021-03-04T04:23:25.120Z]  0:30.51         top_dir = perfdocs_sample["top_dir"]
[task 2021-03-04T04:23:25.120Z]  0:30.51         setup_sample_logger(logger, structured_logger, top_dir)
[task 2021-03-04T04:23:25.120Z]  0:30.51 
[task 2021-03-04T04:23:25.121Z]  0:30.51         from perfdocs.gatherer import frameworks
[task 2021-03-04T04:23:25.121Z]  0:30.51         from perfdocs.verifier import Verifier
[task 2021-03-04T04:23:25.121Z]  0:30.51         from perfdocs.generator import Generator
[task 2021-03-04T04:23:25.121Z]  0:30.51         from perfdocs.utils import read_yaml
[task 2021-03-04T04:23:25.121Z]  0:30.51 
[task 2021-03-04T04:23:25.121Z]  0:30.51         # This test is only for raptor
[task 2021-03-04T04:23:25.121Z]  0:30.51         gatherer = frameworks["raptor"]
[task 2021-03-04T04:23:25.121Z]  0:30.51         with open(perfdocs_sample["config"], "w") as f:
[task 2021-03-04T04:23:25.121Z]  0:30.51             f.write(DYNAMIC_SAMPLE_CONFIG.format("raptor"))
[task 2021-03-04T04:23:25.121Z]  0:30.51 
[task 2021-03-04T04:23:25.121Z]  0:30.51         fg = gatherer(perfdocs_sample["config_2"], top_dir)
[task 2021-03-04T04:23:25.122Z]  0:30.51         fg.get_suite_list = mock.Mock()
[task 2021-03-04T04:23:25.122Z]  0:30.51         fg.get_suite_list.return_value = {
[task 2021-03-04T04:23:25.122Z]  0:30.51             "suite": [perfdocs_sample["example1_manifest"]],
[task 2021-03-04T04:23:25.122Z]  0:30.51             "another_suite": [perfdocs_sample["example2_manifest"]],
[task 2021-03-04T04:23:25.122Z]  0:30.51         }
[task 2021-03-04T04:23:25.122Z]  0:30.51 
[task 2021-03-04T04:23:25.122Z]  0:30.51         v = Verifier(top_dir)
[task 2021-03-04T04:23:25.122Z]  0:30.51         gn = Generator(v, generate=True, workspace=top_dir)
[task 2021-03-04T04:23:25.122Z]  0:30.51 
[task 2021-03-04T04:23:25.122Z]  0:30.51         # Check to make sure that if a test is present under multiple
[task 2021-03-04T04:23:25.122Z]  0:30.51         # suties the urls are generated correctly for the test under
[task 2021-03-04T04:23:25.122Z]  0:30.51         # every suite
[task 2021-03-04T04:23:25.122Z]  0:30.51         for suite, suitetests in fg.get_test_list().items():
[task 2021-03-04T04:23:25.122Z]  0:30.51             url = fg._urls.get(suite)
[task 2021-03-04T04:23:25.122Z]  0:30.51             assert url is not None
[task 2021-03-04T04:23:25.122Z]  0:30.51             assert url[0]["test_name"] == "Example"
[task 2021-03-04T04:23:25.122Z]  0:30.51             assert url[0]["url"] == "Example_url"
[task 2021-03-04T04:23:25.122Z]  0:30.51 
[task 2021-03-04T04:23:25.123Z]  0:30.51 >       perfdocs_tree = gn._perfdocs_tree[0]
[task 2021-03-04T04:23:25.123Z]  0:30.51 E       IndexError: list index out of range
[task 2021-03-04T04:23:25.123Z]  0:30.51 
[task 2021-03-04T04:23:25.123Z]  0:30.51 tools/lint/test/test_perfdocs.py:519: IndexError
[task 2021-03-04T04:23:25.123Z]  0:30.51 ===================== 1 failed, 18 passed in 2.16 seconds ======================```

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → INCOMPLETE

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=337288780&repo=mozilla-central&lineNumber=841

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.