1685786 - Intermittent 0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_verification[perfdocs] TEST-UNEXPECTED-FAIL

Status: RESOLVED INCOMPLETE

(Developer Infrastructure :: Lint and Formatting, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=326215952&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PR7N5DFhReK4zef5q_9E5g/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-01-08T19:09:56.257Z]  0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_start_and_fail[perfdocs] PASSED
[task 2021-01-08T19:09:56.257Z]  0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_start_and_pass[perfdocs] PASSED
[task 2021-01-08T19:09:56.257Z]  0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_bad_paths[perfdocs] PASSED
[task 2021-01-08T19:09:56.257Z]  0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_verification[perfdocs] TEST-UNEXPECTED-FAIL
[task 2021-01-08T19:09:56.257Z]  0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_framework_gatherers[perfdocs] PASSED
[task 2021-01-08T19:09:56.257Z]  0:17.73 tools/lint/test/test_perfdocs.py::test_perfdocs_logger_failure[perfdocs] PASSED
[task 2021-01-08T19:09:56.257Z]  0:17.73 
[task 2021-01-08T19:09:56.257Z]  0:17.73 =================================== FAILURES ===================================
[task 2021-01-08T19:09:56.257Z]  0:17.73 _____________________ test_perfdocs_verification[perfdocs] _____________________
[task 2021-01-08T19:09:56.257Z]  0:17.73 
[task 2021-01-08T19:09:56.257Z]  0:17.73 logger = <MagicMock name='PerfDocLogger' id='4561728208'>
[task 2021-01-08T19:09:56.257Z]  0:17.73 structured_logger = <mozlog.structuredlog.StructuredLogger object at 0x10fe62c10>
[task 2021-01-08T19:09:56.257Z]  0:17.73 perfdocs_sample = {'config': '/var/folders/hg/2485jg4s6vq8ggdnpc4ykz_4000017/T/tmpqfo3y8z_/perfdocs/config.yml', 'index': '/var/folders/...te/perftest.ini', 'test': '/var/folders/hg/2485jg4s6vq8ggdnpc4ykz_4000017/T/tmpqfo3y8z_/suite/perftest_sample.js', ...}
[task 2021-01-08T19:09:56.258Z]  0:17.73 
[task 2021-01-08T19:09:56.258Z]  0:17.73     @mock.patch("perfdocs.logger.PerfDocLogger")
[task 2021-01-08T19:09:56.258Z]  0:17.73     def test_perfdocs_verification(logger, structured_logger, perfdocs_sample):
[task 2021-01-08T19:09:56.258Z]  0:17.73         top_dir = perfdocs_sample["top_dir"]
[task 2021-01-08T19:09:56.258Z]  0:17.73         setup_sample_logger(logger, structured_logger, top_dir)
[task 2021-01-08T19:09:56.258Z]  0:17.73 
[task 2021-01-08T19:09:56.258Z]  0:17.73         from perfdocs.verifier import Verifier
[task 2021-01-08T19:09:56.258Z]  0:17.73 
[task 2021-01-08T19:09:56.258Z]  0:17.73         verifier = Verifier(top_dir)
[task 2021-01-08T19:09:56.258Z]  0:17.73 >       verifier.validate_tree()
[task 2021-01-08T19:09:56.258Z]  0:17.73 
[task 2021-01-08T19:09:56.258Z]  0:17.73 tools/lint/test/test_perfdocs.py:175:
[task 2021-01-08T19:09:56.258Z]  0:17.73 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2021-01-08T19:09:56.258Z]  0:17.73 
[task 2021-01-08T19:09:56.258Z]  0:17.73 self = <perfdocs.verifier.Verifier object at 0x1151c1850>
[task 2021-01-08T19:09:56.258Z]  0:17.73 
[task 2021-01-08T19:09:56.258Z]  0:17.73     def validate_tree(self):
[task 2021-01-08T19:09:56.259Z]  0:17.73         """
[task 2021-01-08T19:09:56.259Z]  0:17.73             Validate the `perfdocs` directory that was found.
[task 2021-01-08T19:09:56.259Z]  0:17.73             Returns True if it is good, false otherwise.
[task 2021-01-08T19:09:56.259Z]  0:17.73 
[task 2021-01-08T19:09:56.259Z]  0:17.73             :return bool: True/False => Passed/Failed Validation
[task 2021-01-08T19:09:56.259Z]  0:17.73             """
[task 2021-01-08T19:09:56.259Z]  0:17.73         found_good = 0
[task 2021-01-08T19:09:56.259Z]  0:17.73 
[task 2021-01-08T19:09:56.259Z]  0:17.73         # For each framework, check their files and validate descriptions
[task 2021-01-08T19:09:56.259Z]  0:17.73         for matched in self._gatherer.perfdocs_tree:
[task 2021-01-08T19:09:56.259Z]  0:17.73             # Get the paths to the YAML and RST for this framework
[task 2021-01-08T19:09:56.259Z]  0:17.73             matched_yml = os.path.join(matched["path"], matched["yml"])
[task 2021-01-08T19:09:56.259Z]  0:17.73             matched_rst = os.path.join(matched["path"], matched["rst"])
[task 2021-01-08T19:09:56.259Z]  0:17.73 
[task 2021-01-08T19:09:56.259Z]  0:17.73             _valid_files = {
[task 2021-01-08T19:09:56.259Z]  0:17.73                 "yml": self.validate_yaml(matched_yml),
[task 2021-01-08T19:09:56.259Z]  0:17.73                 "rst": True,
[task 2021-01-08T19:09:56.259Z]  0:17.73             }
[task 2021-01-08T19:09:56.260Z]  0:17.73             if not read_yaml(matched_yml)["static-only"]:
[task 2021-01-08T19:09:56.260Z]  0:17.73                 _valid_files["rst"] = self.validate_rst_content(matched_rst)
[task 2021-01-08T19:09:56.260Z]  0:17.73 
[task 2021-01-08T19:09:56.260Z]  0:17.73             # Log independently the errors found for the matched files
[task 2021-01-08T19:09:56.260Z]  0:17.73             for file_format, valid in _valid_files.items():
[task 2021-01-08T19:09:56.260Z]  0:17.73                 if not valid:
[task 2021-01-08T19:09:56.260Z]  0:17.73                     logger.log("File validation error: {}".format(file_format))
[task 2021-01-08T19:09:56.260Z]  0:17.73             if not all(_valid_files.values()):
[task 2021-01-08T19:09:56.260Z]  0:17.73                 continue
[task 2021-01-08T19:09:56.260Z]  0:17.73             found_good += 1
[task 2021-01-08T19:09:56.260Z]  0:17.73 
[task 2021-01-08T19:09:56.260Z]  0:17.73             self._check_framework_descriptions(matched)
[task 2021-01-08T19:09:56.260Z]  0:17.73 
[task 2021-01-08T19:09:56.260Z]  0:17.73         if not found_good:
[task 2021-01-08T19:09:56.260Z]  0:17.73 >           raise Exception("No valid perfdocs directories found")
[task 2021-01-08T19:09:56.260Z]  0:17.73 E           Exception: No valid perfdocs directories found
[task 2021-01-08T19:09:56.260Z]  0:17.73 
[task 2021-01-08T19:09:56.260Z]  0:17.73 tools/lint/perfdocs/verifier.py:310: Exception
[task 2021-01-08T19:09:56.261Z]  0:17.73 ====================== 1 failed, 5 passed in 3.33 seconds ======================
[task 2021-01-08T19:09:56.261Z]  0:17.73 Setting retcode to 1 from /Users/cltbld/tasks/task_1610132735/checkouts/gecko/tools/lint/test/test_perfdocs.py
[task 2021-01-08T19:09:57.575Z]  0:19.05 /Users/cltbld/tasks/task_1610132735/checkouts/gecko/tools/lint/test/test_yaml.py
[task 2021-01-08T19:09:57.575Z]  0:19.05 ============================= test session starts ==============================
[task 2021-01-08T19:09:57.575Z]  0:19.05 platform darwin -- Python 3.7.4, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /Users/cltbld/tasks/task_1610132735/checkouts/gecko/obj-x86_64-apple-darwin18.6.0/_virtualenvs/python-test/bin/python
[task 2021-01-08T19:09:57.575Z]  0:19.05 rootdir: /Users/cltbld/tasks/task_1610132735/checkouts/gecko, inifile: /Users/cltbld/tasks/task_1610132735/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2021-01-08T19:09:57.575Z]  0:19.05 collecting ... collected 1 item
[task 2021-01-08T19:09:57.575Z]  0:19.05 
[task 2021-01-08T19:09:57.575Z]  0:19.05 tools/lint/test/test_yaml.py::test_basic[yamllint] PASSED
[task 2021-01-08T19:09:57.575Z]  0:19.05 
[task 2021-01-08T19:09:57.576Z]  0:19.05 =========================== 1 passed in 2.54 seconds ===========================
[task 2021-01-08T19:10:11.661Z]  0:33.13 /Users/cltbld/tasks/task_1610132735/checkouts/gecko/tools/lint/test/test_eslint.py
[task 2021-01-08T19:10:11.661Z]  0:33.13 ============================= test session starts ==============================
[task 2021-01-08T19:10:11.661Z]  0:33.13 platform darwin -- Python 3.7.4, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /Users/cltbld/tasks/task_1610132735/checkouts/gecko/obj-x86_64-apple-darwin18.6.0/_virtualenvs/python-test/bin/python
[task 2021-01-08T19:10:11.661Z]  0:33.13 rootdir: /Users/cltbld/tasks/task_1610132735/checkouts/gecko, inifile: /Users/cltbld/tasks/task_1610132735/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2021-01-08T19:10:11.661Z]  0:33.13 collecting ... collected 3 items
[task 2021-01-08T19:10:11.661Z]  0:33.13 
[task 2021-01-08T19:10:11.661Z]  0:33.13 tools/lint/test/test_eslint.py::test_lint_with_global_exclude[eslint] PASSED
[task 2021-01-08T19:10:11.661Z]  0:33.13 tools/lint/test/test_eslint.py::test_no_files_to_lint[eslint] PASSED
[task 2021-01-08T19:10:11.661Z]  0:33.13 tools/lint/test/test_eslint.py::test_bad_import[eslint] PASSED
[task 2021-01-08T19:10:11.661Z]  0:33.13 
[task 2021-01-08T19:10:11.662Z]  0:33.13 ========================== 3 passed in 22.26 seconds ===========================
[task 2021-01-08T19:10:11.662Z]  0:33.13 Return code from mach python-test: 1
[task 2021-01-08T19:10:11.721Z] Test configuration changed. Regenerating backend.
[task 2021-01-08T19:10:11.721Z] No build detected, test metadata may be incomplete.
[taskcluster 2021-01-08T19:10:11.794Z]    Exit Code: 1
[taskcluster 2021-01-08T19:10:11.794Z]    User Time: 5m34.208642s
[taskcluster 2021-01-08T19:10:11.794Z]  Kernel Time: 4m19.28557s
[taskcluster 2021-01-08T19:10:11.794Z]    Wall Time: 4m34.199558271s
[taskcluster 2021-01-08T19:10:11.794Z]       Result: FAILED
[taskcluster 2021-01-08T19:10:11.794Z] === Task Finished ===
[taskcluster 2021-01-08T19:10:11.794Z] Task Duration: 4m34.203208998s
[taskcluster 2021-01-08T19:10:11.794Z] [mounts] Preserving cache: Moving "/Users/cltbld/tasks/task_1610132735/checkouts" to "/Users/cltbld/caches/b3Fp0HSKT7SHD4vKL-imwA"
[taskcluster:error] exit status 1```

Comment 1

[Intermittent Failures Robot]

1 failures in 2956 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1685786&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 2

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 3

[Intermittent Failures Robot]

1 failures in 3966 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1685786&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 5

[Intermittent Failures Robot]

3 failures in 3854 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• linux1804-64: 2
  • opt: 2
• windows10-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1685786&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 3224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1685786&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 7

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.