Intermittent [test-verify] [test-coverage] No tests run or test summary not found

RESOLVED FIXED in Firefox 62

Status

defect
P5
normal
RESOLVED FIXED
Last year
Last year

People

(Reporter: intermittent-bug-filer, Assigned: jmaher)

Tracking

({intermittent-failure})

Version 3
mozilla62
Points:
---

Firefox Tracking Flags

(firefox62 fixed)

Details

(Whiteboard: [stockwell disable-recommended])

Attachments

(1 attachment, 1 obsolete attachment)

Reporter

Description

Last year
treeherder
Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=176595819&repo=autoland

https://queue.taskcluster.net/v1/task/CzqfXzTARweE8qnqYR9vCg/runs/0/artifacts/public/logs/live_backing.log

17:12:36     INFO -  REFTEST SUITE-START | Running 0 tests
17:12:36     INFO -  REFTEST INFO | Result summary:
17:12:36     INFO -  REFTEST INFO | Successful: 0 (0 pass, 0 load only)
17:12:36     INFO -  REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception)
17:12:36     INFO -  REFTEST INFO | Known problems: 0 (0 known fail, 0 known asserts, 0 random, 0 skipped, 0 slow)
17:12:36     INFO -  REFTEST SUITE-END | Shutdown
17:12:36     INFO -  REFTEST INFO | :::
17:12:36     INFO -  REFTEST INFO | ::: Test verification summary for:
17:12:36     INFO -  REFTEST INFO | :::
17:12:36     INFO -  REFTEST INFO | ::: Z:/task_1525279957/build/tests/reftest/tests/layout/reftests/bugs/1219985-ref-opaque-with-rendering.html
17:12:36     INFO -  REFTEST INFO | :::
17:12:36     INFO -  REFTEST INFO | ::: 1. Run each test 10 times in one browser. : Pass
17:12:36     INFO -  REFTEST INFO | ::: 2. Run each test 5 times in a new browser each time. : Pass
17:12:36     INFO -  REFTEST INFO | ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
17:12:36     INFO -  REFTEST INFO | ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
17:12:36     INFO -  REFTEST INFO | :::
17:12:36     INFO -  REFTEST INFO | ::: Test verification PASSED
17:12:36     INFO -  REFTEST INFO | :::
17:12:36     INFO - Return code: 0
17:12:36    ERROR - No tests run or test summary not found
17:12:36     INFO - TinderboxPrint: reftest-reftest<br/><em class="testfail">T-FAIL</em>
17:12:36  WARNING - # TBPL WARNING #
17:12:36  WARNING - setting return code to 1
17:12:36  WARNING - TinderboxPrint: Per-test run of ...\1219985-ref-opaque-with-rendering.html<br/>: WARNING
17:12:36     INFO - Running command: ['Z:\\task_1525279957\\build\\venv\\Scripts\\python', '-u', 'Z:\\task_1525279957\\build\\tests\\reftest\\runreftest.py', '--appname=Z:\\task_1525279957\\build\\application\\firefox\\firefox.exe', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=Z:\\task_1525279957\\build\\symbols', '--log-raw=Z:\\task_1525279957\\build\\blobber_upload_dir\\reftest_raw.log', '--log-errorsummary=Z:\\task_1525279957\\build\\blobber_upload_dir\\reftest_errorsummary.log', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=Z:\\task_1525279957\\build', '--suite=reftest', '--verify-max-time=900', '--verify', u'Z:/task_1525279957/build/tests/reftest/tests/layout/reftests/bugs/1219985-ref-transparent-with-rendering.html'] in Z:\task_1525279957\build
17:12:36     INFO - Copy/paste: Z:\task_1525279957\build\venv\Scripts\python -u Z:\task_1525279957\build\tests\reftest\runreftest.py --appname=Z:\task_1525279957\build\application\firefox\firefox.exe --utility-path=tests/bin --extra-profile-file=tests/bin/plugins --symbols-path=Z:\task_1525279957\build\symbols --log-raw=Z:\task_1525279957\build\blobber_upload_dir\reftest_raw.log --log-errorsummary=Z:\task_1525279957\build\blobber_upload_dir\reftest_errorsummary.log --cleanup-crashes --marionette-startup-timeout=180 --sandbox-read-whitelist=Z:\task_1525279957\build --suite=reftest --verify-max-time=900 --verify Z:/task_1525279957/build/tests/reftest/tests/layout/reftests/bugs/1219985-ref-transparent-with-rendering.html
17:12:36     INFO - Using env: {'ALLUSERSPROFILE': 'C:\\ProgramData',
17:12:36     INFO -  'APPDATA': 'Z:\\task_1525279957\\AppData\\Roaming',
17:12:36     INFO -  'COMMONPROGRAMFILES': 'C:\\Program Files\\Common Files',
17:12:36     INFO -  'COMMONPROGRAMFILES(X86)': 'C:\\Program Files (x86)\\Common Files',
17:12:36     INFO -  'COMMONPROGRAMW6432': 'C:\\Program Files\\Common Files',
17:12:36     INFO -  'COMPUTERNAME': 'I-0F84183F37EB7',
17:12:36     INFO -  'COMSPEC': 'C:\\Windows\\system32\\cmd.exe',
17:12:36     INFO -  'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/autoland',
17:12:36     INFO -  'GECKO_HEAD_REV': '1292058fb7248a32e4ad3062397c0c86defaaaab',
17:12:36     INFO -  'HOMEDRIVE': 'C:',
17:12:36     INFO -  'HOMEPATH': '\\Users\\task_1525279957',
17:12:36     INFO -  'LOCALAPPDATA': 'Z:\\task_1525279957\\AppData\\Local',
17:12:36     INFO -  'LOGONSERVER': '\\\\I-0F84183F37EB7',
17:12:36     INFO -  'MINIDUMP_SAVE_PATH': 'Z:\\task_1525279957\\build\\blobber_upload_dir',
17:12:36     INFO -  'MINIDUMP_STACKWALK': 'Z:\\task_1525279957\\build\\win32-minidump_stackwalk.exe',
17:12:36     INFO -  'MOZILLABUILD': 'C:\\mozilla-build',
17:12:36     INFO -  'MOZ_AUTOMATION': '1',
17:12:36     INFO -  'MOZ_NODE_PATH': 'Z:\\task_1525279957\\build\\node-win32.exe',
17:12:36     INFO -  'MOZ_UPLOAD_DIR': 'Z:\\task_1525279957\\build\\blobber_upload_dir',
17:12:36     INFO -  'NUMBER_OF_PROCESSORS': '8',
17:12:36     INFO -  'OS': 'Windows_NT',
Summary: Intermittent No tests run or test summary not found → Intermittent [test-verify] No tests run or test summary not found
In the past week there are 80 failures, mainly on Windows 10 x64 opt/debug/pgo and Windows 7 debug/opt/pgo. Lower occurrence rate is seen on android-4-3-armv7-api16 and Linux platforms, opt/debug build types.

Recent log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=177103800&lineNumber=6812

Log snippet:
10:02:39     INFO -  SUITE-START | Running 1 tests
10:02:39     INFO -  :::
10:02:39     INFO -  ::: Test verification summary for:
10:02:39     INFO -  :::
10:02:39     INFO -  ::: devtools/client/inspector/animation-old/test/browser_animation_refresh_when_active.js
10:02:39     INFO -  :::
10:02:39     INFO -  ::: 1. Run each test 10 times in one browser. : Pass
10:02:39     INFO -  ::: 2. Run each test 5 times in a new browser each time. : Pass
10:02:39     INFO -  ::: 3. Run each test 10 times in one browser, in chaos mode. : Pass
10:02:39     INFO -  ::: 4. Run each test 5 times in a new browser each time, in chaos mode. : Pass
10:02:39     INFO -  :::
10:02:39     INFO -  ::: Test verification PASSED
10:02:39     INFO -  :::
10:02:39     INFO -  Buffered messages finished
10:02:39     INFO -  SUITE-END | took 0s
10:02:39     INFO - Return code: 0
10:02:39    ERROR - No tests run or test summary not found
Whiteboard: [stockwell needswork]
Flags: needinfo?(gbrown)
All TV -- all tier 2 -- failures will not recur on subsequent pushes: nothing to worry about, probably.

Failures are approx 2x normal because a lot of files were modified and with test-verify chunking, that resulted in 2 TV chunks per platform.

Failures are approx 4x (2x x 2x) normal because failures happened both on check-in and back-out.


This appears to be a curious case, in that tests did actually run, on all platforms, but on Windows, "No tests run ..." was reported. Because they were browser-chrome? Something to do with recent evaluate_parser() changes??
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
I am looking into this- quite possibly a "rebase" error on my patches- I should have a fix up for review by end of day tomorrow.
Assignee: nobody → jmaher
Flags: needinfo?(gbrown)
odd, this is windows only, the android/linux annotations are incorrect on this bug.  I get green on try for linux and reproduce the failure on windows.
Comment hidden (Intermittent Failures Robot)
Duplicate of this bug: 1459718
Comment on attachment 8974104 [details] [diff] [review]
adjust the summarization for the unittest parser

Review of attachment 8974104 [details] [diff] [review]:
-----------------------------------------------------------------

Clearing the review mostly because I don't understand why the value = 1 is needed / how it works. Maybe I just need an explanation.

::: testing/mozharness/mozharness/mozilla/testing/unittest.py
@@ +82,5 @@
>              for key in keys:
>                  joined_summary[key] += previous_summary[key]
>                  value = getattr(self, key) - previous_summary[key]
> +                if value < 0 and getattr(self, key) > 0:
> +                    value = 1

Sorry, but even after our irc chat and thinking more, I'm still confused. What are we doing here? Maybe some examples would help.

@@ +87,5 @@
>                  setattr(self, key, value)
>              self.tbpl_status = TBPL_SUCCESS
>              self.worst_log_level = INFO
> +            self.crashed = False
> +            self.leaked = False

Not harmful, but this parser doesn't have 'crashed' and 'leaked' attributes currently.

The base class, OutputParser, has 'num_errors', 'num_warnings' etc. Do they need to be reset?
Attachment #8974104 - Flags: review?(gbrown)
Comment hidden (Intermittent Failures Robot)
Summary: Intermittent [test-verify] No tests run or test summary not found → Intermittent [test-verify] [test-coverage] No tests run or test summary not found
Duplicate of this bug: 1460139
here are some try runs:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jmaher@mozilla.com&fromchange=7308952d71925eb08b6484d900fc5003fc3eba9d&group_state=expanded&tochange=26ebb4be072a4d42a01145d975de5886605501ca

the one on the bottom is all the tv jobs, the one on the top is all the linux jobs to ensure I didn't break any parsing.

Overall the changes are:
* remove use of previous_summary for DesktopParser as it was not parsing the full accumulated log
* remove all work in TestSummary parser as it is for Marionette only
* adjust the WPT verify code to use a previous_summary in a named variable so we don't confuse it with success_codes (which caused return_code=0 to fail)
Attachment #8974104 - Attachment is obsolete: true
Attachment #8974531 - Flags: review?(gbrown)
Comment on attachment 8974531 [details] [diff] [review]
adjust the summarization for the unittest parser and fix web-platform-tests TVw

Review of attachment 8974531 [details] [diff] [review]:
-----------------------------------------------------------------

Wow, much simpler! Hopefully that's all that's needed.
Attachment #8974531 - Flags: review?(gbrown) → review+
Comment hidden (Intermittent Failures Robot)

Comment 16

Last year
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/193e46b398bf
Test-Verify summarization fails on windows. r=gbrown

Comment 17

Last year
bugherder
https://hg.mozilla.org/mozilla-central/rev/193e46b398bf
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla62
Comment hidden (Intermittent Failures Robot)
Duplicate of this bug: 1460798
Comment hidden (Intermittent Failures Robot)
You need to log in before you can comment on or make changes to this bug.