1462434 - Intermittent KeyError: '.html',KeyError: '.js'

Status: RESOLVED FIXED

(Testing :: Code Coverage, defect)

Description

[Natalia Csoregi [:nataliaCs]]

Failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=24bae072acb09114c367e6b9ffde9261b2ad8a58&selectedJob=179014680 

Task details: https://tools.taskcluster.net/groups/D2k2vhpfStOenL1nY2GZ_A/tasks/KHOVPgG-RqKoeNKUcr-i4A/details

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=179014680&repo=mozilla-central&lineNumber=7894

17:22:43     INFO - Did not find a baseline test for: devtools/client/inspector/markup/test/browser_markup_events_react_development_15.4.1_jsx.js
17:22:43    ERROR - Exception during post-action for run-tests: Traceback (most recent call last):
17:22:43    ERROR -   File "Z:\task_1526575675\mozharness\mozharness\base\script.py", line 2027, in run_action
17:22:43    ERROR -     method(action, success=success and self.return_code == 0)
17:22:43    ERROR -   File "Z:\task_1526575675\mozharness\mozharness\mozilla\testing\codecoverage.py", line 417, in _package_coverage_data
17:22:43    ERROR -     baseline_coverage = baseline_tests_cov['.js']
17:22:43    ERROR - KeyError: '.js'
17:22:43     INFO - Running post-action listener: _resource_record_post_action
17:22:43     INFO - [mozharness: 2018-05-17 17:22:43.516000Z] Finished run-tests step (success)
17:22:43    FATAL - Aborting due to failure in post-action listener.
17:22:43    FATAL - Running post_fatal callback...
17:22:43    FATAL - Exiting -1
17:22:43     INFO - Running post-run listener: _resource_record_post_run
17:22:43     INFO - Validating Perfherder data against Z:\task_1526575675\mozharness\external_tools\performance-artifact-schema.json
17:22:43     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"subtests": [{"name": "cpu_percent", "value": 36.75838815789475}, {"name": "io_write_bytes", "value": 32678787584}, {"name": "io.read_bytes", "value": 28399104}, {"name": "io_write_time", "value": 720}, {"name": "io_read_time", "value": 1}], "extraOptions": ["e10s", "taskcluster-c4.2xlarge"], "name": ".overall"}, {"subtests": [{"name": "time", "value": 4.956000089645386}, {"name": "cpu_percent", "value": 14.884374999999999}], "name": ".install"}, {"subtests": [{"name": "time", "value": 0.037000179290771484}], "name": ".stage-files"}, {"subtests": [{"name": "time", "value": 1242.4649999141693}, {"name": "cpu_percent", "value": 36.8422378199835}], "name": ".run-tests"}]}
17:22:43     INFO - Total resource usage - Wall time: 1247s; CPU: 37.0%; Read bytes: 28399104; Write bytes: 32678787584; Read time: 1; Write time: 720
17:22:43     INFO - TinderboxPrint: CPU usage<br/>36.8%
17:22:43     INFO - TinderboxPrint: I/O read bytes / time<br/>28,399,104 / 1
17:22:43     INFO - TinderboxPrint: I/O write bytes / time<br/>32,678,787,584 / 720
17:22:43     INFO - TinderboxPrint: CPU idle<br/>6,222.6 (62.1%)
17:22:43     INFO - TinderboxPrint: CPU system<br/>500.6 (5.0%)
17:22:43     INFO - TinderboxPrint: CPU user<br/>3,252.8 (32.5%)
17:22:43     INFO - install - Wall time: 5s; CPU: 15.0%; Read bytes: 49152; Write bytes: 57126400; Read time: 0; Write time: 1
17:22:43     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
17:22:43     INFO - run-tests - Wall time: 1242s; CPU: 37.0%; Read bytes: 28317184; Write bytes: 32548591104; Read time: 1; Write time: 718
17:22:44     INFO - Running post-run listener: _upload_blobber_files
17:22:44  WARNING - Blob upload gear skipped. Missing cmdline options.
17:22:44     INFO - Running post-run listener: copy_logs_to_upload_dir
17:22:44     INFO - Copying logs to upload dir...
17:22:44     INFO - mkdir: Z:\task_1526575675\build\upload\logs
[taskcluster 2018-05-17T17:22:44.096Z]    Exit Code: 4294967295
[taskcluster 2018-05-17T17:22:44.096Z]    User Time: 0s
[taskcluster 2018-05-17T17:22:44.096Z]  Kernel Time: 0s
[taskcluster 2018-05-17T17:22:44.096Z]    Wall Time: 26m25.214009s
[taskcluster 2018-05-17T17:22:44.096Z]       Result: FAILED

Comment 1

[Marco Castelluccio [:marco]]

:sparky, could you look into this?

Comment 2

[Greg Mierzwinski [:sparky]]

:marco, sure thing. From what I can see so far, it looks like there are 10 tests in the chunk that were run and that the baseline was not one of them. I think it was removed from the list of tests to run (it's limited to 10 tests).

Comment 3

[Intermittent Failures Robot]

4 failures in 639 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 4

Platform breakdown:
* windows10-64-ccov: 2
* linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1462434&startday=2018-05-14&endday=2018-05-20&tree=trunk

Comment 4

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1462434 - Prevent baseline coverage tests from being skipped.

This patch prevents baseline coverage tests from being skipped when too many tests are being run.

Review commit: https://reviewboard.mozilla.org/r/246280/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/246280/

Comment 5

[Greg Mierzwinski [:sparky]]

Here's a try push for the patch which tests test-coverage with 10 tests in TC and TCw: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0266a527732d4ece6f61a9df8b40ca9dc0df6a5

Comment 6

[Marco Castelluccio [:marco]]

For simplicity, we could allow the baseline tests to always run even if we're in test-verify and not test-coverage. After all these tests are very short and are very unlikely to change (so they will rarely be run in test-verify).

Comment 7

[Greg Mierzwinski [:sparky]]

(In reply to Marco Castelluccio [:marco] from comment #6)
> For simplicity, we could allow the baseline tests to always run even if
> we're in test-verify and not test-coverage. After all these tests are very
> short and are very unlikely to change (so they will rarely be run in
> test-verify).

Good point, I'll make this change.

Comment 8

[Greg Mierzwinski [:sparky]]

Comment on attachment 8980302 [details]
Bug 1462434 - Prevent baseline coverage tests from being skipped.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246280/diff/1-2/

Comment 9

[Code Review Bot [:reviewbot]]

Comment on attachment 8980302 [details]
Bug 1462434 - Prevent baseline coverage tests from being skipped.

https://reviewboard.mozilla.org/r/246280/#review252906


Code analysis found 2 defects in this patch:
 - 2 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: testing/mozharness/scripts/desktop_unittest.py:876
(Diff revision 2)
> +                executed_too_many_tests = False
>                  for per_test_args in self.query_args(suite):
> +                    # Make sure baseline code coverage tests are never
> +                    # skipped and that having them run has no influence
> +                    # on the max number of actual tests that are to be run.
> +                    is_baseline_test = 'baselinecoverage' in per_test_args[-1] if self.per_test_coverage else False

Error: Line too long (115 > 99 characters) [flake8: E501]

::: testing/mozharness/scripts/web_platform_tests.py:346
(Diff revision 2)
> +            executed_too_many_tests = False
>              for per_test_args in self.query_args(suite):
> +                # Make sure baseline code coverage tests are never
> +                # skipped and that having them run has no influence
> +                # on the max number of actual tests that are to be run.
> +                is_baseline_test = 'baselinecoverage' in per_test_args[-1] if self.per_test_coverage else False

Error: Line too long (111 > 99 characters) [flake8: E501]

Comment 10

[Greg Mierzwinski [:sparky]]

Comment on attachment 8980302 [details]
Bug 1462434 - Prevent baseline coverage tests from being skipped.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/246280/diff/2-3/

Comment 11

[Greg Mierzwinski [:sparky]]

The try runs below are still running.

1) Here's a try run for this patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=653a9614c3b30cb1edf58f8aa8f64b6872a75ece
2) And here's a try run on a browser-chrome test using desktop_unittest.py to make sure everything still works as expected: https://treeherder.mozilla.org/#/jobs?repo=try&revision=035d59ccf78d8f15abbfb279f6c7dc959fbd939a

I had to test for (2) after hitting the following error: https://treeherder.mozilla.org/#/jobs?repo=try&revision=41faa9659f81099353d4e6dd6f724f2137751f77

(The flake8 errors were fixed in the last review push).

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8980302 [details]
Bug 1462434 - Prevent baseline coverage tests from being skipped.

https://reviewboard.mozilla.org/r/246280/#review252922

thanks for working on this- I assume the try run will be fine

Comment 13

[Greg Mierzwinski [:sparky]]

Your assumption was correct, the try runs are fine and we can land it.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f248f9c40e79
Prevent baseline coverage tests from being skipped. r=jmaher

Comment 15

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/f248f9c40e79