1677896 - Perma Android browsertime KeyError: 'speedometer'

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=322069365&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/M7HMD7jRSg2xcwPCNFoJIQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  raptor-browsertime Info: Wrote data to blobber_upload_dir/browsertime-results/speedometer
[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  raptor-browsertime-android Info: removing reverse socket connections
[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  adb command_output: adb -s HT7A21A02882 wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  perftest-results-handler Info: retrieving browsertime test results
[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  perftest-results-handler Info: found browsertime results at /builds/task_160564119676176/workspace/build/blobber_upload_dir/browsertime-results/speedometer/browsertime.json
[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  perftest-results-handler Info: parsing results from browsertime json
[task 2020-11-17T19:29:53.562Z] 19:29:53     INFO -  perftest-results-handler Info: parsed new benchmark result: {'statistics': {'dcf': {u'p99': 133, u'mdev': 0, u'min': 133, u'p90': 133, u'max': 133, u'median': 133, u'p10': 133, u'stddev': 0, u'mean': 133}, 'fnbpaint': {u'p99': 133, u'mdev': 0, u'min': 133, u'p90': 133, u'max': 133, u'median': 133, u'p10': 133, u'stddev': 0, u'mean': 133}, 'loadtime': {u'p99': 132, u'mdev': 0, u'min': 132, u'p90': 132, u'max': 132, u'median': 132, u'p10': 132, u'stddev': 0, u'mean': 132}}, 'name': u'speedometer', 'url': (u'http://127.0.0.1:39477/Speedometer/index.html?raptor',), 'lower_is_better': False, 'measurements': {'dcf': [133], 'fnbpaint': [133], 'loadtime': [132]}, 'subtest_unit': u'ms', 'extra_options': ['nocondprof', 'fenix'], 'bt_ver': u'9.4.1', 'alert_threshold': u'2.0', 'subtest_lower_is_better': True, 'cold': False, 'type': 'benchmark', 'unit': u'score', 'browser': {u'userAgent': u'Mozilla/5.0 (Android 8.0.0; Mobile; rv:84.0) Gecko/84.0 Firefox/84.0', u'windowSize': u'414x664'}}
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -  perftest-output Info: preparing browsertime results for output
[task 2020-11-17T19:29:53.563Z] 19:29:53    ERROR -  Traceback (most recent call last):
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/raptor.py", line 205, in <module>
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      main()
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/raptor.py", line 151, in main
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      success = raptor.run_tests(raptor_test_list, raptor_test_names)
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/browsertime/android.py", line 195, in run_tests
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      return super(BrowsertimeAndroid, self).run_tests(tests, test_names)
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/perftest.py", line 399, in run_tests
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      return self.process_results(tests, test_names)
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/perftest.py", line 427, in process_results
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      return self.results_handler.summarize_and_output(self.config, tests, test_names)
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/results.py", line 810, in summarize_and_output
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      output.summarize(test_names)
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/output.py", line 1533, in summarize
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      subtests, vals = self.parseSpeedometerOutput(test)
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -    File "/builds/task_160564119676176/workspace/build/tests/raptor/raptor/output.py", line 455, in parseSpeedometerOutput
[task 2020-11-17T19:29:53.563Z] 19:29:53     INFO -      data = test["measurements"]["speedometer"]
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO -  KeyError: 'speedometer'
[task 2020-11-17T19:30:04.579Z] 19:29:53    ERROR - Return code: 1
[task 2020-11-17T19:30:04.579Z] 19:29:53  WARNING - setting return code to 1
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO - Killing logcat pid 795.
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO - Copying Raptor results to upload dir:
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO - /builds/task_160564119676176/workspace/build/blobber_upload_dir/perfherder-data.json
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO - Copying raptor results from /builds/task_160564119676176/workspace/build/raptor.json to /builds/task_160564119676176/workspace/build/blobber_upload_dir/perfherder-data.json
[task 2020-11-17T19:30:04.579Z] 19:29:53 CRITICAL - Error copying results /builds/task_160564119676176/workspace/build/raptor.json to upload dir /builds/task_160564119676176/workspace/build/blobber_upload_dir/perfherder-data.json
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO - [Errno 2] No such file or directory: u'/builds/task_160564119676176/workspace/build/raptor.json'
[task 2020-11-17T19:30:04.579Z] 19:29:53     INFO - Running post-action listener: _package_coverage_data```

Comment 1

[Cristina Coroiu [:ccoroiu]]

This is failing on Android 7.0 MotoG5 Shippable opt too:
https://treeherder.mozilla.org/logviewer?job_id=322161119&repo=mozilla-central&lineNumber=1902

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1677896 - Remove license header from raptor benchmark script. r?#perftest

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/561b182ae314
Remove license header from raptor benchmark script. r=perftest-reviewers,tarek

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/561b182ae314ca60c36b20fbe4e79b6a48328085

Comment 6

[Intermittent Failures Robot]

15 failures in 888 pushes (0.017 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 15

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-shippable: 5
• android-hw-g5-7-0-arm7-api-16-shippable: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1677896&startday=2020-11-18&endday=2020-11-18&tree=all

Comment 7

[Intermittent Failures Robot]

20 failures in 4810 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 20

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-shippable: 10
• android-hw-g5-7-0-arm7-api-16-shippable: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1677896&startday=2020-11-16&endday=2020-11-22&tree=all