1595040 - Raptor power tests are failing on reference browser

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The raptor power tests are perma-failing on reference browser as seen here: https://treeherder.mozilla.org/#/jobs?repo=reference-browser&searchStr=Rap-P

An example of a failure log can be found here: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=274984358&repo=reference-browser&lineNumber=1568

Traceback (most recent call last):
  File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 1758, in <module>
    main()
  File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 1726, in main
    success = raptor.run_tests(raptor_test_list, raptor_test_names)
  File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 1442, in run_tests
    return super(RaptorAndroid, self).run_tests(tests, test_names)
  File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 227, in run_tests
    self.run_test(test, timeout=int(test.get('page_timeout')))
  File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 1481, in run_test
    finish_android_power_test(self, test['name'])
  File "/builds/worker/workspace/build/tests/raptor/raptor/power.py", line 276, in finish_android_power_test
    raptor.os_baseline_data['values'][power_measure]
  File "/builds/worker/workspace/build/tests/raptor/raptor/power.py", line 264, in calculate_pc
    baseline_measure
ZeroDivisionError: float division by zero

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:sparky could you take a look at this?

Comment 2

[Greg Mierzwinski [:sparky]]

This is happening because those tests are running in the main perf pool of devices which can't run power tests. The worker type for the failing tests is gecko-t-bitbar-gw-perf-g5, while the correct one should be gecko-t-bitbar-gw-batt-g5.

Where do these tests come from or where are they defined?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Hmm.. I'm not sure we should be running the power tests against reference browser. The tests are defined in the GitHub project at https://github.com/mozilla-mobile/reference-browser. :jlorenzo may be able to help out troubleshooting why these are running.

Comment 4

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

That's a very good question. As far as I know, r-b has run the power tests since the beginning of raptor on r-b[1]. I don't mind helping to remove them, so more people are getting familiar with taskgraph on mobile[2]

[1] https://github.com/mozilla-mobile/reference-browser/pull/713/files#diff-168d551d2c0830884a5caeb50ae576e9R85
[2] https://johanlorenzo.github.io/blog/2019/10/24/taskgraph-is-now-deployed-to-the-biggest-mozilla-mobile-projects.html

Comment 5

[Robert Wood [:rwood]]

Hey :sparky would you be able to pick this up with :jlorenzo? Thanks!

Comment 6

[Greg Mierzwinski [:sparky]]

:davehunt, do we want to disable them or fix them so that they work?

If we want to fix them, then we need to add this worker-type specification to the speedometer task.

I think the specification would need to be:

    worker-type:
        by-abi:
            # TODO arm32 on arm64
            armeabi-v7a: t-bitbar-gw-batt-g5
            arm64-v8a: t-bitbar-gw-batt-p2

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Greg Mierzwinski [:sparky] from comment #6)

:davehunt, do we want to disable them or fix them so that they work?

Let's disable them. I don't think we need to run power tests on the reference browser. We can always enable them again if this becomes a requirement.

Comment 8

[Greg Mierzwinski [:sparky]]

I've created a PR for this change here: https://github.com/mozilla-mobile/reference-browser/pull/961

Comment 9

[Greg Mierzwinski [:sparky]]

PR has been merged, marking this bug as fixed.