1627434 - Perma Tier 2 raptor-perftest Critical: Failed to run <testname>-fennec68: tabs[0] is undefined

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=296265109&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Pn7YlXHPSwWxPUdU4dzljQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-04-04T11:30:13.917Z] 11:12:17     INFO -  raptor-webext-android Info: starting fennec
[task 2020-04-04T11:30:13.917Z] 11:12:18     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell am force-stop org.mozilla.firefox, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:18     INFO -  adb Granting important runtime permissions to org.mozilla.firefox
[task 2020-04-04T11:30:13.917Z] 11:12:19     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pm grant org.mozilla.firefox android.permission.WRITE_EXTERNAL_STORAGE, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:20     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pm grant org.mozilla.firefox android.permission.READ_EXTERNAL_STORAGE, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:21     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pm grant org.mozilla.firefox android.permission.ACCESS_COARSE_LOCATION, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:22     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pm grant org.mozilla.firefox android.permission.ACCESS_FINE_LOCATION, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:23     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pm grant org.mozilla.firefox android.permission.CAMERA, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:24     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pm grant org.mozilla.firefox android.permission.RECORD_AUDIO, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:13.917Z] 11:12:24     INFO -  adb launch_application: am start -W -n org.mozilla.firefox/org.mozilla.gecko.BrowserApp -a android.intent.action.VIEW --es args "-profile /sdcard/raptor/profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6 --es env2 MOZ_WEBRENDER=0" -d about:blank
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell am start -W -n org.mozilla.firefox/org.mozilla.gecko.BrowserApp -a android.intent.action.VIEW --es args "-profile /sdcard/raptor/profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6 --es env2 MOZ_WEBRENDER=0" -d about:blank, timeout: None, root: False, timedout: None, exitcode: 0, output: Starting: Intent { act=android.intent.action.VIEW dat=about:blank cmp=org.mozilla.firefox/org.mozilla.gecko.BrowserApp (has extras) }
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  Status: ok
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  Activity: org.mozilla.firefox/org.mozilla.gecko.BrowserApp
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  ThisTime: 1087
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  TotalTime: 1087
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  WaitTime: 1114
[task 2020-04-04T11:30:13.917Z] 11:12:26     INFO -  Complete
[task 2020-04-04T11:30:13.917Z] 11:12:27     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell pidof org.mozilla.firefox, timeout: None, root: False, timedout: None, exitcode: 0, output: 11621
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: received webext_status: raptor runner.js is loaded!
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: received webext_status: test name is: raptor-youtube-playback-fennec68-live
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: received webext_status: test settings url is: http://127.0.0.1:35049/json/raptor-youtube-playback-fennec68-live.json
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: received webext_status: starting raptorRunner
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: reading test settings from json/raptor-youtube-playback-fennec68-live.json
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: sent test settings to webext runner
[task 2020-04-04T11:30:13.917Z] 11:12:29     INFO -  raptor-control-server Info: received webext_status: * pausing 1 seconds to let browser settle... *
[task 2020-04-04T11:30:13.917Z] 11:12:30     INFO -  raptor-control-server Info: received webext_status: openinig new tab
[task 2020-04-04T11:30:13.917Z] 11:12:31     INFO -  raptor-control-server Info: received webext_status: opened new empty tab: 2
[task 2020-04-04T11:30:13.917Z] 11:12:31     INFO -  raptor-control-server Info: received webext_status: running 1 pagecycles of http://yttest.prod.mozaws.net/2019/main.html?muted=true&exclude=1,2,9,10,17,18,21,22,26,28,30,32,39,40,47,48,55,56,63,64,71,72,79,80,83,84,89,90,95,96&test_type=playbackperf-test&command=run&raptor=true
[task 2020-04-04T11:30:13.917Z] 11:12:32     INFO -  raptor-control-server Info: received webext_status: begin page cycle 1
[task 2020-04-04T11:30:13.917Z] 11:12:32     INFO -  raptor-control-server Info: received webext_status: found active tab with id 2
[task 2020-04-04T11:30:13.917Z] 11:12:32     INFO -  raptor-control-server Info: received webext_status: update tab 2 for http://yttest.prod.mozaws.net/2019/main.html?muted=true&exclude=1,2,9,10,17,18,21,22,26,28,30,32,39,40,47,48,55,56,63,64,71,72,79,80,83,84,89,90,95,96&test_type=playbackperf-test&command=run&raptor=true
[task 2020-04-04T11:30:13.917Z] 11:12:32     INFO -  raptor-control-server Info: received webext_status: tab 2 updated
[task 2020-04-04T11:30:13.917Z] 11:30:13     INFO -  raptor-control-server Info: received webext_status: results received
[task 2020-04-04T11:30:13.917Z] 11:30:13     INFO -  raptor-control-server Info: received webext_results: {u'expected_browser_cycles': 1, u'subtest_unit': u'score', u'name': u'raptor-youtube-playback-fennec68-live', u'lower_is_better': True, u'measurements': {u'youtube-playbackperf-test': [[{u'PlaybackPerf.H264.144p15@2X': {u'droppedFrames': 0, u'decodedFrames': 225}, u'PlaybackPerf.VP9.720p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.H264.720p30@2X': {u'droppedFrames': 15, u'decodedFrames': 463}, u'PlaybackPerf.H264.1080p30@2X': {u'droppedFrames': 224, u'decodedFrames': 488}, u'PlaybackPerf.H264.480p30@1X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.H264.1080p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.H264.720p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 451}, u'PlaybackPerf.H264.720p30@0.5X': {u'droppedFrames': 0, u'decodedFrames': 451}, u'PlaybackPerf.H264.144p15@1.25X': {u'droppedFrames': 0, u'decodedFrames': 228}, u'PlaybackPerf.H264.360p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.VP9.240p30@1X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.VP9.144p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 452}, u'PlaybackPerf.VP9.144p30@1X': {u'droppedFrames': 0, u'decodedFrames': 452}, u'PlaybackPerf.VP9.1080p60@2X': {u'droppedFrames': 459, u'decodedFrames': 496}, u'PlaybackPerf.VP9.1080p30@1.5X': {u'droppedFrames': 212, u'decodedFrames': 447}, u'PlaybackPerf.H264.240p30@1X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.VP9.480p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.H264.720p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 449}, u'PlaybackPerf.VP9.360p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.VP9.480p30@1X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.H264.360p30@2X': {u'droppedFrames': 9, u'decodedFrames': 461}, u'PlaybackPerf.H264.720p60@2X': {u'droppedFrames': 168, u'decodedFrames': 915}, u'PlaybackPerf.VP9.240p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.H264.360p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 451}, u'PlaybackPerf.H264.720p30@0.25X': {u'droppedFrames': 0, u'decodedFrames': 451}, u'PlaybackPerf.VP9.720p60@0.25X': {u'droppedFrames': 0, u'decodedFrames': 903}, u'PlaybackPerf.VP9.720p60@1.5X': {u'droppedFrames': 423, u'decodedFrames': 915}, u'PlaybackPerf.H264.240p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 457}, u'PlaybackPerf.VP9.480p30@2X': {u'droppedFrames': 13, u'decodedFrames': 453}, u'PlaybackPerf.VP9.1080p30@2X': {u'droppedFrames': 193, u'decodedFrames': 317}, u'PlaybackPerf.H264.1080p60@2X': {u'droppedFrames': 586, u'decodedFrames': 619}, u'PlaybackPerf.H264.720p30@1X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.VP9.360p30@2X': {u'droppedFrames': 7, u'decodedFrames': 455}, u'PlaybackPerf.H264.1080p60@1X': {u'droppedFrames': 525, u'decodedFrames': 876}, u'PlaybackPerf.VP9.240p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 451}, u'PlaybackPerf.H264.480p30@2X': {u'droppedFrames': 15, u'decodedFrames': 451}, u'PlaybackPerf.H264.720p60@1X': {u'droppedFrames': 0, u'decodedFrames': 905}, u'PlaybackPerf.VP9.720p30@1X': {u'droppedFrames': 0, u'decodedFrames': 456}, u'PlaybackPerf.VP9.720p60@0.5X': {u'droppedFrames': 0, u'decodedFrames': 906}, u'PlaybackPerf.H264.144p15@1.5X': {u'droppedFrames': 0, u'decodedFrames': 225}, u'PlaybackPerf.H264.720p60@1.25X': {u'droppedFrames': 16, u'decodedFrames': 893}, u'PlaybackPerf.H264.480p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 443}, u'PlaybackPerf.H264.240p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 457}, u'PlaybackPerf.H264.480p30@1.5X': {u'droppedFrames': 1, u'decodedFrames': 449}, u'PlaybackPerf.H264.1080p30@1X': {u'droppedFrames': 0, u'decodedFrames': 452}, u'PlaybackPerf.VP9.240p30@2X': {u'droppedFrames': 4, u'decodedFrames': 455}, u'PlaybackPerf.VP9.720p60@2X': {u'droppedFrames': 762, u'decodedFrames': 976}, u'PlaybackPerf.VP9.720p60@1X': {u'droppedFrames': 1, u'decodedFrames': 906}, u'PlaybackPerf.VP9.1080p60@1.25X': {u'droppedFrames': 316, u'decodedFrames': 897}, u'PlaybackPerf.VP9.720p30@2X': {u'droppedFrames': 19, u'decodedFrames': 456}, u'PlaybackPerf.H264.720p60@1.5X': {u'droppedFrames': 28, u'decodedFrames': 911}, u'PlaybackPerf.VP9.1080p60@1X': {u'droppedFrames': 357, u'decodedFrames': 687}, u'PlaybackPerf.VP9.144p30@2X': {u'droppedFrames': 9, u'decodedFrames': 454}, u'PlaybackPerf.H264.144p15@1X': {u'droppedFrames': 0, u'decodedFrames': 226}, u'PlaybackPerf.H264.360p30@1X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.VP9.720p30@1.5X': {u'droppedFrames': 1, u'decodedFrames': 453}, u'PlaybackPerf.VP9.144p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 450}, u'PlaybackPerf.H264.1080p60@1.5X': {u'droppedFrames': 846, u'decodedFrames': 918}, u'PlaybackPerf.VP9.1080p60@1.5X': {u'droppedFrames': 697, u'decodedFrames': 901}, u'PlaybackPerf.VP9.1080p30@1X': {u'droppedFrames': 96, u'decodedFrames': 453}, u'PlaybackPerf.VP9.720p60@1.25X': {u'droppedFrames': 129, u'decodedFrames': 897}, u'PlaybackPerf.H264.1080p30@1.25X': {u'droppedFrames': 0, u'decodedFrames': 453}, u'PlaybackPerf.H264.1080p60@1.25X': {u'droppedFrames': 294, u'decodedFrames': 908}, u'PlaybackPerf.VP9.360p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 446}, u'PlaybackPerf.VP9.360p30@1X': {u'droppedFrames': 0, u'decodedFrames': 454}, u'PlaybackPerf.VP9.480p30@1.5X': {u'droppedFrames': 0, u'decodedFrames': 451}, u'PlaybackPerf.H264.240p30@2X': {u'droppedFrames': 8, u'decodedFrames': 464}, u'PlaybackPerf.VP9.1080p30@1.25X': {u'droppedFrames': 31, u'decodedFrames': 443}}]]}, u'browser_cycle': u'1', u'subtest_lower_is_better': True, u'cold': False, u'type': u'benchmark', u'page': u'http://yttest.prod.mozaws.net/2019/main.html?muted=true&exclude=1,2,9,10,17,18,21,22,26,28,30,32,39,40,47,48,55,56,63,64,71,72,79,80,83,84,89,90,95,96&test_type=playbackperf-test&command=run&raptor=true', u'unit': u'score', u'alert_threshold': 2}
[task 2020-04-04T11:30:13.917Z] 11:30:13     INFO -  perftest-results-handler Info: received results in RaptorResultsHandler.add
[task 2020-04-04T11:30:41.973Z] 11:30:13     INFO -  raptor-control-server Info: received webext_error: tabs[0] is undefined
[task 2020-04-04T11:30:41.973Z] 11:30:13     INFO -  raptor-control-server Info: received request to shutdown the browser
[task 2020-04-04T11:30:41.973Z] 11:30:13     INFO -  raptor-control-server Info: shutting down android app org.mozilla.firefox
[task 2020-04-04T11:30:41.973Z] 11:30:15     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell am force-stop org.mozilla.firefox, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb shell_bool: adb -s ZY322LDKCW wait-for-device shell test -d /sdcard/raptor/profile/minidumps, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell sync, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb shell_bool: adb -s ZY322LDKCW wait-for-device shell test -d /sdcard/raptor/profile/minidumps, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb command_output: adb -s ZY322LDKCW wait-for-device pull /sdcard/raptor/profile/minidumps /tmp/tmppmaD5k/minidumps, timeout: None, timedout: None, exitcode: 0, output: /sdcard/raptor/profile/minidumps/: 0 files pulled, 0 skipped.
[task 2020-04-04T11:30:41.973Z] 11:30:16 CRITICAL -  raptor-perftest Critical: Failed to run raptor-youtube-playback-fennec68-live: tabs[0] is undefined
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  Stack:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  getCurrentTabId@moz-extension://777d7714-e087-4796-bafd-363d4ab4e460/runner.js:255:3
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  raptor-webext-android Info: removing reverse socket connections
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb command_output: adb -s ZY322LDKCW wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  raptor-webext-android Info: skipping check_for_crashes: application has not been launched
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  raptor-webext Info: removing webext /builds/task_1585996647/workspace/build/tests/raptor/raptor/webextension/../../webext/raptor
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  raptor-webext-android Info: removing test folder for raptor: /sdcard/raptor
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell sync, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:16     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell rm -r /sdcard/raptor, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:17     INFO -  adb shell_output: adb -s ZY322LDKCW wait-for-device shell sync, timeout: None, root: False, timedout: None, exitcode: 0, output:
[task 2020-04-04T11:30:41.973Z] 11:30:17     INFO -  adb shell_bool: adb -s ZY322LDKCW wait-for-device shell test -e /sdcard/raptor, timeout: None, root: False, timedout: None, exitcode: 1, output:
[task 2020-04-04T11:30:41.973Z] 11:30:17     INFO -  raptor-control-server Info: shutting down control server
[task 2020-04-04T11:30:41.973Z] 11:30:17     INFO -  raptor-webext Info: finished
[task 2020-04-04T11:30:41.973Z] 11:30:17    ERROR - Return code: 1
[task 2020-04-04T11:30:41.973Z] 11:30:17  WARNING - setting return code to 1

Comment 1

[Intermittent Failures Robot]

1 failures in 5015 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627434&startday=2020-03-30&endday=2020-04-05&tree=all

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The failure is in getCurrentTabId:

[task 2020-04-04T11:30:41.973Z] 11:30:16 INFO - getCurrentTabId@moz-extension://777d7714-e087-4796-bafd-363d4ab4e460/runner.js:255:3

It means that the call to queryForTabs({ currentWindow: true, active: true }) which forwards those options to ext.tabs.query() returns an empty array. This API is supported since Firefox 45 so it should work fine for Fennec 68. So why don't we get the current tab here? As it looks like so far this was a one-off failure only. Lets observe the next days.

Comment 4

[Intermittent Failures Robot]

15 failures in 4823 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 11
• android-hw-p2-8-0-android-aarch64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627434&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #2)

It means that the call to queryForTabs({ currentWindow: true, active: true }) which forwards those options to ext.tabs.query() returns an empty array. This API is supported since Firefox 45 so it should work fine for Fennec 68. So why don't we get the current tab here? As it looks like so far this was a one-off failure only. Lets observe the next days.

This is happen constantly now and way too often. Andrew, any idea why this webext API call for Fennec 68 fails? Is there a known bug? Here how the Raptor webext uses it:

const tabs = await queryForTabs({ currentWindow: true, active: true });

And in such a case we get undefined returned. It's only happening with Fennec 68ESR.

Comment 6

[Andrew Swan [:aswan]]

Sorry I have no idea. Especially on 68 where nothing appears to have changed recently

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

It shouldn't have been caused by a recent change for Fennec. I changed the Raptor webext recently (bug 1625892) in how to interact with tabs. And that problem is related to the currently selected one. Hereby the above API call is used to get that one.

So it's more a question if one of the properties (currentWindow or active) are incorrectly used some times on Fennec. Or if the query method gets called if there is a tab transition?

Comment 8

[Noemi Erli[:noemi_erli]]

This failed again in today's merge on Android:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&revision=853b0e791775ce726149209092a003ed5f001b0c&selectedJob=298209480

R/b reveals that this is a perma fail:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&searchStr=7e55ec5b3ae7d3c3e6376b8b10a425310b30d6a3&tochange=853b0e791775ce726149209092a003ed5f001b0c&fromchange=8243d7c3eeaf8f544ad8607d5ebe7a9b7c479286

Comment 9

[Intermittent Failures Robot]

29 failures in 4528 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 29

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 7
• android-hw-g5-7-0-arm7-api-16: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627434&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Finally I was able to get Fennec tests running locally and I can see this failure. I'll have a look what's going on here.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

A test where I can always reproduce it is raptor-scn-power-idle-bg-fennec68. It should put the browser into background, but that's not happening for me. It's still visible on my screen. Not sure if that is a feature which isn't correctly supported for Fennec? How do GeckoView based tests behave? Are those still working? I cannot test myself because I have Android 10 on my phone and testing is blocked due to scoped storage (bug 1631671).

Greg, any idea?

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Also querying for tabs with { currentWindow: true, active: true } doesn't reveal any tab. Querying the current window only, I get two tabs. So something is clearly wrong with the status of the currently selected tab.

Comment 14

[Greg Mierzwinski [:sparky]]

I tested this out on the Android 7 phones and from what I see, once the Fennec app is backgrounded, we lose all defined tabs - everything still works on Fenix though (for instance).

In reality, we have no need to know the active tabs in Raptor during this test (there should only be an about:blank tab). We just need to wait 10 minutes, request app foregrounding after the timer is done, then kill it and the test.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We don't see it in Fenix because we aren't opening a tab during initialization of Raptor:
https://searchfox.org/mozilla-central/rev/41c3ea3ee8eab9ce7b82932257cb80b703cbba67/testing/raptor/webext/raptor/runner.js#743-745

So I assume we simply have to add a check for isBackgroundTest to those linies, and maybe others to make this failure go away.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Yes, to circumvent the problem with Fennec I re-added the caching of the current tab id. Whenever Fennec is in the background there is no active tab being returned. Given that is not a normal situation for users of Fennec, I don't see that this will ever get fixed.

Lets see how that works:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab241a506343a9ac3865c488a5af92dcaf006af4

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1634371 - [raptor] Only create a new initial tab when tests require it.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1627434 - [raptor] Cache current tab id to support Fennec for background tests.

When Raptor moves Fennec into the background querying for
the active tab doesn't return any value. To fix that the
current tab id can be cached, which also doesn't require
us to always have to query for the current tab when updating
or closing it.

Depends on D72032

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/07408b255fec
[raptor] Only create a new initial tab when tests require it. r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/0e9235af0109
[raptor] Cache current tab id to support Fennec for background tests. r=perftest-reviewers,sparky

Comment 20

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/07408b255fec
https://hg.mozilla.org/mozilla-central/rev/0e9235af0109

Comment 21

[Intermittent Failures Robot]

11 failures in 5582 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 11

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 6
• android-hw-p2-8-0-android-aarch64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627434&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 22

[Marian Raiciof [:marauder]]

== Change summary for alert #25716 (as of Sat, 25 Apr 2020 20:21:03 GMT) ==

Improvements:

3% raptor-tp6-office-firefox-cold-mozproxy-replay macosx1014-64-shippable opt 93.00 -> 96.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25716

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Marian Raiciof [:marauder] from comment #22)

== Change summary for alert #25716 (as of Sat, 25 Apr 2020 20:21:03 GMT) ==

Improvements:

3% raptor-tp6-office-firefox-cold-mozproxy-replay macosx1014-64-shippable opt 93.00 -> 96.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=25716

Marian, how does that relate to the perf changes as filed as bug 1633366?

Comment 24

[Marian Raiciof [:marauder]]

Hey Henrik,
There are changesets/patches that cause improvements on some tests and regressions for others.
I think that even the same test can have improvements or regression only on a single platform and maybe the opossite on another.
But i can't tell why is that happening, i don't have the knowledge to tell how a patch/changeset works in background in relation to a test.

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hey Sheriffs, please backout the following changeset, which has been caused a performance regression as covered by bug 1633366.

https://hg.mozilla.org/mozilla-central/rev/07408b255fec

Thanks.

Comment 26

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/45c58c03f69e
Backed out changeset 07408b255fec as requested by whimboo for causing bug 1633366

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is still fixed. The patch was optional.

Comment 28

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged: https://hg.mozilla.org/mozilla-central/rev/45c58c03f69e

Comment 29

[Alexandru Michis [:malexandru]]

Whimboo, this is still happening on central, though intermittently:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&revision=0f9c5a59e45d34f53ab4d44058ed016d4bacf942&searchStr=android%2C7.0%2Cmotog5%2Craptor%2Cperformance%2Ctests%2Con%2Cfirefox%2Cwithout%2Ce10s&selectedTaskRun=Catv8yMmSMe7nJUK1oZxnw-0

Should we make a new general bug, or reopen this one?

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The failure message is different. So I don't see why it has been classified against this bug.

Comment 31

[Andreea Pavel [:apavel]]

Filled bug 1634909 and corrected the classifications.