Perma Android-hw Raptor failures Exception: org.mozilla.fennec_aurora is not installed when Gecko 66 merges to Beta on 2019-01-21

RESOLVED FIXED in Firefox 66

Status

defect
P1
normal
RESOLVED FIXED
5 months ago
4 months ago

People

(Reporter: CosminS, Assigned: rwood)

Tracking

Version 3
mozilla66
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox64 unaffected, firefox65 unaffected, firefox66+ fixed, firefox67 fixed)

Details

Attachments

(3 attachments, 1 obsolete attachment)

Reporter

Description

5 months ago
[Tracking Requested - why for this release]:

Central as beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&revision=8862dbd18dec54761197bf5433fec673accc5310&searchStr=raptor&selectedJob=219958706

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219958706&repo=try&lineNumber=1000

13:10:04     INFO -  adb Ignoring attempt to chmod external storage
13:10:04     INFO -  adb shell_output: adb -s HT85S1A02429 wait-for-device shell pm list package org.mozilla.fennec_aurora; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
13:10:04    ERROR -  Traceback (most recent call last):
13:10:04     INFO -    File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 646, in <module>
13:10:04     INFO -      main()
13:10:04     INFO -    File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 616, in main
13:10:04     INFO -      raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
13:10:04     INFO -    File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 275, in run_test
13:10:04     INFO -      raise Exception('%s is not installed' % self.config['binary'])
13:10:04     INFO -  Exception: org.mozilla.fennec_aurora is not installed
13:10:04    ERROR - Return code: 1
13:10:04  WARNING - setting return code to 1
13:10:04     INFO - Killing logcat pid 426.

This looks to be a regression from https://hg.mozilla.org/mozilla-central/rev/8ad51093f3f0. Robert, could you please take a look? Thank you.
Flags: needinfo?(rwood)
Reporter

Updated

5 months ago
Blocks: 1511356
Summary: Perma Android-hw Raptor failures Exception: org.mozilla.fennec_aurora is not installed → Perma Android-hw Raptor failures Exception: org.mozilla.fennec_aurora is not installed when Gecko 66 merges to Beta on 2019-01-21
Assignee

Comment 1

5 months ago
:gbrown, are any other tests run on fennec on beta? Any idea why fennec wouldn't be installed on the beta builds? Thanks.
Flags: needinfo?(rwood) → needinfo?(gbrown)
Assignee

Comment 2

5 months ago
Oh wait.. it's the app name isn't it... for beta it's probably called 'org.mozilla.fennec_beta'. Nvm. I'll need to change the expected app name when running on beta.
Flags: needinfo?(gbrown)
Assignee

Updated

5 months ago
Priority: -- → P1
Assignee

Updated

5 months ago
Assignee: nobody → rwood
Status: NEW → ASSIGNED
Comment hidden (Intermittent Failures Robot)

Tracking for 66 to make sure we land a fix before the merge.

Comment 8

4 months ago
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e920686642fa
[raptor] Use the correct app name when running on fennec on beta; r=davehunt

Comment 9

4 months ago
bugherder
Status: ASSIGNED → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66

This is still occurring: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221591679&repo=try&lineNumber=986

Central as beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=1c6cd4f780f78385d99f72b6469058b59b1b840c&selectedJob=221591679

13:30:47 INFO - adb Ignoring attempt to chmod external storage
13:30:48 INFO - adb shell_output: adb -s HT85S1A02429 wait-for-device shell pm list package org.mozilla.fennec_aurora; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
13:30:48 ERROR - Traceback (most recent call last):
13:30:48 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 654, in <module>
13:30:48 INFO - main()
13:30:48 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 624, in main
13:30:48 INFO - raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
13:30:48 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 275, in run_test
13:30:48 INFO - raise Exception('%s is not installed' % self.config['binary'])
13:30:48 INFO - Exception: org.mozilla.fennec_aurora is not installed
13:30:48 ERROR - Return code: 1
13:30:48 WARNING - setting return code to 1
13:30:48 INFO - Killing logcat pid 428.
13:30:48 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
13:30:48 INFO - copying raptor results to upload dir:
13:30:48 INFO - /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
13:30:48 INFO - copying raptor results from /builds/worker/workspace/build/raptor.json to /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
13:30:48 CRITICAL - Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
13:30:48 INFO - [Errno 2] No such file or directory: u'/builds/worker/workspace/build/raptor.json'
13:30:48 INFO - Running post-action listener: _package_coverage_data
13:30:48 INFO - Running post-action listener: _resource_record_post_action
13:30:48 INFO - Running post-action listener: process_java_coverage_data
13:30:48 INFO - Running post-action listener: stop_device
13:30:48 INFO - Killing logcat pid 428.

Flags: needinfo?(rwood)
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 13

4 months ago

(In reply to Robert Wood [:rwood] from comment #12)

Created attachment 9036342 [details]
Bug 1517738 - Set correct binary name when running on fennec on beta;
r?davehunt

Thanks :dvarga, sorry that fix didn't do the trick. I have made a new fix/patch ^. Is there any way that you could please try this patch on beta before I land it? Or is there a way that I can push to try but use beta, myself?

Status: RESOLVED → REOPENED
Flags: needinfo?(rwood)
Resolution: FIXED → ---
Assignee

Updated

4 months ago
Flags: needinfo?(dvarga)
Reporter

Comment 14

4 months ago

Robert, this is a beta-sim with the latest patch applied. https://treeherder.mozilla.org/#/jobs?repo=try&revision=46a6ecc22b8497aaa0d50725dcd89afc05b07afd Will monitor to see if this still happens.

Flags: needinfo?(dvarga)
Assignee

Comment 15

4 months ago

(In reply to Cosmin Sabou [:CosminS] from comment #14)

Thank you kindly!

Robert, this started occurring: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=c0732327e39ba5e2a2ddc9a71bbb6cef6f8c20a9&selectedJob=222209808

14:06:27 INFO - adb shell_output: adb -s HT85S1A02559 wait-for-device shell pm list package org.mozilla.fennec; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
14:06:27 ERROR - Traceback (most recent call last):
14:06:27 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 654, in <module>
14:06:27 INFO - main()
14:06:27 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 624, in main
14:06:27 INFO - raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
14:06:27 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 283, in run_test
14:06:27 INFO - raise Exception('%s is not installed' % self.config['binary'])
14:06:27 INFO - Exception: org.mozilla.fennec is not installed
14:06:27 ERROR - Return code: 1
14:06:27 WARNING - setting return code to 1
14:06:27 INFO - Killing logcat pid 427.
14:06:27 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
14:06:27 INFO - copying raptor results to upload dir:
14:06:27 INFO - /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
14:06:27 INFO - copying raptor results from /builds/worker/workspace/build/raptor.json to /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
14:06:27 CRITICAL - Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
14:06:27 INFO - [Errno 2] No such file or directory: u'/builds/worker/workspace/build/raptor.json'
14:06:27 INFO - Running post-action listener: _package_coverage_data
14:06:27 INFO - Running post-action listener: _resource_record_post_action
14:06:27 INFO - Running post-action listener: process_java_coverage_data
14:06:27 INFO - Running post-action listener: stop_device
14:06:27 INFO - Killing logcat pid 427.

Flags: needinfo?(rwood)

:rwood - I believe the fennec package name on mozilla-beta is org.mozilla.firefox_beta (and org.mozilla.firefox on mozilla-release, if that is also a concern). I'm pretty sure, but you can download, install and 'adb shell pm list packages org.mozilla' to check.

Assignee

Comment 18

4 months ago

Thank you Andreea, and thanks Geoff. Working on another patch, I should probably pull in the mozharness function you noted in Comment 3 instead.

Flags: needinfo?(rwood)
Assignee

Comment 23

4 months ago

(In reply to Robert Wood [:rwood] from comment #22)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=135e718fad3ecfb38c158acac6a6217e71bd195b

Have a patch ready but the fennec build is broken on try - will keep rebasing and retrying...

Assignee

Comment 26

4 months ago

[task 2019-01-18T16:54:46.798Z] 16:54:46 INFO - 16:54:46 INFO - Multi-locale packaging requires the full (non-artifact) 'RecursiveMake' build backend; got ['FasterMake+RecursiveMake'].
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 ERROR - Return code: 1
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 FATAL - 'mach package-multi-locale --locales an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-CA en-GB en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk kn ko lij lo lt lv mai ml mr ms my nb-NO ne-NP nl nn-NO oc or pa-IN pl pt-BR pt-PT rm ro ru sk sl son sq sr sv-SE ta te th tr trs uk ur uz vi wo xh zam zh-CN zh-TW' did not run successfully. Please check log for errors.
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 FATAL - Running post_fatal callback...
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 FATAL - Exiting -1
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 INFO - [mozharness: 2019-01-18 16:54:46.813638Z] Finished package-multi step (failed)
[task 2019-01-18T16:54:46.820Z] 16:54:46 ERROR - Return code: 255
[task 2019-01-18T16:54:46.820Z] 16:54:46 ERROR - 255 not in success codes: [0]
[task 2019-01-18T16:54:46.821Z] 16:54:46 WARNING - setting return code to 2
[task 2019-01-18T16:54:46.821Z] 16:54:46 FATAL - Halting on failure while running ['/usr/bin/python2.7', 'build/src/testing/mozharness/scripts/multil10n.py', '--config-file', 'multi_locale/mozilla-central_android.json', '--config-file', 'multi_locale/android-mozharness-build.json', '--pull-locale-source', '--package-multi', '--summary']

:gbrown, I'm trying to run raptor fennec speedometer on try but keep getting ^, any ideas what is happening here?

Flags: needinfo?(gbrown)

Android artifact builds are okay, when based on a recent tip of mozilla-central:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0a841a83216f09ea472063f3b1b8df57c8900d62

so it must be related to your push. But certainly your raptor patch wouldn't cause this type of failure. Are you doing something to emulate beta on try?

Assignee

Comment 28

4 months ago

(In reply to Geoff Brown [:gbrown] from comment #27)

so it must be related to your push. But certainly your raptor patch wouldn't cause this type of failure. Are you doing something to emulate beta on try?

Thanks Geoff. No, just trying to run raptor speedometer fennec as/is on try, hmm I wonder if pulling in the AndroidMixin at that point has caused some issue but I really can't see how.

Your try push requested speedometer-fennec, which looks like it runs against Nightly builds. Maybe there is a problem with Nightly builds on try, or Nightly artifact builds? I don't know much about Nightly interaction with try, but it seems likely this is the source of the issue.

Let's see if these help us understand:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4b42654d4bab4437e6bd93ace1f728bd8f3a18a
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a59480baf4a1bcbf32241af055c17850e7ae4bd

Flags: needinfo?(gbrown)

It looks like the builds fail if the try fuzzy run requests speedometer-fennec, but is okay if it requests speedometer-geckoview tests instead -- even with your patch.

Assignee

Comment 31

4 months ago

(In reply to Geoff Brown [:gbrown] from comment #30)

It looks like the builds fail if the try fuzzy run requests speedometer-fennec, but is okay if it requests speedometer-geckoview tests instead -- even with your patch.

Thanks Geoff. Ok so I can't try this patch out on try, ugh. The merge is Monday. I don't know what to do now since the merge is in 2 days.

:cosmins, would you be able to test my latest patch (Comment 12) on beta? If it passes I'll land it, if it fails and is still red on beta then I'll need to set raptor speedometer fennec to 'tier3' ASAP before the merge.

Flags: needinfo?(csabou)
Reporter

Comment 32

4 months ago

Robert, we used these changes on today beta simulations: https://hg.mozilla.org/try/rev/8b029cb5f772f6757ddf4b3b5ffcbe34555e2bde

Unfortunately it still fails: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=f1d5cf53e4ae4163666857f9cfa8f72ebd81f61f&searchStr=raptor&selectedJob=222873040

If it's not the correct patch I can let the people who are doing tomorrow's beta-sims to import the right one and follow-up if it still fails or not.

Flags: needinfo?(csabou)
See Also: → 1521382
Assignee

Comment 33

4 months ago

(In reply to Cosmin Sabou [:CosminS] from comment #32)

Robert, we used these changes on today beta simulations: https://hg.mozilla.org/try/rev/8b029cb5f772f6757ddf4b3b5ffcbe34555e2bde

Unfortunately it still fails: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=f1d5cf53e4ae4163666857f9cfa8f72ebd81f61f&searchStr=raptor&selectedJob=222873040

If it's not the correct patch I can let the people who are doing tomorrow's beta-sims to import the right one and follow-up if it still fails or not.

Thanks Cosmin, no that wasn't the correct patch. It's the latest patch update from the phabricator link in Comment 12, this rev:

https://phabricator.services.mozilla.com/D16470

Thanks!

Flags: needinfo?(csabou)
Assignee

Updated

4 months ago
Keywords: leave-open

Comment 35

4 months ago
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1cccd0ed64c3
Set Raptor speedometer fennec job to tier3 due to failure on beta; r=davehunt
Assignee

Comment 36

4 months ago

Ah, so looks like the latest patch (Comment 12) was tried on beta and failed, and Bug 1521382 was filed for that.

So it's good I landed earlier today a patch to set raptor speedometer fennec to tier 3.

Flags: needinfo?(csabou)
Assignee

Comment 37

4 months ago

In my patch in Comment 12 I pulled in the mozharness code that was suggested to get the fennec app name but looks like that doesn't work still:

https://bugzilla.mozilla.org/show_bug.cgi?id=1521382#c0

12:15:48 INFO - AttributeError: 'AndroidMixin' object has no attribute 'installer_path'

Is there another part of Mozharness that I need to pull in now as well as AndroidMixin? Or does 'installer_path" just not work on the bitbar env maybe?

I can't seem to solve this issue, any guidance would be appreciated :) In the mean time I've already demoted this test to 'tier 3' so it won't block the beta merge today.

Flags: needinfo?(gbrown)
Flags: needinfo?(bob)

Your mozharness Raptor class should already be an AndroidMixin:

https://searchfox.org/mozilla-central/rev/330daedbeac2bba296d663668e0e0cf248bc6823/testing/mozharness/mozharness/mozilla/testing/raptor.py#48

so I think you can just call self.query_package_name() instead of creating a new AndroidMixin. Hopefully that will allow query_package_name to see installer_path, which does seem to be defined for raptor:

https://searchfox.org/mozilla-central/rev/330daedbeac2bba296d663668e0e0cf248bc6823/testing/mozharness/mozharness/mozilla/testing/raptor.py#459

Flags: needinfo?(gbrown)

Updated

4 months ago
Flags: needinfo?(bob)
Assignee

Comment 42

4 months ago

(In reply to Geoff Brown [:gbrown] from comment #40)

Can you do something like this?

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=716f4c79a4f47f8ff58fcb418fdbe842f84edf8b

Yep that looks like the solution. Thanks!! I'll land that and see how it goes on beta, then promote back to tier 2 if beta is fine.

Attachment #9036342 - Attachment is obsolete: true
Assignee

Comment 48

4 months ago

(In reply to Robert Wood [:rwood] from comment #47)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab052d5b9f4d361af7681a9625b0937b13a04733

Green on try, going to land this.

If this fixes the issue on beta (which it should) then at that time I'll create another patch (in this same bug) to promote Raptor fennec speedometer back up to tier 2.

Comment 49

4 months ago
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/714ae096d76e
Have Raptor use the correct binary name when running on android in production; r=gbrown
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 52

4 months ago

Marking this as fixed; the fix landed for beta. This used to be a perma fail but is no longer; if there's a different intermittent please file it separately, thanks!

Status: REOPENED → RESOLVED
Last Resolved: 4 months ago4 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.