Closed Bug 1517738 Opened 5 years ago Closed 4 years ago

Perma Android-hw Raptor failures Exception: org.mozilla.fennec_aurora is not installed when Gecko 66 merges to Beta on 2019-01-21

Categories

(Testing :: Raptor, defect, P1)

Product:
Testing
Component:
Raptor
Version:
Version 3
Type:
defect

Tracking

(firefox-esr60 unaffected, firefox64 unaffected, firefox65 unaffected, firefox66+ fixed, firefox67 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla66
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox64 --- unaffected
firefox65 --- unaffected
firefox66 + fixed
firefox67 --- fixed

People

(Reporter: CosminS, Assigned: rwood)

References

Details

Attachments

(3 files, 1 obsolete file)

Bug 1517738 - [raptor] Use the correct app name when running on fennec on beta; r?davehunt
47 bytes, text/x-phabricator-request
Details | Review
Bug 1517738 - Set Raptor speedometer fennec job to tier3 due to failure on beta; r?davehunt
47 bytes, text/x-phabricator-request
Details | Review
Bug 1517738 - Have Raptor use the correct binary name when running on android in production; r?gbrown
47 bytes, text/x-phabricator-request
Details | Review 
[Tracking Requested - why for this release]:

Central as beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&revision=8862dbd18dec54761197bf5433fec673accc5310&searchStr=raptor&selectedJob=219958706

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=219958706&repo=try&lineNumber=1000

13:10:04     INFO -  adb Ignoring attempt to chmod external storage
13:10:04     INFO -  adb shell_output: adb -s HT85S1A02429 wait-for-device shell pm list package org.mozilla.fennec_aurora; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
13:10:04    ERROR -  Traceback (most recent call last):
13:10:04     INFO -    File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 646, in <module>
13:10:04     INFO -      main()
13:10:04     INFO -    File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 616, in main
13:10:04     INFO -      raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
13:10:04     INFO -    File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 275, in run_test
13:10:04     INFO -      raise Exception('%s is not installed' % self.config['binary'])
13:10:04     INFO -  Exception: org.mozilla.fennec_aurora is not installed
13:10:04    ERROR - Return code: 1
13:10:04  WARNING - setting return code to 1
13:10:04     INFO - Killing logcat pid 426.

This looks to be a regression from https://hg.mozilla.org/mozilla-central/rev/8ad51093f3f0. Robert, could you please take a look? Thank you.
Flags: needinfo?(rwood)
Blocks: 1511356
Summary: Perma Android-hw Raptor failures Exception: org.mozilla.fennec_aurora is not installed → Perma Android-hw Raptor failures Exception: org.mozilla.fennec_aurora is not installed when Gecko 66 merges to Beta on 2019-01-21
:gbrown, are any other tests run on fennec on beta? Any idea why fennec wouldn't be installed on the beta builds? Thanks.
Flags: needinfo?(rwood) → needinfo?(gbrown)
Oh wait.. it's the app name isn't it... for beta it's probably called 'org.mozilla.fennec_beta'. Nvm. I'll need to change the expected app name when running on beta.
Flags: needinfo?(gbrown)
Priority: -- → P1
Assignee: nobody → rwood
Status: NEW → ASSIGNED

Tracking for 66 to make sure we land a fix before the merge.

tracking-firefox66: ? → +
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e920686642fa
[raptor] Use the correct app name when running on fennec on beta; r=davehunt

https://hg.mozilla.org/mozilla-central/rev/e920686642fa

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox66: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla66

This is still occurring: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221591679&repo=try&lineNumber=986

Central as beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=1c6cd4f780f78385d99f72b6469058b59b1b840c&selectedJob=221591679

13:30:47 INFO - adb Ignoring attempt to chmod external storage
13:30:48 INFO - adb shell_output: adb -s HT85S1A02429 wait-for-device shell pm list package org.mozilla.fennec_aurora; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
13:30:48 ERROR - Traceback (most recent call last):
13:30:48 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 654, in <module>
13:30:48 INFO - main()
13:30:48 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 624, in main
13:30:48 INFO - raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
13:30:48 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 275, in run_test
13:30:48 INFO - raise Exception('%s is not installed' % self.config['binary'])
13:30:48 INFO - Exception: org.mozilla.fennec_aurora is not installed
13:30:48 ERROR - Return code: 1
13:30:48 WARNING - setting return code to 1
13:30:48 INFO - Killing logcat pid 428.
13:30:48 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
13:30:48 INFO - copying raptor results to upload dir:
13:30:48 INFO - /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
13:30:48 INFO - copying raptor results from /builds/worker/workspace/build/raptor.json to /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
13:30:48 CRITICAL - Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
13:30:48 INFO - [Errno 2] No such file or directory: u'/builds/worker/workspace/build/raptor.json'
13:30:48 INFO - Running post-action listener: _package_coverage_data
13:30:48 INFO - Running post-action listener: _resource_record_post_action
13:30:48 INFO - Running post-action listener: process_java_coverage_data
13:30:48 INFO - Running post-action listener: stop_device
13:30:48 INFO - Killing logcat pid 428.

Flags: needinfo?(rwood)

(In reply to Robert Wood [:rwood] from comment #12)

Created attachment 9036342 [details]
Bug 1517738 - Set correct binary name when running on fennec on beta;
r?davehunt

Thanks :dvarga, sorry that fix didn't do the trick. I have made a new fix/patch ^. Is there any way that you could please try this patch on beta before I land it? Or is there a way that I can push to try but use beta, myself?

Status: RESOLVED → REOPENED
Flags: needinfo?(rwood)
Resolution: FIXED → ---
Flags: needinfo?(dvarga)

Robert, this is a beta-sim with the latest patch applied. https://treeherder.mozilla.org/#/jobs?repo=try&revision=46a6ecc22b8497aaa0d50725dcd89afc05b07afd Will monitor to see if this still happens.

Flags: needinfo?(dvarga)

(In reply to Cosmin Sabou [:CosminS] from comment #14)

Thank you kindly!

Robert, this started occurring: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=c0732327e39ba5e2a2ddc9a71bbb6cef6f8c20a9&selectedJob=222209808

14:06:27 INFO - adb shell_output: adb -s HT85S1A02559 wait-for-device shell pm list package org.mozilla.fennec; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
14:06:27 ERROR - Traceback (most recent call last):
14:06:27 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 654, in <module>
14:06:27 INFO - main()
14:06:27 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 624, in main
14:06:27 INFO - raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
14:06:27 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 283, in run_test
14:06:27 INFO - raise Exception('%s is not installed' % self.config['binary'])
14:06:27 INFO - Exception: org.mozilla.fennec is not installed
14:06:27 ERROR - Return code: 1
14:06:27 WARNING - setting return code to 1
14:06:27 INFO - Killing logcat pid 427.
14:06:27 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
14:06:27 INFO - copying raptor results to upload dir:
14:06:27 INFO - /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
14:06:27 INFO - copying raptor results from /builds/worker/workspace/build/raptor.json to /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
14:06:27 CRITICAL - Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
14:06:27 INFO - [Errno 2] No such file or directory: u'/builds/worker/workspace/build/raptor.json'
14:06:27 INFO - Running post-action listener: _package_coverage_data
14:06:27 INFO - Running post-action listener: _resource_record_post_action
14:06:27 INFO - Running post-action listener: process_java_coverage_data
14:06:27 INFO - Running post-action listener: stop_device
14:06:27 INFO - Killing logcat pid 427.

Flags: needinfo?(rwood)

:rwood - I believe the fennec package name on mozilla-beta is org.mozilla.firefox_beta (and org.mozilla.firefox on mozilla-release, if that is also a concern). I'm pretty sure, but you can download, install and 'adb shell pm list packages org.mozilla' to check.

Thank you Andreea, and thanks Geoff. Working on another patch, I should probably pull in the mozharness function you noted in Comment 3 instead.

Flags: needinfo?(rwood)

(In reply to Robert Wood [:rwood] from comment #22)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=135e718fad3ecfb38c158acac6a6217e71bd195b

Have a patch ready but the fennec build is broken on try - will keep rebasing and retrying...

[task 2019-01-18T16:54:46.798Z] 16:54:46 INFO - 16:54:46 INFO - Multi-locale packaging requires the full (non-artifact) 'RecursiveMake' build backend; got ['FasterMake+RecursiveMake'].
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 ERROR - Return code: 1
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 FATAL - 'mach package-multi-locale --locales an ar as ast az be bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-CA en-GB en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk kn ko lij lo lt lv mai ml mr ms my nb-NO ne-NP nl nn-NO oc or pa-IN pl pt-BR pt-PT rm ro ru sk sl son sq sr sv-SE ta te th tr trs uk ur uz vi wo xh zam zh-CN zh-TW' did not run successfully. Please check log for errors.
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 FATAL - Running post_fatal callback...
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 FATAL - Exiting -1
[task 2019-01-18T16:54:46.813Z] 16:54:46 INFO - 16:54:46 INFO - [mozharness: 2019-01-18 16:54:46.813638Z] Finished package-multi step (failed)
[task 2019-01-18T16:54:46.820Z] 16:54:46 ERROR - Return code: 255
[task 2019-01-18T16:54:46.820Z] 16:54:46 ERROR - 255 not in success codes: [0]
[task 2019-01-18T16:54:46.821Z] 16:54:46 WARNING - setting return code to 2
[task 2019-01-18T16:54:46.821Z] 16:54:46 FATAL - Halting on failure while running ['/usr/bin/python2.7', 'build/src/testing/mozharness/scripts/multil10n.py', '--config-file', 'multi_locale/mozilla-central_android.json', '--config-file', 'multi_locale/android-mozharness-build.json', '--pull-locale-source', '--package-multi', '--summary']

:gbrown, I'm trying to run raptor fennec speedometer on try but keep getting ^, any ideas what is happening here?

Flags: needinfo?(gbrown)

Android artifact builds are okay, when based on a recent tip of mozilla-central:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0a841a83216f09ea472063f3b1b8df57c8900d62

so it must be related to your push. But certainly your raptor patch wouldn't cause this type of failure. Are you doing something to emulate beta on try?

(In reply to Geoff Brown [:gbrown] from comment #27)

so it must be related to your push. But certainly your raptor patch wouldn't cause this type of failure. Are you doing something to emulate beta on try?

Thanks Geoff. No, just trying to run raptor speedometer fennec as/is on try, hmm I wonder if pulling in the AndroidMixin at that point has caused some issue but I really can't see how.

Your try push requested speedometer-fennec, which looks like it runs against Nightly builds. Maybe there is a problem with Nightly builds on try, or Nightly artifact builds? I don't know much about Nightly interaction with try, but it seems likely this is the source of the issue.

Let's see if these help us understand:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4b42654d4bab4437e6bd93ace1f728bd8f3a18a
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a59480baf4a1bcbf32241af055c17850e7ae4bd

Flags: needinfo?(gbrown)

It looks like the builds fail if the try fuzzy run requests speedometer-fennec, but is okay if it requests speedometer-geckoview tests instead -- even with your patch.

(In reply to Geoff Brown [:gbrown] from comment #30)

It looks like the builds fail if the try fuzzy run requests speedometer-fennec, but is okay if it requests speedometer-geckoview tests instead -- even with your patch.

Thanks Geoff. Ok so I can't try this patch out on try, ugh. The merge is Monday. I don't know what to do now since the merge is in 2 days.

:cosmins, would you be able to test my latest patch (Comment 12) on beta? If it passes I'll land it, if it fails and is still red on beta then I'll need to set raptor speedometer fennec to 'tier3' ASAP before the merge.

Flags: needinfo?(csabou)

Robert, we used these changes on today beta simulations: https://hg.mozilla.org/try/rev/8b029cb5f772f6757ddf4b3b5ffcbe34555e2bde

Unfortunately it still fails: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=f1d5cf53e4ae4163666857f9cfa8f72ebd81f61f&searchStr=raptor&selectedJob=222873040

If it's not the correct patch I can let the people who are doing tomorrow's beta-sims to import the right one and follow-up if it still fails or not.

Flags: needinfo?(csabou)
See Also: → 1521382

(In reply to Cosmin Sabou [:CosminS] from comment #32)

Robert, we used these changes on today beta simulations: https://hg.mozilla.org/try/rev/8b029cb5f772f6757ddf4b3b5ffcbe34555e2bde

Unfortunately it still fails: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=f1d5cf53e4ae4163666857f9cfa8f72ebd81f61f&searchStr=raptor&selectedJob=222873040

If it's not the correct patch I can let the people who are doing tomorrow's beta-sims to import the right one and follow-up if it still fails or not.

Thanks Cosmin, no that wasn't the correct patch. It's the latest patch update from the phabricator link in Comment 12, this rev:

https://phabricator.services.mozilla.com/D16470

Thanks!

Flags: needinfo?(csabou)
Keywords: leave-open
Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1cccd0ed64c3
Set Raptor speedometer fennec job to tier3 due to failure on beta; r=davehunt

Ah, so looks like the latest patch (Comment 12) was tried on beta and failed, and Bug 1521382 was filed for that.

So it's good I landed earlier today a patch to set raptor speedometer fennec to tier 3.

Flags: needinfo?(csabou)

In my patch in Comment 12 I pulled in the mozharness code that was suggested to get the fennec app name but looks like that doesn't work still:

https://bugzilla.mozilla.org/show_bug.cgi?id=1521382#c0

12:15:48 INFO - AttributeError: 'AndroidMixin' object has no attribute 'installer_path'

Is there another part of Mozharness that I need to pull in now as well as AndroidMixin? Or does 'installer_path" just not work on the bitbar env maybe?

I can't seem to solve this issue, any guidance would be appreciated :) In the mean time I've already demoted this test to 'tier 3' so it won't block the beta merge today.

Flags: needinfo?(gbrown)
Flags: needinfo?(bob)

Your mozharness Raptor class should already be an AndroidMixin:

https://searchfox.org/mozilla-central/rev/330daedbeac2bba296d663668e0e0cf248bc6823/testing/mozharness/mozharness/mozilla/testing/raptor.py#48

so I think you can just call self.query_package_name() instead of creating a new AndroidMixin. Hopefully that will allow query_package_name to see installer_path, which does seem to be defined for raptor:

https://searchfox.org/mozilla-central/rev/330daedbeac2bba296d663668e0e0cf248bc6823/testing/mozharness/mozharness/mozilla/testing/raptor.py#459

Flags: needinfo?(gbrown)
Flags: needinfo?(bob)

(In reply to Geoff Brown [:gbrown] from comment #40)

Can you do something like this?

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=716f4c79a4f47f8ff58fcb418fdbe842f84edf8b

Yep that looks like the solution. Thanks!! I'll land that and see how it goes on beta, then promote back to tier 2 if beta is fine.

Attachment #9036342 - Attachment is obsolete: true

(In reply to Robert Wood [:rwood] from comment #47)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab052d5b9f4d361af7681a9625b0937b13a04733

Green on try, going to land this.

If this fixes the issue on beta (which it should) then at that time I'll create another patch (in this same bug) to promote Raptor fennec speedometer back up to tier 2.

Pushed by rwood@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/714ae096d76e
Have Raptor use the correct binary name when running on android in production; r=gbrown

Marking this as fixed; the fix landed for beta. This used to be a perma fail but is no longer; if there's a different intermittent please file it separately, thanks!

Status: REOPENED → RESOLVED
Closed: 4 years ago → 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.