1433279 - Update hostutils for Android tests

Status: RESOLVED DUPLICATE of bug 1514075

(Firefox for Android Graveyard :: Testing, enhancement, P5)

Description

[Kris Maglione [:kmag]]

+++ This bug was initially created as a clone of Bug #1415242 +++

The last three patches in bug 1431533 require the first two being present in host-utils before they can land. The mochitest server fails to run on an older xpcshell with those patches applied since the ChromeUtils methods it relies on are missing.

Comment 1

[Bob Clary [:bc] (inactive)]

gbrown: I need to update Autophone as well?

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Bob Clary [:bc:] from comment #1)
> gbrown: I need to update Autophone as well?

I think we should keep host-utils consistent between Autophone and the emulator tests -- just for our sanity and avoiding surprises. I do not know if any autophone tests will fail if bug 1431533 lands without a host-utils update. I don't see any Autophone failures in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=bb369804a51e7665c0b44d3778681ca132cb1c2c

It looks like the last time I updated host-utils, you updated autophone in bug 1415368.

Comment 3

[Geoff Brown [:gbrown]]

I will follow 

https://wiki.mozilla.org/Packaging_Android_host_utilities

this time based on builds from 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=bb369804a51e7665c0b44d3778681ca132cb1c2c

Comment 4

[Geoff Brown [:gbrown]]

Attachment: update tooltool manifests for android host-utils

I created new host-utils archives based on :kmag's builds from his inbound push. That seems to work fine for the current trunk:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=db1c6a433fc0cf67bee7cc805fbb6b41d51244d5

:kmag -- Your patches would not apply cleanly for me, so I could not verify that this host-utils addresses your problem. I'll leave that to you.

Comment 5

[Geoff Brown [:gbrown]]

:nalexander -- Can you help us out with host-utils for mac, just like you did in bug 1415242?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8945877 [details] [diff] [review]
update tooltool manifests for android host-utils

Review of attachment 8945877 [details] [diff] [review]:
-----------------------------------------------------------------

thanks

Comment 7

[Kris Maglione [:kmag]]

I started a try run with my patches applied on top of yours, and it looks good:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=73e7aaa682b8a002729d47071ef0cf57d0aa9363

It's not finished yet, but when I landed with the old host-utils, all of the tests had timed out within 15 minutes, and they're past that point now.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/df7e64bf0966
Update tooltool manifests for Android host-utils (linux); r=jmaher

Comment 9

[Geoff Brown [:gbrown]]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #7)
> I started a try run with my patches applied on top of yours, and it looks
> good:
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=73e7aaa682b8a002729d47071ef0cf57d0aa9363
> 
> It's not finished yet, but when I landed with the old host-utils, all of the
> tests had timed out within 15 minutes, and they're past that point now.

Thanks Kris. Hopefully this host-utils update will work for you. Be aware that there were some robocop failures on your try push which I don't normally see.


leave-open for :bc to follow-up for autophone and :nalexander to follow-up for mac host-utils for local testing on mac. (afaik, neither is required to unblock :kmag).

Comment 10

[Bob Clary [:bc] (inactive)]

did the autophone update dance using the target.tar.bz2, target.common.tests.zip from the same revision as gbrown's push.

Comment 11

[Kris Maglione [:kmag]]

> Thanks Kris. Hopefully this host-utils update will work for you. Be aware
> that there were some robocop failures on your try push which I don't normally
> see.

I noticed, thanks. Those weren't related to the host-utils issues, and I
already fixed them locally.

Comment 12

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/df7e64bf0966

Comment 13

[Geoff Brown [:gbrown]]

These will be auto-updated in local test environments as needed:

$ ./mach reftest layout/reftests/reftest-sanity
Your host utilities are out of date!
You have host-utils-58.0a1.en-US-linux-x86_64 installed, but host-utils-60.0a1.en-US-linux-x86_64 is available
Update host utilities? (Y/n) 
Installing host utilities. This may take a while...

Comment 14

[:Gijs (he/him)]

Is this intentionally still open?

Comment 15

[Nick Alexander :nalexander [he/him]]

(In reply to :Gijs from comment #14)
> Is this intentionally still open?

Yes, 'cuz I haven't updated the macOS utilities.  I'm hoping to make a toolchain task to do it instead of doing it manually, but haven't gotten there.  If you want this closed, file follow-up and I'll do the work there.

Comment 16

[:Gijs (he/him)]

(In reply to Nick Alexander :nalexander from comment #15)
> (In reply to :Gijs from comment #14)
> > Is this intentionally still open?
> 
> Yes, 'cuz I haven't updated the macOS utilities.  I'm hoping to make a
> toolchain task to do it instead of doing it manually, but haven't gotten
> there.  If you want this closed, file follow-up and I'll do the work there.

Nope, that's fine. I will note that it seems https://bugzilla.mozilla.org/show_bug.cgi?id=1432992 may be running into the same issue so we might need to rev' these again anyway...

Comment 17

[Kris Maglione [:kmag]]

(In reply to :Gijs from comment #16)
> Nope, that's fine. I will note that it seems
> https://bugzilla.mozilla.org/show_bug.cgi?id=1432992 may be running into the
> same issue so we might need to rev' these again anyway...

Heh. I almost commented in that bug that we'd probably run into this issue, but I thought the first parts landed in time to make it into the host-utils update

Comment 18

[:Gijs (he/him)]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #17)
> (In reply to :Gijs from comment #16)
> > Nope, that's fine. I will note that it seems
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1432992 may be running into the
> > same issue so we might need to rev' these again anyway...
> 
> Heh. I almost commented in that bug that we'd probably run into this issue,
> but I thought the first parts landed in time to make it into the host-utils
> update

It's also possible I'm mistaken! I'm just inferring based on the failure descriptions of the trypush from mcrr8.

Comment 19

[Kris Maglione [:kmag]]

No, I think you're right. He landed the first part when I finally got the second part of mine to stick. I just got confused about the timing. So we'll need another update for the rest of that to land :(

Comment 20

[Andrew McCreight [:mccr8]]

To be clear here, I need an update that contains the patch in bug 767640 so I can remove Cu, etc. from various server files. I can file a separate bug for that if you want, but of course it would be silly to bump the version on other OSes to some intermediate state, at this point.

Comment 21

[Geoff Brown [:gbrown]]

In bug 1436183 we are updating linux host-utils again, based on https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=65133e49fbfd5306632301f74be7cd15890bdf9f. Until there's a similar update for macOS, anyone trying to run Android mochi/reftests locally on a macOS host may have trouble.

I would love to see future updates handled by a toolchain task -- thanks for looking into that Nick.

Comment 22

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1433279 - Add native macOS host-utils repack toolchain job.

Review commit: https://reviewboard.mozilla.org/r/220072/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/220072/

Comment 23

[Nick Alexander :nalexander [he/him]]

I dug into the toolchain task part far more than was wise.  The attached review request has WIP on supporting macOS toolchain tasks; it might be useful to somebody (me?) in the future: https://reviewboard.mozilla.org/r/220070/.

Unfortunately, this doesn't succeed; and further, I don't think it really can succeed.  Apple makes it Damn Hard to sign code, but there's a subtle thing happening that isn't really visible in the documentation.  I think the posted instructions only work because I had an Apple Developer profile that links back to trusted Apple root certificates.  (I think I no longer have a valid profile, so I think I can't actually follow the instructions.  I might be incorrect.)  Untangling the snarl of conflicting suggestions about self-signed certificates leads me to believe that even if we can get codesign to accept a self-signed certificate (and I couldn't), nobody would be able to run the signed binaries, 'cuz the self-signed certificate wouldn't chain back to a trusted Apple certificate.

All that means that we probably need to use Mozilla's signing service to sign these test binaries.  Possibly with a certificate even more restricted than the cert that we use for try builds, although that one seems about right.

I would be pleased to be proved wrong.

I didn't do the Linux toolchain task bits: they will be trivial.

Comment 24

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander from comment #23)
> I dug into the toolchain task part far more than was wise.  The attached
> review request has WIP on supporting macOS toolchain tasks; it might be
> useful to somebody (me?) in the future:
> https://reviewboard.mozilla.org/r/220070/.
> 
> Unfortunately, this doesn't succeed; and further, I don't think it really
> can succeed.  Apple makes it Damn Hard to sign code, but there's a subtle
> thing happening that isn't really visible in the documentation.  I think the
> posted instructions only work because I had an Apple Developer profile that
> links back to trusted Apple root certificates.  (I think I no longer have a
> valid profile, so I think I can't actually follow the instructions.  I might
> be incorrect.)  Untangling the snarl of conflicting suggestions about
> self-signed certificates leads me to believe that even if we can get
> codesign to accept a self-signed certificate (and I couldn't), nobody would
> be able to run the signed binaries, 'cuz the self-signed certificate
> wouldn't chain back to a trusted Apple certificate.
> 
> All that means that we probably need to use Mozilla's signing service to
> sign these test binaries.  Possibly with a certificate even more restricted
> than the cert that we use for try builds, although that one seems about
> right.
> 
> I would be pleased to be proved wrong.
> 
> I didn't do the Linux toolchain task bits: they will be trivial.

This bit Alex Gaynor today, in the form of a "TypeError: ChromeUtils.import is not a function" message _in the mochitest browser window_ (with no stack) on Mac.

I can't produce a new hostutils for Mac since my (Mozilla) signing certificates have expired -- I'm no longer an iOS developer.  We could try to get an iOS developer who can do the signing... or we can solve the real problem by signing in automation.

Comment 25

[Geoff Brown [:gbrown]]

Today I think :bdahl is hitting a similar issue with:

MochitestServer : launching [u'/Users/bdahl/.mozbuild/android-device/host-utils.37.0a2.en-US.mac/xpcshell', '-g', '/Users/bdahl/.mozbuild/android-device/host-utils.37.0a2.en-US.mac', '-f', '/Users/bdahl/.mozbuild/android-device/host-utils.37.0a2.en-US.mac/components/httpd.js', '-e', "const _PROFILE_PATH = '/var/folders/9r/c_7dd1810vl474__5fss971m0000gn/T/tmpcK76SH.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.251.25.9'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/Users/bdahl/projects/gecko-hg-android/obj-arm-linux-androideabi/_tests/testing/mochitest/server.js']
runtests.py | Server pid: 66427
runtests.py | Websocket server pid: 66428
runtests.py | SSL tunnel pid: 66429
Failed to init NSS:
[warn] kq_init: detected broken kqueue; not using.: Undefined error: 0

Comment 26

[Nick Alexander :nalexander [he/him]]

⋊> ~/M/build-tooltool on master ⨯ tar cfJ host-utils-61.0a1.en-US.mac.tar.xz host-utils-61.0a1.en-US.mac
⋊> ~/M/build-tooltool on master ⨯ python tooltool.py add --visibility public host-utils-61.0a1.en-US.mac.tar.xz
⋊> ~/M/build-tooltool on master ⨯ python tooltool.py upload --authentication-file=auth --message "Bug 1433279 - Update hostutils for Android tests." --manifest=manifest.tt
INFO - host-utils-61.0a1.en-US.mac.tar.xz: starting upload
INFO - host-utils-61.0a1.en-US.mac.tar.xz: uploaded
INFO - notifying server of upload completion for host-utils-61.0a1.en-US.mac.tar.xz
WARNING - Waiting 36 seconds for upload URLs to expire
⋊> ~/M/build-tooltool on master ⨯ cat manifest.tt                                                                                                                                                                              11:05:40
[
  {
    "size": 52350220,
    "visibility": "public",
    "digest": "f8f86d8f2b8f754a79404dc38bf1d12419923d3d496cbd896a979658083b676f2eeda3bf03af21c5ea6389e8ec63459bf8e51bb093e99ef4d67b2a92a3160994",
    "algorithm": "sha512",
    "filename": "host-utils-61.0a1.en-US.mac.tar.xz"
  }
]

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7fa58c628522
Update tooltool manifest for Mac host-utils for Android tests; r=me,a=test-only

Comment 28

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/7fa58c628522

Comment 29

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.

Comment 30

[Geoff Brown [:gbrown]]

We've updated host-utils again recently, and I've noted this bug's wip patch in bug 1294256.