1457012 - Update hostutils for Android tests

Status: RESOLVED DUPLICATE of bug 1514075

(Firefox for Android Graveyard :: Testing, enhancement, P5)

Description

[Kris Maglione [:kmag]]

+++ This bug was initially created as a clone of Bug #1433279 +++

+++ This bug was initially created as a clone of Bug #1415242 +++

This needs to be done again after the first parts of bug 1456035 merge.

Comment 1

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

gbrown: given the rapid refactoring of core parts of XPCOM, maybe it's finally time to make the hostutils a toolchain task (easy for Linux and Windows) and figure out what the story can be for signing with the signing server on macOS (harder).

Alternatively, could we make it so that the things that expect to unpack hostutils can just unpack a Firefox binary directly so that we don't have to repack these bits at all?

Comment 2

[Kris Maglione [:kmag]]

I'd really like to get the rest of bug 1456035 landed in this cycle. I don't want to ship the current half-landed state to release users if I can possibly help it.

Can I get an ETA on this?

Thanks!

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Nick Alexander :nalexander from comment #1)
> gbrown: given the rapid refactoring of core parts of XPCOM, maybe it's
> finally time to make the hostutils a toolchain task (easy for Linux and
> Windows) and figure out what the story can be for signing with the signing
> server on macOS (harder).

I would welcome that and would be happy to help if I can -- but I'm not familiar with toolchain tasks. I think it would be fine to have a linux-only automated solution: that's all we need for success on treeherder. We can fallback to manual updates for macOS to keep mac hosted local tests happy.

> Alternatively, could we make it so that the things that expect to unpack
> hostutils can just unpack a Firefox binary directly so that we don't have to
> repack these bits at all?

We need more than a Firefox installer: some required components are only packaged in the host common.tests.zip. I think a hostutils-consuming task (say, a mochitest run) could download a host (linux/macOS) Firefox install (target.tar.bz2) and a target.common.tests.zip, extract them, and then use that instead of a tooltool-based host-utils archive. I suppose we could eliminate tooltool this way, instead relying on configured links to a host build, but I don't see much advantage to that.

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #2)
> Can I get an ETA on this?

I should be able to put together a linux host-utils update tomorrow.

(I imagine a toolchain solution will take longer - Nick?)

Comment 5

[Geoff Brown [:gbrown]]

There's one more small complication: We need elfhack in host-utils now (at least for treeherder), and elfhack is in neither the firefox installer nor common.tests.zip. It should probably be added to common.tests.zip.

Comment 6

[Geoff Brown [:gbrown]]

:kmag -- This isn't exhaustive, but so far I'm not seeing Android test failures when I push your final patch from bug 
1456035:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c4d14c7215b42f844f2295d374cd2b2e131e74e

Are you sure you need a host-utils update?

Comment 7

[Kris Maglione [:kmag]]

(In reply to Geoff Brown [:gbrown] from comment #6)
> :kmag -- This isn't exhaustive, but so far I'm not seeing Android test
> failures when I push your final patch from bug 
> 1456035:
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=5c4d14c7215b42f844f2295d374cd2b2e131e74e
> 
> Are you sure you need a host-utils update?

I'm pretty sure, yes. The last patch shouldn't be a problem, since none of those files are needed by the mochitest server. The second-to-last patch will be a problem, since it changes files that are used by the mochitest server to user ChromeUtils.generateQI. Removing the stub part 3.1 patch would also break XPCOMUtils.generateQI with the current version of hostutils.

Comment 8

[Geoff Brown [:gbrown]]

Got it. Sorry, I didn't look at your patches closely enough.

Comment 9

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=81175f7151ee1108134d9b0359718e84fcbcbb94

Comment 10

[Geoff Brown [:gbrown]]

Attachment: mkhostutils.sh - script I used to create and upload linux host-utils

Comment 11

[Geoff Brown [:gbrown]]

:bc - Will you update autophone please?

Comment 12

[Bob Clary [:bc] (inactive)]

done. thanks.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/29afe33cb61d
Update Android host-utils for Linux; r=me,a=test-only

Comment 14

[Geoff Brown [:gbrown]]

:nalexander - Bug 1294256 seems like a good place to move forward on the toolchain task idea. You may be our best bet for driving that. :)

Can you also try to deploy a new host-utils for Mac?

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/29afe33cb61d

Comment 16

[Geoff Brown [:gbrown]]

:kmag - You should be able to go ahead with bug 1456035 now.

Comment 17

[Kris Maglione [:kmag]]

(In reply to Geoff Brown [:gbrown] from comment #16)
> :kmag - You should be able to go ahead with bug 1456035 now.

Great. Thanks!

Comment 18

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.

Comment 19

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

(In reply to Geoff Brown [:gbrown] from comment #14)
> :nalexander - Bug 1294256 seems like a good place to move forward on the
> toolchain task idea. You may be our best bet for driving that. :)
> 
> Can you also try to deploy a new host-utils for Mac?

I haven't managed to get to this in 4 months, and I'm pretty sure I no longer have a valid Xcode signing configuration, so this will require another person if and when it's really required.

Comment 20

[Geoff Brown [:gbrown]]

:egao - Can you take this bug?

The initial need here is to deploy a new version of the Android host-utils for linux and osx. For linux, the attached patch should be instructive -- get the parts in place, replace version numbers and paths, and all's well. For osx, it's similar, but requires some additional signing.

In the longer term, we'd like to automate this procedure using a toolchain task. This part is less well defined and will require some investigation.

I'm happy to provide background, answer questions, etc.

Comment 21

[Geoff Brown [:gbrown]]

It looks like elfhack will no longer be required in host-utils - see bug 1499915. :)

Comment 22

[Edwin Takahashi (:egao | infrequent contributor)]

Following the steps here:

https://wiki.mozilla.org/Packaging_Android_host_utilities

This time I will use this build from mozilla-central:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c9fa642e9e3aecaa8852f5c647904eecddaa4450&selectedJob=216676591

Comment 23

[Edwin Takahashi (:egao | infrequent contributor)]

Marking as duplicate of 1514075.