705214 - Generate beta+release update snippets for Android en-US + all single locales

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Aki Sasaki (not active)]

We were creating en-US and multilocale update snippets for betas and releases for Android, but turned them off since we never used them.  (Android releases all went to the Android Market, and we used the Market's update system rather than AUS.)

However, now we want to:

a) create en-US builds
b) create [many] single locale repacks
c) partner repack a certain (5?) number of those APKs for the Android and possibly Amazon Markets

The apks from (a) and (b) will go on our website, and we will want to publish snippets for them since they won't be updated by any market.

Comment 1

[[e-mail] (:bhearsum)]

07:53 < aki|backnov28> [22:49:48] we might generate everything except the hash, and fill that in during signing
07:53 < aki|backnov28> [22:50:00] hash+filesize i guess

We should look at using the signing-on-demand code that's been worked on in bug 509198. If that were used, you'd be able to generate the proper snippets at build time. I see that Android uses "make package" already, so I _think_ the build system code would "just work" if MOZ_SIGN_CMD was set. I'd be happy to help out with this part of things!

Comment 2

[Aki Sasaki (not active)]

Awesome!
I imagine the signing machine side of things needs some work, since I doubt Catlee added an apk signer to his signing-on-demand box.
On releases, we upload 2 sets of apks:

1) nightly-signed apks (generated by "make package" by default.  These will be testable by our tegras, if/when we add tegra testing to our Android releases.  These go into the unsigned/android/ directory.
2) release-signed apks (generated on signing machine).  These go into the android/ directory.

Do you think that's doable? I imagine that's either two "make upload"s or a single make upload after moving files around on disk.  One tricky bit may be that the nightly-signed apks are named exactly the same as the release-signed apks.

If this isn't doable, we can potentially lose the unsigned/android/ bits, and we lose the ability to install+test on tegras.

Comment 3

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] (back nov 28) from comment #2)
> If this isn't doable, we can potentially lose the unsigned/android/ bits,
> and we lose the ability to install+test on tegras.

On second thought, we should probably verify this on a tegra; this may have changed since the appname for release+beta are now different than for nightlies+aurora.

If we're able to install and test release-signed bits, that saves us a lot of headache in this bug.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] (back nov 28) from comment #2)
> Awesome!
> I imagine the signing machine side of things needs some work, since I doubt
> Catlee added an apk signer to his signing-on-demand box.

Yeah, definitely need an enhancement there. The server code is a bit in flux at the moment, too, with some back and forth with infrasec in terms of authorization and other security measures.

> On releases, we upload 2 sets of apks:
> 
> 1) nightly-signed apks (generated by "make package" by default.  These will
> be testable by our tegras, if/when we add tegra testing to our Android
> releases.  These go into the unsigned/android/ directory.
> 2) release-signed apks (generated on signing machine).  These go into the
> android/ directory.
>
> Do you think that's doable? I imagine that's either two "make upload"s or a
> single make upload after moving files around on disk.  One tricky bit may be
> that the nightly-signed apks are named exactly the same as the
> release-signed apks.

We might be able to do something similar to the MOZ_PKG_PRETTYNAMES stuff in desktop to achieve this in terms of laying out the files and uploading them properly.

There could be another hurdle, though, in that I _think_ we're making the assumption that one job is restricted to signing with one particular type of key (enforced through tokens that are issued by the master). This would get SO much easier if we didn't want the nightly-signed ones.

> If this isn't doable, we can potentially lose the unsigned/android/ bits,
> and we lose the ability to install+test on tegras.

Hmmm, why can't install the release-signed APKs on the Tegras?

Comment 5

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> We might be able to do something similar to the MOZ_PKG_PRETTYNAMES stuff in
> desktop to achieve this in terms of laying out the files and uploading them
> properly.
> 
> There could be another hurdle, though, in that I _think_ we're making the
> assumption that one job is restricted to signing with one particular type of
> key (enforced through tokens that are issued by the master). This would get
> SO much easier if we didn't want the nightly-signed ones.

Nightly signing happens on each build machine, so if I understand this it won't be an issue.  We'd need to a) "make package" as we currently do, to get nightly-signed bits, and "make package" with MOZ_SIGN_CMD set, to get release-signed bits.

> > If this isn't doable, we can potentially lose the unsigned/android/ bits,
> > and we lose the ability to install+test on tegras.
> 
> Hmmm, why can't install the release-signed APKs on the Tegras?

* You can't install a differently-signed apk over a currently installed apk of the same ID/name.  This may not be an issue since I *think* we uninstall completely before installing.
* Last I remember, SUT Agent needs to be able to access the files as the shared ID of the app, with the same signature.  However, this may have changed over the many iterations.

So we need to doublecheck on a staging tegra.

Comment 6

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] (back nov 28) from comment #5)
> (In reply to Ben Hearsum [:bhearsum] from comment #4)
> > Hmmm, why can't install the release-signed APKs on the Tegras?
> 
> * You can't install a differently-signed apk over a currently installed apk
> of the same ID/name.  This may not be an issue since I *think* we uninstall
> completely before installing.
> * Last I remember, SUT Agent needs to be able to access the files as the
> shared ID of the app, with the same signature.  However, this may have
> changed over the many iterations.
> 
> So we need to doublecheck on a staging tegra.

copying ../talos-data/fennec/application.ini to ../talos-data/talos/remoteapp.ini
/builds/tegra-018/talos-data/../error.flg
Remote Device Error: updateApp() call failed - exiting
process killed by signal 15
program finished with exit code -1

So testing the release-signed 9.0b3 on a staging tegra died.
The nightly-signed 9.0b3 installed just fine.

So our options seem to be:
1) We solve this via chmod/chown or by uninstallAppAndReboot() instead of updateApp() in the tegra automation.
2) We can "make package" both the nightly-signed (no change) and the release-signed apks, either by running "make package"+"make upload" twice or by having the release-signed apk named differently.
3) We can not have a nightly-signed apk at all, and give up the ability to test our release bits until we do (1) or (2).

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #6)
> (In reply to Aki Sasaki [:aki] (back nov 28) from comment #5)
> > (In reply to Ben Hearsum [:bhearsum] from comment #4)
> > > Hmmm, why can't install the release-signed APKs on the Tegras?
> > 
> > * You can't install a differently-signed apk over a currently installed apk
> > of the same ID/name.  This may not be an issue since I *think* we uninstall
> > completely before installing.
> > * Last I remember, SUT Agent needs to be able to access the files as the
> > shared ID of the app, with the same signature.  However, this may have
> > changed over the many iterations.
> > 
> > So we need to doublecheck on a staging tegra.
> 
> So our options seem to be:
> 1) We solve this via chmod/chown or by uninstallAppAndReboot() instead of
> updateApp() in the tegra automation.
> 2) We can "make package" both the nightly-signed (no change) and the
> release-signed apks, either by running "make package"+"make upload" twice or
> by having the release-signed apk named differently.
> 3) We can not have a nightly-signed apk at all, and give up the ability to
> test our release bits until we do (1) or (2).

If 1) lets us avoid having multiple packages for the same version of Firefox, that seems great. I don't have all the context here though, so whatever you want to do sounds good to me.

Comment 8

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> > So our options seem to be:
> > 1) We solve this via chmod/chown or by uninstallAppAndReboot() instead of
> > updateApp() in the tegra automation.
> 
> If 1) lets us avoid having multiple packages for the same version of
> Firefox, that seems great. I don't have all the context here though, so
> whatever you want to do sounds good to me.

Ok.
If that's gated on me, I doubt I'll have time before EOQ.
I'm willing to point people in the right direction if someone wants to tackle this.

Comment 9

[Aki Sasaki (not active)]

Filed bug 705807 for signing on demand.
I doubt that will be solved in time for the Dec 20 release, so we need a near term fix for the snippet issue (this bug).

Comment 11

[Aki Sasaki (not active)]

This is mostly done in bug 608432.
What needs to be done:

* partner repack signing (probably in bug 659544)
* documentation

Comment 12

[Aki Sasaki (not active)]

Updated the docs.