640432 - Develop a CFX XPI repackaging service w/ external API

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Add-on Builder, enhancement, P1)

Description

[Daniel Buchner [:dbuc]]

Prior to a future 1.1 release of the SDK (likely in mid or late Q2) AMO must have the ability to repackage add-ons when new versions are released so add-ons continue to be compatible.

In talking to Justin Scott (Fligtar), the most attractive solution would be for FlightDeck to provide and API that is able to receive a package or XPI, and create a new XPI from it using a version of the SDK specified by the caller.

In observing the performance of the current XPI creation service, such an offering would likely require more hardware. I would like you to include in this investigation a rough estimate for any additional infra you feel you would need (if possible).

The end of March is when I'd like to have initial feedback on this item. If any further clarification on the goals of this ticket are required, feel free to ping me or schedule a call.

Comment 1

[Piotr Zalewa [:zalun]]

I think it's an equivalent of 
1. upload XPI from Dashboard 
2. click on Download
3. delete Add-on

Fairly simple as we already have all functionalities

The only problem is to fix upload to support libraries as well

Comment 2

[Piotr Zalewa [:zalun]]

Problem with private key ...

http://flightdeck.zalewa.info/xpi/repackage/123/awesome/

"Your package.json says our ID is: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack But I don't have a corresponding private key in: /tmp/ePEmWNaWpp/keydir/jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack If you are the original developer of this package and have recently copied the source code from a different machine to this one, you should copy the private key into the file named above"

Comment 3

[Wil Clouser [:clouserw]]

(In reply to comment #2)
> Problem with private key ...
> 
> http://flightdeck.zalewa.info/xpi/repackage/123/awesome/
> 
> "Your package.json says our ID is: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack But
> I don't have a corresponding private key in:
> /tmp/ePEmWNaWpp/keydir/jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack If you are the
> original developer of this package and have recently copied the source code
> from a different machine to this one, you should copy the private key into the
> file named above"

Hmm?  Can you explain what steps led you to that message?

Comment 4

[Piotr Zalewa [:zalun]]

I extract data from install.rdf and package dir, in install.rdf there is an id field which contains a string paired with private key. To create a xpi with given id one has to provide private key as well. I could generate one for each xpi created, but as I understand it is the same as changing the owner of the add-on.

"[...] Otherwise, if you are a new developer who has made a copy of an existing package to use as a starting point, you need to remove the 'id' property from package.json, so that we can generate a new id and keypair. This will disassociate our new package from the old one."

Comment 5

[Wil Clouser [:clouserw]]

Brian/Daniel:  Apparently you guys came up with a solution for the signing problem.  Can you please post an update here?

Comment 6

[Daniel Buchner [:dbuc]]

Signing is a separate matter from generating the ID keys for each XPI. Signing will be a decision that occurs later, but either way will very closely mirror the way we will do ID key generation as described below:

The service shall require the caller to submit with the post data their private key which will be use to generate the XPI that includes an ID key derived from that private key. If no private key is passed, the service will assume the add-on is from the Add-on Builder and look for the private key there.

Additionally, this requires that we let users upload private keys to the Builder, or have it generate one for them and store it.

A benefit sending the private key over the wire as a API parameter is that external applications can use the packaging feature even if they do not use AMO or the Builder for any add-on related services (think an Eclipse or TextMate plugin that creates add-ons with your data).

Please post any questions or requests for clarifications here and Brian or I will be sure to answer them promptly.

Comment 7

[Wil Clouser [:clouserw]]

You're going to require a private key from anyone that is hosting an add-on on AMO?  That's a big change on the AMO side that no one was expecting.

Additionally, you're talking about uploading private keys.  Private keys are one of the most protected resources by people who know what they are doing - I don't think it's good practice to ask people to upload them:  not good for us to be responsible for storing them and not good for end users to be in the habit of giving their private keys to anyone.

Comment 8

[Brian Warner [:warner :bwarner]]

[summary: For now, let's change the JID when transferring an addon between
Flightdeck and the SDK, and not attempt to transfer keys at all.]

So..

We don't have a story yet for signatures in add-ons. (We've tried to get the
discussion started
(https://forums.mozilla.org/addons/viewtopic.php?f=27&t=2939&sid=674a50a331b533d1403c978eee25d481)
but so far nobody's participating. Please jump in there, or in the
corresponding post on the addon-sdk mailing list, and let us know what you
think.)

A year ago, I added some key-handling code to the SDK so that we'd have some
tools available for when we finally do figure out the story:

 1: the SDK creates a keypair for each addon, the first time you do 'cfx xpi'
 2: it derives the JID and addon ID from the public verifying key, writes it
    into package.json
 3: it stores the private signing key in ~/.jetpack/keys/
 4: it complains at appropriate times to teach people to copy or replace keys
    when appropriate, so they don't accidentally delete private keys or leave
    them behind on an old computer

In particular, deriving the JID from the single per-addon key completely
eliminates the common "which key should I use" question, which (in my
experience) is what enables safe design of things like upgrade mechanisms and
secure data storage. With this approach, our future story could include
statements like:

 * XPIs presented to the AMO reviewers include components signed by the addon
   author, not relying upon transport-layer security between the author and
   the AMO upload server
 * updated versions of the .xpi will be signed, and the Add-On Manager can
   check these signatures before installing the update
 * the SDK provides an API module for each addon to publish data in an
   unspoofable way, through a URL that includes the JID


This approach requires that the per-addon private key remain available to the
add-on developer, hence the error message when you run 'cfx xpi' and don't
have the private key.

If we were actually using that key to e.g. sign XPIs during 'cfx xpi', then
XPI generation on Flightdeck would also require the privkey (to maintain the
same JID), and we'd need some mechanism to transfer them from the SDK's
~/.jetpack/keys/ into the flightdeck server. Likewise, if a developer starts
with flightdeck and "goes unplugged" to develop locally with the SDK, they'd
need to be able to download the automatically-generated privkey.

But, in either case, we could just change the JID instead of transferring the
privkey. Especially now, when we don't really have a signing story, it's
probably easiest to declare that the JID must change when you transfer a
project between the SDK and Flightdeck, and not try to build any sort of
key-transfer mechanism.

> Additionally, you're talking about uploading private keys. Private keys are
> one of the most protected resources by people who know what they are doing
> - I don't think it's good practice to ask people to upload them: not good
> for us to be responsible for storing them and not good for end users to be
> in the habit of giving their private keys to anyone.

Yeah, I understand. In this case, the private key is narrowly scoped: its
possessor has the authority to define what qualifies as valid addon code for
one specific JID, e.g. anyone who knows the privkey
"pwltj4d3pp23ybkjwmqsfbausxaukvcljnwbwq3f5r3mtcxwnfga" gets to control an
(imaginary) addon I wrote named "MySuperDuperAddon" with a JID of
"jid0-aWkr4xfT5JCADAg2LXVzPWbzUx4". Depending upon where our signing story
goes, that may mean the right to access stored data under that JID, or the
right to provide upgrades for old versions of that specific addon, or the
right to submit code to AMO review for consideration of replacing a specific
addon.

By uploading the MySuperDuperAddon privkey to Flightdeck, I'd be delegating
those rights to Flightdeck. This is a necessary step if I want Flightdeck to
create XPIs for me (and I want them to have those signature-based security
properties).

I fully agree that we need to teach users to be thoughtful about delegating
authority, and make it clear what sort of power they're giving away (just
like with OAuth tokens). It'd be nice if they didn't have to give up raw key
material to do it, but if "no client-side code" is Flightdeck's mission, then
I don't see any other way to do it.

Other options: I could have Flightdeck create an unsigned XPI and give it to
me for signing, but then I'd have to run SDK code locally, which defeats some
of the purpose of Flightdeck. I could withhold the key and ship unsigned
XPIs, but then they couldn't have those signature-based rights. We could not
use signatures in/on XPIs at all, and not bother with key-derived-JIDs (and
I'm not ruling this out, mind you), but that could tie our hands later,
leaving us with a big design gap as we try to add non-JID-based keys later
and have to answer the question of whether a given signature is by the
"right" key or not.

Comment 9

[Brian Warner [:warner :bwarner]]

Also, see Bug 654588 where I propose that we remove the JID keys from the SDK, which probably moots all of this.

Comment 10

[Piotr Zalewa [:zalun]]

Current repackage service is working as follows (all in celery):

1. download xpi
2. copy SDK to some temp directory (tSDK)
3. unpack xpi into tSDK/packages/
4. generate package.json for all packages
5. generate keypair (by calling cfx xpi)
6. build xpi
7. hack in the right guid
   a. unpack fresh xpi
   b. create new temp zip with content from xpi
   c. remove xpi
   d. rename temp zip as xpi

the xpi is available to download in using the standard download XPI method

I'm not closing the bug as I will need to implement the 1.0b5 features, which will speed up the process - it currently takes 7s to repackage a xpi including download time.

Comment 11

[Piotr Zalewa [:zalun]]

Keypair requirement removed from SDK 1.0b5.
repackaging service is now working

Comment 12

[zx.1416.zx]

Attachment: 1415384496855.jpg

Comment 13

[Daniel Holbert [:dholbert]]

Comment on attachment 8519064 [details]
1415384496855.jpg

zx.1416.zx, please stop messing with flags & attachments on random bugs. If you want to experiment, use http://landfill.bugzilla.org/ , not this production bugzilla instance.