920081 - Create a management command for directly submitting packaged apps for internal use

Status: RESOLVED FIXED

(Marketplace Graveyard :: General, defect, P3)

Description

[Matt Basta [:basta]]

This API will be used for making the packaged dev marketplace possible.

The API would involve a single endpoint. For instance:

/api/v1/apps/direct/(id|slug)/

The client would PUT a zip file to the endpoint with OAuth authentication. The API would then carry out:

- Create a new version
- Save the zip for the new version
- Sign the zip
- Make the new version active

Things that should not happen:

- Validation (though it could be added to fire off warnings)
- App Review


The API should only allow clients that have been explicitly whitelisted in an API key/app ID mapping to perform this action. Since it's for internal use, it might make sense to put this configuration in settings rather than in the DB.

For instance, cvan's OAuth token might be mapped to the production marketplace app.

25 bonus points if we create a management command that allows this to be done from the terminal, as well.

Comment 1

[Andrew Williamson [:eviljeff]]

(imo) we should instead start implementation on the reviewer API then just use the existing submission API to submit, followed by the reviewer API to approve.

Comment 2

[Matt Basta [:basta]]

This is much higher priority than P4 since it blocks is from becoming a packaged app. At least p2. 

(In reply to Andrew Williamson [:eviljeff] from comment #1)
> (imo) we should instead start implementation on the reviewer API then just
> use the existing submission API to submit, followed by the reviewer API to
> approve.

That's not the point. We would use this to deploy marketplace updates and automate deployments of dev as a packaged app. This is not intended to be used for apps that need a review.

Comment 3

[Andrew Williamson [:eviljeff]]

(In reply to Matt Basta [:basta] from comment #2)
> (In reply to Andrew Williamson [:eviljeff] from comment #1)
> > (imo) we should instead start implementation on the reviewer API then just
> > use the existing submission API to submit, followed by the reviewer API to
> > approve.
> 
> That's not the point. We would use this to deploy marketplace updates and
> automate deployments of dev as a packaged app. This is not intended to be
> used for apps that need a review.

okay, but what's the argument against implementing something we're going to do anyway at some point?  As opposed to writing a whole new API that will only be used by one app.

Comment 4

[Matt Basta [:basta]]

(In reply to Andrew Williamson [:eviljeff] from comment #3)
> okay, but what's the argument against implementing something we're going to
> do anyway at some point?  As opposed to writing a whole new API that will
> only be used by one app.

Our current implementation requires upwards of six API calls, including polling to determine whether validation has completed. It's also unclear in the docs whether it's possible to submit a new version of an existing app through the API (I'm not sure that it's possible currently).

Additionally, we can't update the marketplace through the existing API since the validator blocks apps that use the origin app://marketplace.firefox.com (our own protections working against us).

On top of that, there's currently no API (or at least no documented API) for marking an app as reviewed.

None of this touches on delays and checks that would have to be added to an implementation to make sure that caching issues or pinning issues don't muck up the submission process.

Comment 5

[Christopher Van Wiemeersch [:cvan]]

(In reply to Matt Basta [:basta] from comment #4)

> Additionally, we can't update the marketplace through the existing API since
> the validator blocks apps that use the origin app://marketplace.firefox.com
> (our own protections working against us).

This is the biggest blocker for me. I was considering working on bug 877454 to work around this.

But I very much like this solution for getting an automatic deploy process going for -dev (and daily for stage and weekly for prod) for packaged apps.

The API's submission flow is too complicated and needs to be fixed. I agree we should work on it soonish though. But until we have an automatic deploy process we cannot become a true packaged app, which blocks us from many more things.

Comment 6

[Matt Basta [:basta]]

(In reply to Christopher Van Wiemeersch [:cvan] from comment #5)
> The API's submission flow is too complicated and needs to be fixed. I agree
> we should work on it soonish though. But until we have an automatic deploy
> process we cannot become a true packaged app, which blocks us from many more
> things.

To be fair, the current submission APIs map essentially 1:1 with the human-friendly submission flow. If you were going to build a front-end to the Marketplace using the API, it wouldn't be the worst API to have. Unfortunately, though, it doesn't really do a great job of supporting automated submissions where everything is ready before the first request is made.

Comment 7

[Andy McKay]

> It's also unclear in the docs whether it's possible to submit a new version of an existing app
> through the API (I'm not sure that it's possible currently).

Last time I checked, this is possible.

Comment 8

[Chuck Harmston [:chuck]]

My only argument against a DELETE would be that we're deleting a property of a resource, and not a resource itself.

That said, I'm not especially pedantic about such things. Carry on!

Comment 9

[Chuck Harmston [:chuck]]

Wrong tab, wrong bug. Ignore that!

Comment 10

[Christopher Van Wiemeersch [:cvan]]

Sounds like people don't want us to become a packaged app anymore. No priority for this then. Manual package updates should suffice.

Comment 11

[Christopher Van Wiemeersch [:cvan]]

We should do this.

Comment 12

[Christopher Van Wiemeersch [:cvan]]

Wiki outlining the manual packaging, submitting, an approving+signing instructions:
https://github.com/mozilla/fireplace/wiki/Creating-a-new-Packaged-App-version

Commands for `make package`, `make submit_package`, `make approve_package`:
https://github.com/mozilla/yogafire/blob/master/Makefile#L24-L95

Comment 13

[Christopher Van Wiemeersch [:cvan]]

Mat and I talked on IRC, but this is what I started with: https://github.com/cvan/zamboni/commit/2e7cf0de3

(It doesn't actually work, but it's an outline of how it could possibly work.)

In Yogafire's fabfile.py for -dev, we should automatically create a new package + create a new version for 'marketplace-package' + sign it.
For stage and production, we should have a management command that we call from the command line to create the package, create the version, and sign the package.

Comment 14

[Mathieu Pillard [:mat]]

PR for the management command approach https://github.com/mozilla/zamboni/pull/1977

Comment 15

[Mathieu Pillard [:mat]]

Management command fixed in https://github.com/mozilla/zamboni/commit/4c7527943b14a0438bf453a4425493eb9f32fd7e

Syntax: manage.py upload_new_marketplace_package <path-to-zip>

I'll look into having this called automatically every time we deploy yogafire.

Comment 16

[Mathieu Pillard [:mat]]

The management command works on -dev, I just pushed a new package using it (need to be ran with apache user, if you are curious), but I'm running into transaction issues - I'm hoping the upgrade to django 1.6.4 will fix those.

Comment 17

[Mathieu Pillard [:mat]]

This is done. With this bug and jason's work in bug 1005967, yogafire now builds, uploads and sign a new dev package for every commit in yogafire.

Details of a deploy (click "deploy"): https://dreadnot-dev.addonsadm.private.phx1.mozilla.com/stacks/marketplace-dev.allizom.org-yogafire/regions/phx1/deployments/5