962879 - Support CLI for offline APK generation

Status: RESOLVED FIXED

(Marketplace Graveyard :: Integration, defect, P2)

Description

[Austin King [:ozten]]

Exact requirements TBD but we should have a CLI for the APK Factory Service that:
* Is easy to install on most platforms
* Converts an OWA to an APK
* Signs APK with a debug cert (Reviewer not Release)
* Can work with offline or local network apps

Comment 1

[Austin King [:ozten]]

CLI Design proposal

(Assume production will live at https://apk-review.mozilla.org and https://apk-review.mozilla.org, which is TBD)

* Developer does `npm install -g mozilla-apk-cli` to install the toolchain
* Developer does `mozilla-apk-cli http://localhost:8000/manifest.webapp myapp.apk`
* mozilla-apk-cli options
  * --endpoint - Base url of a APK Factory Server (useful for dev/qa, not for end users)
  * --manifestoverride - url to treat as the OWA manifest url during build
* By default, mozilla-apk-cli will hit https://apk-review.mozilla.org/cli_build
* mozilla-apk-cli program will do the following:
  * Download the manifest url
  * Optionally download any packaged app
  * POST manifest url, manifest, and packaged app to the Review instance of the APK Factory Service
  * Write response to an apk file or output useful error messages
* Apk is not cached server side
* Apk will be signed with a debug key

Options dropped from the prototype cli
* buildDir - build happens server side
* cacheDir - no caching of developer builds is allowed
* force - system will be force=true at all times
* output - output apk filename is now required

Benefits:
Solution has minimal dependencies - NodeJS and doesn't require developers to install Java / Android SDK.

Solution maintains existing attack surface area for security.

Cons:
Developer must be online as generation happens server side.

Questions:
I'm tempted to drop manifest override, the only benefit I can imagine is letting a developer have some influence over the java packagename.

jhugman, myk - Thoughts?

Comment 2

[Austin King [:ozten]]

> (Assume production will live at https://apk-review.mozilla.org and https://apk-review.mozilla.org, which is TBD)

https://apk-review.mozilla.org and https://apk-release.mozilla.org rather.

Comment 3

[Austin King [:ozten]]

Other properties to QA
* CLI builds shouldn't end up in S3
* CLI builds shouldn't end up in MySQL APK Metadata table

Comment 4

[Austin King [:ozten]]

jhugman or myk: If the manifest file is a local file and it is a packaged app, should the package_path be resolved against the locale file or the manifest override?

My hunch is local.

Comment 5

[Austin King [:ozten]]

First stab at cli.js. Basic use cases work...

What's broken:

    Packaged apps using a local manifest file
    npm install of the cli app


Code https://github.com/mozilla/apk-factory-service/pull/44

How to try it out...

git pull origin master
ode bin/cli.js --endpoint=http://dapk.net http://deltron3030.testmanifest.com/manifest.webapp deltron3030.apk

Note: to hit your local dev environment, use --endpoint=http://localhost:8080

Comment 6

[Myk Melez [:myk] [@mykmelez]]

(In reply to Austin King [:ozten] from comment #1)

>   * --manifestoverride - url to treat as the OWA manifest url during build

Nit: If we keep this flag, then I would separate the two words here with a dash:

  --manifest-override


> * By default, mozilla-apk-cli will hit
> https://apk-review.mozilla.org/cli_build
> * mozilla-apk-cli program will do the following:
>   * Download the manifest url

Note that the prototype allowed the manifest "URL" to be either a URL or a path to a file on the local filesystem.  I don't think we need to support this option for hosted apps, but I do think we need to support it for packaged apps, see below.


>   * Optionally download any packaged app

Packaged apps are popular in part because they don't require you to stand up a web server nor host any files, so the common case for packaged apps will be that all files reside on the local machine, and there is no web server to obtain them from.

Nor, most likely, will the developer have created a mini-manifest, since Marketplace auto-generates those from the information in the manifest, and they're only useful when hosting the package.

And the developer will be working from source files, not packaged files; so it'll be quite helpful to handle the packaging of the files on their behalf, especially across multiple change/build/test cycles.  (This is what the App Manager does for packaged apps being tested on FxOS devices or the Simulator.)

Thus mozilla-apk-cli should support generating APKs for packaged apps whose contents are entirely on the local machine, i.e.:

  mozilla-apk-cli /path/to/appdir myapp.apk

Where mozilla-apk-cli does the following:

  * (ideally) archive the contents of appdir into a ZIP archive ("the package");
    (saves developer from having to do this themselves each time they make a change)

  * post the package and its manifest to the Review instance of the APK Factory Service;
    (or just post the package and let the service extract the manifest from it)

  * write response to APK file or output useful error messages.


> Options dropped from the prototype cli
> * buildDir - build happens server side
> * cacheDir - no caching of developer builds is allowed
> * force - system will be force=true at all times
> * output - output apk filename is now required

Yup, makes sense.  wget and curl behave differently by default, with wget deciding a filename and curl outputting to stdout.  But this requirement to specify a filename is a better default, because simple and explicit.  We can always add wget- and/or curl-like behavior later to simplify further for frequent and/or particular fliers.


> Benefits:
> Solution has minimal dependencies - NodeJS and doesn't require developers to
> install Java / Android SDK.

Yup, I like it for these reasons too.


> Solution maintains existing attack surface area for security.

A bonus!


> Cons:
> Developer must be online as generation happens server side.

This is ok, even for packaged apps whose contents are entirely state-side.  There are some use cases for fully-offline generation, like hacking on a plane.  But this requirement is entirely reasonable for the 80% case, and we should tackle offline later.


> Questions:
> I'm tempted to drop manifest override, the only benefit I can imagine is
> letting a developer have some influence over the java packagename.
> 
> jhugman, myk - Thoughts?

The manifest override flag in the prototype implementation was useful to me on a couple occasions when I needed to create an APK for someone else's app and modify (or create) the manifest myself.  I pointed the CLI at a local manifest and used the manifest override flag to identify the URL at which that manifest was expected to reside.

But I'm unsure how useful this would be to actual webapp developers.  I think the common cases will be:

1. The developer is working on a hosted app entirely on a web server accessible to their local machine, f.e. a local development server, i.e. localhost; a private development server on an internal network; or a public development server, like dapk.net.

2. The developer is working on a packaged app whose files are entirely on their local machine, and they have not even constructed a mini-manifest, much less hosted it on a development server.

In case 1, we don't need to support a manifest override, as the developer can simply point the CLI to the actual URL of the manifest.  In case 2, we also don't need to support a manifest override, as there is no mini-manifest URL to override.

Some cases for which a manifest override might come in handy is when a developer is "appifying" an existing website or working on a hosted app that loads some content from the production server on which the developer expects the app to be hosted eventually.  Then the developer might want the APK's manifest URL and origin to be different from the actual URL/origin of the manifest.

But these also aren't 80% cases, so I think we should drop the manifest override option and then revisit it later as appropriate.

Comment 7

[Austin King [:ozten]]

Ok, we will drop overrideManifest.

> And the developer will be working from source files, not packaged files; so it'll be quite helpful to handle the packaging of the files on their behalf, especially across multiple change/build/test cycles.  (This is what the App Manager does for packaged apps being tested on FxOS devices or the Simulator.)

News to me, we will build this. Any special requirements about the zip file and it's contents?

We'll derive the package name from the App Name.

Although the developer specifies a path to the packaged app's root source files, we will require a manifest.webapp file to be there.

It would be great to get some help on this "create the packaged app for the developer" part.

Comment 8

[James Hugman [:jhugman] [@jhugman]]

This all looks sensible.

My attitude to offline can be generalized to "relying on as few external machines as possible". If this can drop to zero, then this is ideal. There is room here for discussion of "Rambo architecture" http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html , but that's not what I mean in this case. 

Originally, the two manifest flags were about not standing up a development server – a manifest could be hosted on the filesystem or a server, then paths (icons, package_path) could resolve to some place on the fs. The apk would be built with an overide-manifest as the manifest URL, but actually getting its resources from the FS.

Before any of the client side was written, I thought it'd be nice to be able to package up a hosted app in an archive. Then, stuffing the contents of the package into appcache once it reached fennec.

For an MVP, none of this is desirable, so TL;DR: I agree with Myk.

Comment 9

[andre]

I just want to add a comment about the importance of offline generation. Firefox OS is being sold in markets where network connection is not as prevalent as the US and EU. 

Networks fail a lot in South America which is the main market for Firefox OS right now. Because of this, it is interesting to have a toolset that doesn't depend on external servers. 

Also if you're trying to use this inside enterprises, there is a huge chance that you won't be allowed to use external servers.

Comment 10

[Austin King [:ozten]]

Making good progress on CLI, working in this branch:
https://github.com/mozilla/apk-factory-service/tree/bug-962879-cli-packaged-apps

Comment 11

[Austin King [:ozten]]

Packaged apps are done and can use testing!

    npm install mozilla-apk-cli

Source: https://github.com/mozilla/apk-cli

Comment 12

[Iulian Timis]

Please add STR here or mark it with [qa-] if no QA is needed.

Comment 13

[Austin King [:ozten]]

There are three modes to test: hosted, packaged app in a zip, and packaged app from a directory

Resources:
* https://github.com/ozten/packaged-app - has zip file as well as source code in a directory
* http://testmanifest.com/ - hosted apps

Steps to Reproduce
1) npm install mozilla-apk-cli
2)
2a) ./node_modules/.bin/mozilla-apk-cli http://fox9974.testmanifest.com/manifest.webapp test_app.apk
2b) ./node_modules/.bin/mozilla-apk-cli ../packged_app/package.zip test_app.apk
2c) ./node_modules/.bin/mozilla-apk-cli ../packged_app/www test_app.apk
3) Side load this apk onto Android

This can be done by putting them on a web server or using adb

adb install test_app.apk

Expected:
App should be launchable.

Note: You must remove an app from Android before pushing a second time. These are 'debug' testing apps and not updatable.