1023084 - Offer a command line tool that communicates with the device and webapps actor

Status: RESOLVED FIXED

(DevTools Graveyard :: WebIDE, defect)

Description

[Paul Rouget [:paul]]

We want developers to be able to push apps to the device without using the AppManager.

The same way the App Manager offers extensions like Simulators or ADBAddonHelper (see bug 1012760), it should offer a "Command line tools" addon, which would install a script (JS or Python?) which would communicate with the webapps actor to offer the same features as the appmanager from the command line.

Here are some projects that we could leverage:
- ffdb.py: https://github.com/kripken/emscripten/blob/85c15a40f96d3e14416bcbef1fd76537080798b5/tools/ffdb.py#L160
- https://github.com/ochameau/firefox-client
- marionette
- https://github.com/campd/python-fxdevtools

Comment 1

[Paul Rouget [:paul]]

My personal preference goes to ffdb.py as it's maintained, used by another Mozilla team, in python (python is usually more spread than node), very specialized (designed to talk to the device/simulator), one single file, and it already offers a nice set of command line options.

Comment 2

[Soledad Penades [:sole] [:spenades]]

I must have misunderstood something in the way the thing you built worked (or didn't work, now I'm not sure of anything :-P), but couldn't you just build a CLI interface to a running appmanager/firefox process that then communicates with the device, whatever that device is?

Also, I vote against Python. So many versions and so much confusion between 2.x and 3 and it's hard to get the version you want running in your machine.

Comment 3

[Paul Rouget [:paul]]

(In reply to Soledad Penades [:sole] [:spenades] from comment #2)
> I must have misunderstood something in the way the thing you built worked
> (or didn't work, now I'm not sure of anything :-P), but couldn't you just
> build a CLI interface to a running appmanager/firefox process that then
> communicates with the device, whatever that device is?

That's how it works. But the CLI interface doesn't "interface" on Mac.

> Also, I vote against Python. So many versions and so much confusion between
> 2.x and 3 and it's hard to get the version you want running in your machine.

afaik, Python 2 is everywhere. Only notable exception being a version of Linux called "archlinux". Node is usually never installed. xpcshell either.

Comment 4

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Paul, you had mentioned in our meeting that you had more specific background information on what were trying to accomplish (use cases, why we want this, etc.).  Could you please add that here to better inform our discussion?  Also, I think you mentioned there was another similar bug with more details, but which bug were you thinking of?

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Paul Rouget [:paul] (slow to respond. Ping me on IRC) from comment #3)
> > Also, I vote against Python. So many versions and so much confusion between
> > 2.x and 3 and it's hard to get the version you want running in your machine.
> 
> afaik, Python 2 is everywhere. Only notable exception being a version of
> Linux called "archlinux".

Also, that thing called Windows does not have Python. ;)

Comment 6

[Jukka Jylänki]

The latest version of ffdb.py lives at url https://github.com/kripken/emscripten/blob/incoming/tools/ffdb.py

The requirements/use cases that we have for FFOS:
 - end-users, think (native C/C++) indie game developers and hobbyists, need to be able to drive the phone and automate from cli when they develop Emscripten games and target FFOS. These people are accustomed to Android, Winphone and iOS development model and toolchains, and expect the same ease from FFOS.
 - game engine and middleware developers have the need to be able to automate from cli for both development and unit testing purposes for their own engines.
 - game engine and middleware developers need to be able to provide automation capabilities to users using their products.
 - the automation tool must be extremely simple and embeddable, that can be integrated as a hidden and automatic part of such middleware/sdk engines, as if it "didn't exist". The result must look seamless to the developer. This is why ffdb.py is a single file and will stay that way for the foreseeable future.
 - we cannot require flashing a dev build of a phone, so requiring setting up marionette or building anything from source (B2G, or other codebases) is out of the question. Marionette is great for internal Mozilla unit test suites, but it's too heavyweight to impose on other developers who need to automate and unit test as well.

While ffdb.py lives in the Emscripten repo, it is not tied to Emscripten development, but it can operate on arbitrary packaged apps. A native ffdb.exe version would be nice, but not important at all in the Emscripten comunity, since requiring python for us is ok, because the Emscripten compiler already requires it, and it ships as part of Emscripten SDK ( http://emscripten.org/ ). ffdb.py also ships as part of the SDK, so people get python, Emscripten compiler and ffdb.py automatically in a single bundle, like they expect.

Comment 7

[Paul Rouget [:paul]]

Jukka, thanks a lot for all these details.

Ryan, right, no python on Windows. I was doing the distinction between python2 and python3. I believe requiring python is not a big deal. But it might be too early to discuss that.

And I was also thinking about bug 1000993, which is not directly related to this bug, but it's part of the same story (supporting phonegap).

Comment 8

[Paul Rouget [:paul]]

Another command line is being developed for validation: https://groups.google.com/d/msg/mozilla.dev.marketplace/SDG2S-2u_Rw/b3qzmxhZqIcJ

Comment 9

[Fred Wenzel [:wenzel]]

What's the use case for an additional command line tool that is separate from the WebIDE?

As you know, my team is working on grunt/gulp command line tasks for developers (consuming, not writing them, mostly), but most of them should IMHO eventually be exposed by the app manager, through its CLI interface. For instance, no need to have yet another a separate tool to do app submission if I can throw it at the app manager programmatically and say "submit this for me, will you?"

The embeddable case is valid, and I can see a separate toolchain for this that does not require a running version of Firefox on a server somewhere. At any rate, a JS module is probably preferable because it can be wrapped many ways (server-side, client, standalone). (And I'm saying this as a Pythonista myself).

Ryan, I am happy to chat, but let's not all go off into different directions without talking to each other.

Comment 10

[Rory Petty]

+1 Fred's comment

IMO, the most valuable use case is having AppManager/WebIDE having two way shell communication with other CLI tools. The Node tooling community has really taken off (grunt, gulp, yeoman, etc), so it would be nice to leverage all of this work in WebIDE.

Comment 11

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I intend to schedule a meeting soon with the various teams at work here to see where we can unify the approach if it makes sense.  I'll reach out to you when that happens.  Stay tuned.

Comment 12

[Paul Rouget [:paul]]

(In reply to Fred Wenzel [:wenzel] from comment #9)
> As you know, my team is working on grunt/gulp command line tasks for
> developers (consuming, not writing them, mostly)

This is getting out of hand. Really. Every week I hear about people working on command line tools.

We have 4, 5 maybe 6 projects doing similar things. I don't believe in different toolchains in different languages using different tools, all addressing not-so-different requirements. We need only *one* toolchain to address all these needs. Using the same code base, same language, and same tools.

Everybody, please hold on a little bit. Ryan will try to get a good understanding of the situation (existing projects, requirements, possible solutions, ...), and make sure product hear about it and figure out a plan.

I know we all have different priorities & agenda, but let's not hurry too much. Let's talk.

Comment 13

[tofumatt [:tofumatt]]

I don't think the agenda is that different; make web development easier. What 4-6 other tools specifically are you referring to?

Comment 14

[Paul Rouget [:paul]]

- ffdb.py, emscripten team
- firefox-client, devtools team (Alex)
- python-fxdevtools, by dcamp
- marionette, B2G QA
- firefox-webapp, market-place
- webide cli, devtools app team
- gaia build system also has way to push apps

Comment 15

[Fred Wenzel [:wenzel]]

(In reply to Paul Rouget [:paul] (slow to respond. Ping me on IRC) from comment #12)
> I know we all have different priorities & agenda, but let's not hurry too
> much. Let's talk.

Yup, definitely. Thanks Ryan for setting up a meeting. That's why I stepped in, so our and your product people can make sure users' (developers'!) needs are met and we don't just keep piling on. No problem with Ryan making code happen, so long as we cover the relevant requirements.

Our agendas probably largely coincide. Some of the tools you mention are vaguely similar, but if they are 100% internal tools we don't have to care about them very much at this point (but they might be replaced by what this bug is about, if it's useful for them). We can/should inform those groups however so they can provide input. As for external developers, it's important we don't ship various solutions to the same problem with no way to determine which they should use.

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

A number of people met to discuss tools for apps last week.  I've summarized[1] what was discussed.

This bug's original aim was to purpose a single "master" CLI tool for all things apps.  For now, I am not convinced we want to make such a tool, so I will close this bug.  We can always reopen / refile if the situation changes.

There are other bugs related to items that arose from the discussion:

* Hook in WebIDE before pushing apps to do addition build steps, etc. (bug 1000993)
* CLI tools should have a way to push apps (bug 1035185)
* CLI tools should have a way to trigger debugging (bug 1035192)

[1]: https://wiki.mozilla.org/DevTools/WebIDE/External_Tools

Comment 17

[Paul Rouget [:paul]]

I don't think we need a "master CLI" _today_. But I'd like to keep this bug open as a way to track more cli projects (if any), and possibly define the scope of such a master CLI.

Comment 18

[nicola greco]

I would rather have some good spec on this and a great documentation on RDP functionalities with FirefoxOS so that people can just use libraries such as firefox-client or volcan to build their own deployment scripts and tools (think of Gulp or Grunt).

Comment 19

[Jukka Jylänki]

Based on user request, I migrated ffdb.py from Emscripten to live in its own github repository juj/emrun, see here https://github.com/juj/emrun . Emscripten repository will continue to have a copy of the latest ffdb.py, but the development of it now occurs in juj/emrun repository.

Comment 20

[Paul Rouget [:paul]]

Let's add to this list Nicola's work: https://github.com/nicola/ - see all the moz-*-b2g repositories.

Comment 21

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Most recent activity in this area is happening in node-firefox[1].

[1]: https://github.com/mozilla/node-firefox

Comment 22

[Alexandre Poirot [:ochameau]]

node-firefox can be considered as a cli tool doing all that.
It even has some alternatives like ffdb.py or firefox-client.

I don't think there is much more to be done here, especially given the announcements around firefox OS.