543463 - Make it easier for non-Mozilla devs to get and use the JavaScript Shell

Status: RESOLVED FIXED

(Core :: JavaScript Engine, enhancement)

Description

[David Humphrey (:humph)]

I love the command line js shell.  Lately I've been working with a lot of web devs, and converting them over to using it.  We've built a whole bunch of automated test infrastructure using it, for example.  But, it's currently too hard for people who don't regularly build mozilla-central to obtain.  In much the same way that the python shell makes interactive development easier and more enjoyable, I would love to see a way to point more people at the js shell.

Some people will say that I should have them use rhino.  That is one option, but its java dependencies make it almost as hard to get running.  I also know about the various in-browser versions of this.  I'm specifically talking about making it easier for more people to do what I do every day, and type ./js

If there are issues standing in the way of making this happen, I'd be glad to look for some students to work on fixing them on the way to making this happen.

Comment 1

[David Mandelin [:dmandelin]]

>In much the same way that the python shell makes interactive development
>easier and more enjoyable, I would love to see a way to point more people
>at the js shell.

Sounds great to me. What specifically do you think we should do? Building a JS shell from source is pretty easy, although I don't know if we have easy directions posted in an easy-to-design place. Or do you think we should have a binary distribution?

Comment 2

[David Humphrey (:humph)]

Building this isn't hard if you're setup to do it, no.  But setting up a build environment (esp on Windows) for this is going to cause many pure web-devs to forget it.

So I think that having binaries is important.  I'm not sure that it needs to keep pace with trunk or anything, since there won't be an update mechanism.  But a binary of the last major release would be awesome.

For Linux distros (or even on Mac with Mac Ports), we could consider getting something in there to build it from source.

Comment 3

[Mike Shaver (:shaver emeritus)]

I thought we packaged a shell in our nightlies.  Maybe that's xpcshell, but ./js is tiny enough that we could add it without much pain, I suspect.

Comment 4

[(not currently active) Ted Mielczarek]

We stick xpcshell in the test package, but not js. Of course, that makes for a pretty big download if all you want is the shell. We could pretty easily just zip up the js shell by itself and upload it alongside everything else.

Comment 5

[Mike Shaver (:shaver emeritus)]

My expectation is that many such developers would already have a Firefox install, and therefore could just look in it for the shell.  No?

Comment 6

[(not currently active) Ted Mielczarek]

If we shipped the js shell in the Firefox package, that would work, yes. We don't currently do so. (We don't ship xpcshell in there either, we upload it separately in the test package.)

Comment 7

[David Humphrey (:humph)]

What needs to happen next on this?  Is there something I can help do?

Comment 8

[David Mandelin [:dmandelin]]

(In reply to comment #7)
> What needs to happen next on this?  Is there something I can help do?

How about a list of what things need to be included in a binary distro? Such as:

- a 'js' binary built with what flags? threadsafe, tracevis, etc?
- multiple 'js' binaries built with different flags?
- js static/dynamic libraries?
- documentation?
- sample JS programs?
- perf tools?
- debugging tools?
- documentation?
- anything else?

Comment 9

[David Humphrey (:humph)]

Likely there are various use cases, and your list seems to cover many, if not all of them.  So as to not turn this into a project scoped too large to get done, I'd suggest a first effort might include these basics from your list:

- a single 'js' binary ("default" flags)
- simple documentation on the shell itself, perhaps just a README with links to articles on MDC
- some simple example js programs, perhaps to demonstrate features of our js engine

The other items you list all sound great, but will they turn this into a bigger/later project?  If I look at Rhino as a comparison, I see:

- js binary
- src, build tools, tests
- examples

I don't think we need to ship src or tests with this, since that is readily available now, and we're targeting people who can't easily build the src (e.g., web developers).

Comment 10

[David Mandelin [:dmandelin]]

I agree, keep it simple is the way to go. Would you mind filing bugs on the README and the example programs? 

Once we have those in place do we just want to do the builds manually on a few key platforms, or should we try to get that automated? Or farm it out to whoever has platform X?

The last question is, where do we host the downloads? Who can help us with that?

Comment 11

[David Humphrey (:humph)]

> I agree, keep it simple is the way to go. Would you mind filing bugs on the
> README and the example programs? 

Done.

> Once we have those in place do we just want to do the builds manually on a few
> key platforms, or should we try to get that automated? Or farm it out to
> whoever has platform X?

It's not up to me, but if this isn't automated, it won't happen again.  Ideally we would make this available on the same platforms as Firefox.  Can this get tied into our build/release process?

Comment 12

[(not currently active) Ted Mielczarek]

Our 1.9.2 and trunk builds already build the js shell
by default. It'd be simple to tar or zip up the binary and upload it alongside the firefox package.

Comment 13

[David Humphrey (:humph)]

We were discussing this on irc the other day, and I just wanted to add a note here that getting the "fast new js in Firefox 4" out there as a shell devs can use, too, would be very awesome.

Comment 14

[David Humphrey (:humph)]

Watching projects like the spidermonkey node stuff happening, I'm even more interested in seeing this happen.  Given that I'm the one asking for this, I'll volunteer to do it if I can get Ted/Dave to give me some guidance (I'll find you on mail/irc).

Comment 15

[David Humphrey (:humph)]

14:06 < ted> so, what i'd do is hook into "make package" somewhere and package up a separate js.zip or js.gz or whatever
14:06 < ted> and then just add it to the UPLOAD_FILES list, and it will get uploaded alongside our other bits

Comment 16

[David Humphrey (:humph)]

OK, I've got the plumbing in place to do this now.  Can a JS person help me understand which bits, besides dist/bin/js, need to get put into the packaged zip?  e.g., which libraries need to go with it, so that the js binary will run?

Comment 17

[David Humphrey (:humph)]

Ted helped me on irc, adding these now:

@executable_path/libplds4.dylib (compatibility version 1.0.0, current version 1.0.0)
@executable_path/libplc4.dylib (compatibility version 1.0.0, current version 1.0.0)
@executable_path/libnspr4.dylib (compatibility version 1.0.0, current version 1.0.0)

Comment 18

[David Humphrey (:humph)]

Attachment: Package jsshell with make package

Comment 19

[(not currently active) Ted Mielczarek]

Comment on attachment 537569 [details] [diff] [review]
Package jsshell with make package

Review of attachment 537569 [details] [diff] [review]:
-----------------------------------------------------------------

Once you fix up a few things, you should push this to try. If it works, I think you ought to get your JS bin package in your try build directory.

::: toolkit/mozapps/installer/packager.mk
@@ +80,5 @@
>  SDK_PATH = sdk/
>  endif
>  SDK_SUFFIX    = $(PKG_SUFFIX)
>  SDK           = $(SDK_PATH)$(PKG_BASENAME).sdk$(SDK_SUFFIX)
> +JSSHELL_BIN   = $(DIST)/bin/js$(BIN_SUFFIX) \

maybe BINS (plural?)

@@ +81,5 @@
>  endif
>  SDK_SUFFIX    = $(PKG_SUFFIX)
>  SDK           = $(SDK_PATH)$(PKG_BASENAME).sdk$(SDK_SUFFIX)
> +JSSHELL_BIN   = $(DIST)/bin/js$(BIN_SUFFIX) \
> +                $(DIST)/bin/libplds4$(DLL_SUFFIX) \

s/lib/$(LIB_PREFIX)/ (so this works on Windows)

@@ +84,5 @@
> +JSSHELL_BIN   = $(DIST)/bin/js$(BIN_SUFFIX) \
> +                $(DIST)/bin/libplds4$(DLL_SUFFIX) \
> +                $(DIST)/bin/libplc4$(DLL_SUFFIX) \
> +                $(DIST)/bin/libnspr4$(DLL_SUFFIX) \
> +                $(NULL)

Our (undocumented) Makfile style is two-space indent for continuations, like:
JSSHELL_BIN = \
  $(DIST)/bin/js$(BIN_SUFFIX) \
  ...
  $(NULL)

@@ +100,5 @@
>  PKG_SUFFIX	= .tar
>  INNER_MAKE_PACKAGE 	= $(CREATE_FINAL_TAR) - $(MOZ_PKG_DIR) > $(PACKAGE)
>  INNER_UNMAKE_PACKAGE	= $(UNPACK_TAR) < $(UNPACKAGE)
>  MAKE_SDK = $(CREATE_FINAL_TAR) - $(MOZ_APP_NAME)-sdk > $(SDK)
> +MAKE_JSSHELL = $(CREATE_FINAL_TAR) - $(JSSHELL_BIN) -h > $(JSSHELL_PKG)

If you don't want to go through all this trouble, you can just make it a zip file on all platforms. Shouldn't matter, since it's a new thing.

@@ +679,5 @@
>  	$(SYSINSTALL) $(IFLAGS1) $(MOZ_PKG_REMOVALS_GEN) $(DIST)/$(STAGEPATH)$(MOZ_PKG_DIR)$(_BINPATH)
>  endif # MOZ_PKG_REMOVALS
> +# Package JavaScript Shell
> +	@echo "Packaging JavaScript Shell..."
> +	$(MAKE_JSSHELL)

I think we build the JS shell in all build configurations now, but I'm not 100% sure. This is probably okay.

Comment 20

[David Humphrey (:humph)]

Attachment: Patch v2

Fixed nits, switched to zip for all platforms, fixed per-platform library dependencies.  Try server builds pass, and resulting shells work:

https://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-476a0609ae75/

Comment 21

[(not currently active) Ted Mielczarek]

Comment on attachment 538344 [details] [diff] [review]
Patch v2

Review of attachment 538344 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good!

::: toolkit/mozapps/installer/packager.mk
@@ +95,5 @@
> +  $(DIST)/bin/$(LIB_PREFIX)plc4$(DLL_SUFFIX) \
> +  $(NULL)
> +endif
> +JSSHELL_PKG   = $(DIST)/jsshell.zip
> +MAKE_JSSHELL  = $(ZIP) -9j $(JSSHELL_PKG) $(JSSHELL_BINS)

Does this wind up with relative paths in your zip? (like bin/js.exe) Is that what you want?

Comment 22

[David Humphrey (:humph)]

Thanks for the reviews.  I don't know, I'm torn.  I think a zip can dump 3 or 4 files into the current dir.  Let's take it as is, I think.

Comment 23

[David Humphrey (:humph)]

Removing these other bugs, as I don't think they really need to block.  I'll fix them separately.

Comment 24

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/projects/build-system/rev/d12dda590f4a

Comment 25

[David Humphrey (:humph)]

I'm just looking at https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/ and wondering if I need to put an OS name on jsshell.zip (jsshell-$(some os var).zip) so that they don't overwrite each other in that combined dir.  The try builds are split into per-OS dirs.

Ted?

Comment 26

[(not currently active) Ted Mielczarek]

Yeah, good call. You can stick a $(MOZ_PKG_PLATFORM) in there. If you'd like to avoid duplicating that junk, you can define a variable for the zip filename in package-name.mk where we define all the other stuff.

Comment 27

[John Ford [:jhford] CET/CEST Berlin Time]

Can you make it end with 'jsshell.zip' instead of putting the platform between 'jsshell' and the file extension?  If you prefer that naming scheme, we need to change the relevant part of automation.

http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l189

Comment 28

[David Humphrey (:humph)]

Attachment: Add platform name to jsshell package

This is on top of what you landed yesterday, and changes the name of the jsshell package to include the platform.

Comment 29

[David Humphrey (:humph)]

Attachment: Buildbot changes for platform name addition

John, I don't think putting the platform name first makes sense.  Here's a fix for the platform name as done in my latest patch.

Comment 30

[(not currently active) Ted Mielczarek]

Comment on attachment 538696 [details] [diff] [review]
Add platform name to jsshell package

You need to change the mention in UPLOAD_FILES as well.

John: why does the automation need to know about this file anyway?

Comment 31

[David Humphrey (:humph)]

Attachment: Add platform name to jsshell package + upload fix

Thanks Ted.  Fixed that other use.

Comment 32

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/projects/build-system/rev/055e5a11a673

Comment 33

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/mozilla-central/rev/d12dda590f4a
http://hg.mozilla.org/mozilla-central/rev/055e5a11a673

Comment 34

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 538697 [details] [diff] [review]
Buildbot changes for platform name addition

http://hg.mozilla.org/build/buildbotcustom/rev/8b5fc9186d03 landed on default and will roll into production later today/very early tomorrow as releases permit (currently running the 3.6.18 release).

Comment 35

[(not currently active) Ted Mielczarek]

Backed out of m-c because of bug 663796. The buildbot patch here should fix that, but I can't leave m-c broken until that gets reconfigured. We'll validate that it's fixed on build-system and these can reland on m-c.

http://hg.mozilla.org/mozilla-central/rev/479e2681c25f
http://hg.mozilla.org/mozilla-central/rev/4540ce8760b6
http://hg.mozilla.org/mozilla-central/rev/d36cf8e6da68
http://hg.mozilla.org/mozilla-central/rev/6e42ce13b93d

Comment 36

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #30)
> John: why does the automation need to know about this file anyway?

it only needs to know so that it doesn't assign that url to the packageUrl.  I am writing up a bug to fix parse_make_upload to hopefully avoid this in future.

Comment 37

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #36)
> (In reply to comment #30)
> > John: why does the automation need to know about this file anyway?
> 
> it only needs to know so that it doesn't assign that url to the packageUrl. 
> I am writing up a bug to fix parse_make_upload to hopefully avoid this in
> future.

filed 663825

Comment 38

[Mike Hommey [:glandium]]

The big problem with the js shell is that it's not suitable for widespread use, because the debugging functions it provides are dangerous. It's been a mistake from linux distros (and here I'm talking with my Debian hat on) to ship the js shell as it is. If we want to ship a js shell, we should ship one that is minimalistic, not the one that currently exists.

Comment 39

[(not currently active) Ted Mielczarek]

http://hg.mozilla.org/projects/build-system/rev/76b7d4ad1659
http://hg.mozilla.org/projects/build-system/rev/925918881390

Comment 40

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/mozilla-central/rev/76b7d4ad1659
http://hg.mozilla.org/mozilla-central/rev/925918881390

Comment 41

[(not currently active) Ted Mielczarek]

(In reply to comment #38)
> The big problem with the js shell is that it's not suitable for widespread
> use, because the debugging functions it provides are dangerous.

I don't really now much about the details here, but I trust your experience. That being said, we're not shipping this with much visibility at the moment, so the right course of action is probably to file a followup on creating a new shell that's more suitable, perhaps related to the much-needed refactoring that bug (that I can't find right now) about how js.cpp and xpcshell have so much copy-pasted code.

Comment 42

[Nicholas Nethercote [inactive]]

(In reply to comment #38)
> The big problem with the js shell is that it's not suitable for widespread
> use, because the debugging functions it provides are dangerous. It's been a
> mistake from linux distros (and here I'm talking with my Debian hat on) to
> ship the js shell as it is. If we want to ship a js shell, we should ship
> one that is minimalistic, not the one that currently exists.

I'd love to know more details about this, including what should be removed from the shell to make it "safe".

Comment 43

[Mike Hommey [:glandium]]

(In reply to comment #42)
> (In reply to comment #38)
> > The big problem with the js shell is that it's not suitable for widespread
> > use, because the debugging functions it provides are dangerous. It's been a
> > mistake from linux distros (and here I'm talking with my Debian hat on) to
> > ship the js shell as it is. If we want to ship a js shell, we should ship
> > one that is minimalistic, not the one that currently exists.
> 
> I'd love to know more details about this, including what should be removed
> from the shell to make it "safe".

I don't remember the details. I might have that somewhere on my archives, but I remember this was brought a long time ago on #jsapi. Anyways, the best IMO would be to start from scratch and add what would be required for the shell to be useful, not as a js engine debugging tool, but as a generic javascript script excution environment. Apart from a few functions for console and files, I don't think there's much that is required... But as Ted notes, we should file a new bug for that.