1108771 - (bootstrapfennec) Implement |mach bootstrap| for mobile/android

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nick Alexander :nalexander [he/him]]

Right now, configuring a development environment for building Firefox for Android (application == mobile/android) is pretty damn hard: see https://wiki.mozilla.org/Mobile/Fennec/Android#Building_Fennec.  I'd like to make this easier.
 
I've looked into the relevant code at [1], and it looks like this is a little complicated because the bootstrapper assumes that we're building Firefox for Desktop and the relevant incantations don't allow parsing arguments.  I propose that we either provide an alternate script location, which handles the mobile/android application, or we query the user to ask what she wants to build.

Then we can implement the long tail of package managing for all the various OS configurations we can stomach.

[1] http://dxr.mozilla.org/mozilla-central/source/python/mozboot/

Comment 1

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander from comment #0)
> Right now, configuring a development environment for building Firefox for
> Android (application == mobile/android) is pretty damn hard: see
> https://wiki.mozilla.org/Mobile/Fennec/Android#Building_Fennec.  I'd like to
> make this easier.
>  
> I've looked into the relevant code at [1], and it looks like this is a
> little complicated because the bootstrapper assumes that we're building
> Firefox for Desktop and the relevant incantations don't allow parsing
> arguments.  I propose that we either provide an alternate script location,
> which handles the mobile/android application, or we query the user to ask
> what she wants to build.

gps, can you give an opinion on how this should be handled?

Comment 2

[Nick Alexander :nalexander [he/him]]

ckitching: I intend to push on this for the platform I use, Mac OS X.  Can you rank the list of UNIX-like systems in [1] in some order of "most valuable" to "least valuable" to support?  Doesn't need to be perfect, but considering criteria like number of total users, proportion of users who are likely to develop, technical ability of those users, ease of implementation, etc.  I'd like to try to support at least one Linux on day one and I don't know how to chose which one.  Thanks!

[1] http://dxr.mozilla.org/mozilla-central/source/python/mozboot/mozboot

Comment 3

[Gregory Szorc [:gps]]

Feel free to add arguments or a wizard to mach bootstrap.

Main requirement is that you can run the main bootstrap.py from a fresh OS install. This means supporting Python 2.6 (at least for the initial bootstrap bits - you can always install 2.7 then re-invoke with it if you wanted to get crazy).

Comment 4

[Nick Alexander :nalexander [he/him]]

As a note: a contributor started pushing on this at [1].  Also, this is a little more complicated (I think) than the desktop configuration because we need to feed the paths of things like the Android SDK and NDK into a mozconfig.  (It's possible desktop already does this, but I didn't see it.)

[1] https://github.com/taha-ibrahim-draidia/FennecEnvSetup

Comment 5

[Chris Kitching [:ckitching]]

(In reply to Nick Alexander :nalexander from comment #4)
> As a note: a contributor started pushing on this at [1].  Also, this is a
> little more complicated (I think) than the desktop configuration because we
> need to feed the paths of things like the Android SDK and NDK into a
> mozconfig.

Finding the paths of things on Linux, generally, is nontrivial. Lots of different package managers and so on. You could in principle become compatible with all/lots of them, but that might not be a brilliant use of time.

In general, you're gonna end up writing a lot of depressing distro-specific quirks, so you definitely need to prioritise.

I suggest you prioritise literally by how many users are on the distro. 

While it might be tempting to consider how many programmers might be on a particular distro, I don't think that's wise. A "hackier" distro, such as Arch, will tend to be used by people who already just don't care. Those people have already had to configure complicated and irritating things to make their system work, so they're not going to be too bothered about following a set of clear (but annoyingly long) instructions on the wiki to get their build working.
The distros primarily populated by humans, however, would likely benefit from a shiny red button to transition from pull to working IDE. Going by the latest stats from distrowatch.org, this gives your top three, sort of unsurprisingly, as:
1) Mint
2) Ubuntu
3) Debian


Lastly, I should probably mention the question of "Do we really want to do this?". 

Automating things that happen exactly once isn't necessarily an optimal use of time:
http://xkcd.com/1319/

One could claim that "If a user can't follow a set of simple instructions on a wiki to get started they're not going to be able to write useful code."

Making it easy to go straight from pull to IDE may allow complete beginners to start writing and submitting patches. This could be good: someone's first programming project could be Firefox. This could be the inspiration they need to get into programming, launching them into a life of sunshine, rainbows, and bunny rabbits.
Or this could be bad, leading to an influx of dreadful patches written by people even sloppier than myself, flooding reviewers and slowing progress.

Contributors face other, more substantial problems: the lack of sane documentation comments in many places being the most striking, affecting newcomers and old-timers alike. Making it easy to start up and get into a codebase that is then entirely incomprehensible is possibly even worse than people not being able to build: that might be strongly discouraging for a newcomer, depriving them of the aforementioned bunny rabbits.
On my first day I was tasked with fixing an "obvious edgecase" in a class called something like DoorHangerPopup. The class contained very little in the way of comments, and absolutely nowhere did it say what the heck it *was*. It ended up taking me about a day to solve this problem, but it'd've taken the original author mere seconds to write a little comment at the top in Javadoc notation specifying what this class was, and what it definitely wasn't (to prevent the sort of feature-creep that has led to INIParser extending INISection, for example...).
That, of course, is a rather more nontrivial problem to solve. I also do rather like rolling out this rant whenever someone asks me for an opinion on the subject of contributors being able to get started - not least because it *still* affects me. This problem may very well represent the largest source of wasted developer time that Mozilla currently experiences, and it feels like changing people's attitudes towards it is essentially impossible. "My time is better spent writing more code than spending thirty seconds documenting the function I just wrote", they say... :/

I digress.

Comment 6

[Gregory Szorc [:gps]]

Enabling a turnkey development environment is absolutely a good use of time. If it weren't, tools like Vagrant and Docker wouldn't have taken off at the velocity that have.

Yes, bootstrap tends to have many branches for different operating system. Target the popular ones. People will fill in the gaps on less popular distros. They've done it already with the desktop bootstrap tool. They'll do it again for Fennec bootstrap.

Comment 7

[Nicholas Nethercote [inactive]]

> One could claim that "If a user can't follow a set of simple instructions on a wiki
> to get started they're not going to be able to write useful code."

I've been a Mozilla employee and contributor for six years. I've written thousands of patches. A while back I tried to set up Fennec builds on my Linux desktop and I utterly failed, even after lots of back and forth with Fennec developers on #fennec. I ended up getting it working on my Mac laptop, but even then the whole experience was horrendous.

Comment 8

[:Gavin Sharp [email: [e-mail]]]

(In reply to Chris Kitching [:ckitching] from comment #5)
> One could claim that "If a user can't follow a set of simple instructions on
> a wiki to get started they're not going to be able to write useful code."

And in such cases one would be deeply wrong, so much so that others would feel it necessary to explicitly point that out in Bugzilla lest others confuse it for a broadly-shared Mozilla opinion.

Comment 9

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: bz://1108771/nalexander

Comment 10

[Nick Alexander :nalexander [he/him]]

/r/1561 - Bug 1108771 - Part 1: Prompt for which application to bootstrap. r=gps
/r/1563 - Bug 1108771 - Part 2: Implement |mach bootstrap| for mobile/android on Debian-like systems. r=gps
/r/1565 - Bug 1108771 - Part 3: Factor _ensure_*_packages out of OS X bootstrapper. r=gps
/r/1567 - Bug 1108771 - Part 4: Implement |mach bootstrap| for mobile/android on OS X with homebrew. r=gps

Pull down these commits:

hg pull review -r 74c826c883581856d6ad348f267e069e67e02b74

Comment 11

[Nick Alexander :nalexander [he/him]]

/r/1561 - Bug 1108771 - Part 1: Prompt for which application to bootstrap. r=gps
/r/1563 - Bug 1108771 - Part 2: Implement |mach bootstrap| for mobile/android on Debian-like systems. r=gps
/r/1565 - Bug 1108771 - Part 3: Factor _ensure_*_packages out of OS X bootstrapper. r=gps
/r/1567 - Bug 1108771 - Part 4: Implement |mach bootstrap| for mobile/android on OS X with homebrew. r=gps

Pull down these commits:

hg pull review -r 74c826c883581856d6ad348f267e069e67e02b74

Comment 12

[Nick Alexander :nalexander [he/him]]

stefan: you offered to test this, at least on Mac.  (If you run a Debian-based system, I'd appreciate you testing there too.)

mcomella: you run an Arch-based system; perhaps take a look at getting this running on Arch in your copious spare time?

Comment 13

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/1561/#review965

Most excellent.

Comment 14

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/1563/#review967

::: python/mozboot/mozboot/android.py
(Diff revision 1)
> +# If we add unicode_literals, Python 2.6.1 (required for OS X 10.6) breaks.

I think there was talking of dropping OS X 10.6 support from bootstrap. Meh.

::: python/mozboot/mozboot/android.py
(Diff revision 1)
> +        subprocess.check_call(['wget', '--continue', url])

It may not be a safe assumption that wget is installed. Might want to add it to system packages if it isn't there already.

::: python/mozboot/mozboot/android.py
(Diff revision 1)
> +            cmd = ['tar', 'jvxf']

IIRC some versions of tar don't accept the options like this. BSDs perhaps? Python has built-in support for reading archives. That's probably safer. https://docs.python.org/2/library/archiving.html

::: python/mozboot/mozboot/debian.py
(Diff revision 1)
> +        mozbuild_path = os.path.expanduser(os.path.join('~', '.mozbuild'))

This should honor the `MOZBUILD_STATE_PATH` env variable if it is set.

::: python/mozboot/mozboot/debian.py
(Diff revision 1)
> +        self.sdk_path = os.getenv('ANDROID_SDK_HOME', os.path.join(mozbuild_path, 'android-sdk-linux'))
> +        self.ndk_path = os.getenv('ANDROID_NDK_HOME', os.path.join(mozbuild_path, 'android-ndk-r8e'))

Please form the habit of using os.environ for environment gets and sets.

Comment 15

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/1565/#review969

Ship It!

Comment 16

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/1567/#review971

::: python/mozboot/mozboot/osx.py
(Diff revision 1)
> +        raise NotImplementedError("We don't yet support bootstrapping Firefox for Android with Macports. " +
> +                                  "We don't know of a package that installs the Java 7 JDK.")
> +
> +    def suggest_macports_mobile_android_mozconfig(self):
> +        raise NotImplementedError("We don't yet support bootstrapping Firefox for Android with Macports. " +
> +                                  "We don't know of a package that installs the Java 7 JDK.")

A link to a bug URL in the error message would be helpful.

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8538190 [details]
MozReview Request: bz://1108771/nalexander

Great patch series! I love seeing improvements to the on-boarding process.

I can't remember if you were around, but sometime during Portlandia I made a comment that I would like to add a wizard or something to make creating mozconfigs easier. I love the suggested mozconfigs snippets you snuck into this patch series. But they may be short-lived if we ever get a proper wizard built into mach.

Comment 18

[Nick Alexander :nalexander [he/him]]

/r/1561 - Bug 1108771 - Part 1: Prompt for which application to bootstrap. r=gps
/r/1563 - Bug 1108771 - Part 2: Implement |mach bootstrap| for mobile/android on Debian-like systems. r=gps
/r/1565 - Bug 1108771 - Part 3: Factor _ensure_*_packages out of OS X bootstrapper. r=gps
/r/1567 - Bug 1108771 - Part 4: Implement |mach bootstrap| for mobile/android on OS X with homebrew. r=gps

Pull down these commits:

hg pull review -r b1f31559ab561303878f8ee4a4c97c67d8208e3d

Comment 19

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Nick Alexander :nalexander from comment #12)
> mcomella: you run an Arch-based system; perhaps take a look at getting this
> running on Arch in your copious spare time?

Sure, I filed bug 1114718.

Comment 20

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/1559/#review1077

Ship It!

Comment 21

[Nick Alexander :nalexander [he/him]]

https://reviewboard.mozilla.org/r/1563/#review1017

> It may not be a safe assumption that wget is installed. Might want to add it to system packages if it isn't there already.

Yeah, remnant from when this was part of a Bootstrapper class that added wget to install_system_packages.  I don't think it's worth trying to make this bullet-proof; I'll add a note in the docstring and add it to the classes I've implemented.

> IIRC some versions of tar don't accept the options like this. BSDs perhaps? Python has built-in support for reading archives. That's probably safer. https://docs.python.org/2/library/archiving.html

I considered this, looked at the options, and concluded that it's not one line (why not?) and that I didn't want Python unpacking .5G badly.  I'd rather future devs improve the shell tool chain than restrict to whatever Python supports.

Comment 22

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/integration/fx-team/rev/f5758bdccc0a
https://hg.mozilla.org/integration/fx-team/rev/3ecd0c6da57b
https://hg.mozilla.org/integration/fx-team/rev/2b353a88f2b9
https://hg.mozilla.org/integration/fx-team/rev/cfbcc324f038

Comment 23

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/f5758bdccc0a
https://hg.mozilla.org/mozilla-central/rev/3ecd0c6da57b
https://hg.mozilla.org/mozilla-central/rev/2b353a88f2b9
https://hg.mozilla.org/mozilla-central/rev/cfbcc324f038

Comment 25

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1108771 - Part 1: Prompt for which application to bootstrap. r=gps

Comment 26

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1108771 - Part 2: Implement |mach bootstrap| for mobile/android on Debian-like systems. r=gps

Comment 27

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1108771 - Part 3: Factor _ensure_*_packages out of OS X bootstrapper. r=gps

Comment 28

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1108771 - Part 4: Implement |mach bootstrap| for mobile/android on OS X with homebrew. r=gps