777530 - Setup B2G panda builds in production

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Jonathan Griffin (:jgriffin)]

B2G test automation on real hardware will be focused on panda boards.  We need to get B2G panda builds in production so we can start developing the rest of the test infrastructure around them.

Instructions for building B2G for pandas:

1. git clone git://github.com/mozilla-b2g/B2G.git
2. cd B2G
3. ./config.sh pandaboard
4. ./build.sh

These builds shouldn't be publicly available; they will contain proprietary codecs (like the build in bug 769450).

We're still working on identifying the best mechanism to flash theses onto the actual panda boards.

Comment 1

[Armen [:armenzg]]

jgriffin, it seems that our current blocker (besides setting the pandas on racks) is figuring out the flashing process.
Is this assessment accurate?

Is this b2g build the one that you see with highest priority?

Do you have one of these builds up somewhere so people can attempt to install it on a panda?

Is there any other blockers that you are aware of?

Would we just testing that we can install the OS on the panda board?
Would we have unit tests work coming up after the panda gets re-imaged with this image?

For now it seems that we can get started by producing these builds and putting them on a private location on ftp.

> We're still working on identifying the best mechanism to flash theses onto
> the actual panda boards.
>
Adding Jake from ServerOps:Releng who has worked with the re-imaging process for the panda boards.
Does it sound good to file a bug?

Adding hwine and moconnor to be on the loop of possible IT work derived from this work.

I think we could create a bug to "test the builds produced in bug 777530" which would be dependent on "figure out flashing approach" and "set up the pandas up".

Sounds good?

Comment 2

[Armen [:armenzg]]

This is build#14 on https://etherpad.mozilla.org/b2g-builds
and has priority#2.

#1 is builds for Otoro which are not on releng's plate.

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #1)
> jgriffin, it seems that our current blocker (besides setting the pandas on
> racks) is figuring out the flashing process.
> Is this assessment accurate?
> 

Yes.

> Is this b2g build the one that you see with highest priority?

Yes.

> 
> Do you have one of these builds up somewhere so people can attempt to
> install it on a panda?

No, I'll make a build and post a link on this bug.

> 
> Is there any other blockers that you are aware of?

None at this time, but I wouldn't be surprised to find others as this work proceeds.
> 
> Would we just testing that we can install the OS on the panda board?
> Would we have unit tests work coming up after the panda gets re-imaged with
> this image?

Yes, we'll have unittest work coming up after the panda gets re-imaged.  We're working on porting mochitests, reftests, and xpcshell tests to B2G panda.  We already have them working on B2G emulators, and don't anticipate that getting them to work on pandas will be a lot of extra work, beyond what's required to get B2G running smoothly on pandas in general.

> 
> For now it seems that we can get started by producing these builds and
> putting them on a private location on ftp.
> 
> > We're still working on identifying the best mechanism to flash theses onto
> > the actual panda boards.
> >
> Adding Jake from ServerOps:Releng who has worked with the re-imaging process
> for the panda boards.

I'm also cc'ing Ted, who I think is involved with this.

> Does it sound good to file a bug?
> 
> Adding hwine and moconnor to be on the loop of possible IT work derived from
> this work.
> 
> I think we could create a bug to "test the builds produced in bug 777530"
> which would be dependent on "figure out flashing approach" and "set up the
> pandas up".
> 
> Sounds good?

Sounds excellent, thanks!

Comment 4

[(not currently active) Ted Mielczarek]

I talked to mwu about this, and the B2G builds don't use u-boot as a boot loader like the Linaro Android image we're using on the pandas. u-boot makes life a lot easier because it lets us do PXE netbooting so we can re-image devices remotely.

Also, we'll want to make sure we get the kernel patch that Linaro's kernel has to fix the MAC address:
https://bugzilla.mozilla.org/show_bug.cgi?id=731670#c6

Comment 5

[Chris AtLee [:catlee]]

Which parts of the build need to be uploaded?

It looks like output is generated in out/target/product/panda:
-> % ls -lhS
total 98M
-rw-r--r--  1 catlee catlee  76M Aug 16 12:37 system.img
-rw-r--r--  1 catlee catlee  11M Aug 16 11:57 userdata.img
-rw-r--r--  1 catlee catlee 4.0M Aug 16 11:57 recovery.img
-rw-r--r--  1 catlee catlee 3.7M Aug 16 11:57 boot.img
-rwxr-xr-x  1 catlee catlee 3.5M Aug 16 11:56 kernel
-rw-r--r--  1 catlee catlee 492K Aug 16 11:57 ramdisk-recovery.img
-rw-r--r--  1 catlee catlee 162K Aug 16 11:57 ramdisk.img
-rw-r--r--  1 catlee catlee  36K Aug 16 12:37 installed-files.txt
-rw-r--r--  1 catlee catlee  12K Aug 16 11:54 clean_steps.mk
drwxr-xr-x  3 catlee catlee 4.0K Aug 16 11:57 data
drwxr-xr-x 12 catlee catlee 4.0K Aug 16 12:37 obj
drwxr-xr-x  3 catlee catlee 4.0K Aug 16 11:57 recovery
drwxr-xr-x  8 catlee catlee 4.0K Aug 16 11:57 root
drwxr-xr-x  5 catlee catlee 4.0K Aug 16 11:57 symbols
drwxr-xr-x 11 catlee catlee 4.0K Aug 16 12:37 system
-rw-r--r--  1 catlee catlee  569 Aug 16 11:54 previous_build_config.mk
-rw-r--r--  1 catlee catlee   21 Aug 16 11:54 android-info.txt

Comment 6

[Jonathan Griffin (:jgriffin)]

For flashing, we only need system.img, userdata.img, and boot.img.

For forensic purposes, we probably also want system/build.prop and system/sources.xml.

Comment 7

[Chris AtLee [:catlee]]

not focusing on this ATM - looking into improving test capacity on other platforms. should be back on this next week.

Comment 8

[Jonathan Griffin (:jgriffin)]

We can probably use the same approach for this we are considering for emulator tests.  That is, use some process to create a panda build and stage it somewhere as a static image.  Then, for each test run, flash this static image onto the panda, then install the relevant gecko build (the armv7a ICS gecko build already being created by buildbot) onto the panda, and then run the tests.

We'd have to figure out how to handle Gaia for tests in which that matters, in this scenario.

The static panda images would have to be updated "infrequently" (fortnightly at worst, according to cjones), but it's not clear how often that would be.  To make life easier, it's probably best if there were some automation to handle this.

Whether we use this approach or the 'full build on each commit' approach is probably up to rel-eng, since either way would provide an adequate basis for test automation.

Comment 9

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for panda builds

Comment 10

[cmtalbert]

(In reply to Jonathan Griffin (:jgriffin) from comment #8)
> We can probably use the same approach for this we are considering for
> emulator tests.  That is, use some process to create a panda build and stage
> it somewhere as a static image.  Then, for each test run, flash this static
> image onto the panda, then install the relevant gecko build (the armv7a ICS
> gecko build already being created by buildbot) onto the panda, and then run
> the tests.
..snip..
> Whether we use this approach or the 'full build on each commit' approach is
> probably up to rel-eng, since either way would provide an adequate basis for
> test automation.

I think I'd want to shy away from just updating gecko on the panda boards. Once we have the panda flashing worked out, I'd like to flash the panda with a fresh OS each time unless that proves unmaintainable.  This gets us away from errors where we fail to update gaia and/or gonk on the fortnightly basis. Also, this would enable people to run gonk changes against try, which I think is something we will want eventually.  So, flashing the entire OS seems the safer and more hands off option unless it doesn't work for some reason.

Comment 11

[Justin Wood (:Callek)]

Comment on attachment 660943 [details] [diff] [review]
buildbot-configs for panda builds

>diff --git a/mozilla/b2g_config.py b/mozilla/b2g_config.py
>index 67b8676..7442a38 100644
>--- a/mozilla/b2g_config.py
>+++ b/mozilla/b2g_config.py
>@@ -17,16 +17,17 @@ GLOBAL_VARS.update(b2g_localconfig.GLOBAL_VARS.copy())
> 
> GLOBAL_VARS.update({
>     'platforms': {
>         'ics_armv7a_gecko': {},
>         'ics_armv7a_gecko-debug': {},
>         'linux32_gecko': {},
>         'macosx64_gecko': {},
>         'win32_gecko': {},
>+        'panda': {},

nit: can we do a panda-b2g or some such naming, since we'll be using this platform for Fennec with a completely different slave pool and build-process?

Comment 12

[Chris AtLee [:catlee]]

(In reply to Clint Talbert ( :ctalbert ) from comment #10)
> (In reply to Jonathan Griffin (:jgriffin) from comment #8)
> > We can probably use the same approach for this we are considering for
> > emulator tests.  That is, use some process to create a panda build and stage
> > it somewhere as a static image.  Then, for each test run, flash this static
> > image onto the panda, then install the relevant gecko build (the armv7a ICS
> > gecko build already being created by buildbot) onto the panda, and then run
> > the tests.
> ..snip..
> > Whether we use this approach or the 'full build on each commit' approach is
> > probably up to rel-eng, since either way would provide an adequate basis for
> > test automation.
> 
> I think I'd want to shy away from just updating gecko on the panda boards.
> Once we have the panda flashing worked out, I'd like to flash the panda with
> a fresh OS each time unless that proves unmaintainable.  This gets us away
> from errors where we fail to update gaia and/or gonk on the fortnightly
> basis. Also, this would enable people to run gonk changes against try, which
> I think is something we will want eventually.  So, flashing the entire OS
> seems the safer and more hands off option unless it doesn't work for some
> reason.

My current approach is going to use a static gonk snapshot which is referenced by a manifest in the mozilla-central code. So gonk will not be built as part of the panda builds.

Testing new gonk versions on try would be possible if you could upload a new gonk snapshot...but they're very large at the moment (~700MB), so we may need to look at other options here.

Comment 13

[Chris AtLee [:catlee]]

Attachment: mozharness script for b2g builds

Comment 14

[cmtalbert]

(In reply to Chris AtLee [:catlee] from comment #12)

> My current approach is going to use a static gonk snapshot which is
> referenced by a manifest in the mozilla-central code. So gonk will not be
> built as part of the panda builds.
> 
> Testing new gonk versions on try would be possible if you could upload a new
> gonk snapshot...but they're very large at the moment (~700MB), so we may
> need to look at other options here.

Ok, that sounds ok for now. Let's get it in staging and see how much of a problem replacing only the Gecko is or isn't. There are a lot of variables here and we really don't know what we don't know yet.

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 661235 [details] [diff] [review]
mozharness script for b2g builds

This looks good.
There's a lot of commentary below, but I think the main items are:

* more error checking at some point
* do we want/need to reboot?  pull build/tools and set reboot_command
* set buildbot properties?

I think the lack of reboot is the main thing that might hurt other jobs, though it's probably less painful on linux build slaves than windows or test slaves.

>diff --git a/configs/b2g/releng.py b/configs/b2g/releng.py
>new file mode 100644
>--- /dev/null
>+++ b/configs/b2g/releng.py
>@@ -0,0 +1,18 @@
>+#!/usr/bin/env python
>+import os
>+config = {
>+    "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"),
>+    "ssh_user": "b2gbld",
>+    "upload_remote_host": "stage.mozilla.org",

This is good for production.
I don't currently have a way to make it easy to [automatically] switch to a staging config file for non-release automation... if you have any ideas, they're welcome.

>diff --git a/mozharness/mozilla/tooltool.py b/mozharness/mozilla/tooltool.py
>new file mode 100644
>--- /dev/null
>+++ b/mozharness/mozilla/tooltool.py
>@@ -0,0 +1,12 @@
>+class TooltoolMixin(object):
>+    """Mixin class for handling tooltool manifests.
>+    Requires self.config['tooltool_servers'] to be a list of base urls
>+    """
>+    def tooltool_fetch(self, manifest, output_dir=None):
>+        """docstring for tooltool_fetch"""
>+        tooltool = self.query_exe('tooltool.py')
>+        cmd = [tooltool]

You can use query_exe('tooltool.py', return_type='list') for this.
Then you won't hit an error if someone specifies the tooltool.py exe as a list.

I'm strongly considering making return_type='list' default, but I haven't made my mind up about it yet.

>+        for s in self.config['tooltool_servers']:
>+            cmd.extend(['--url', s])
>+        cmd.extend(['fetch', '-m', manifest, '-o'])
>+        self.run_command(cmd, cwd=output_dir)

Do we have/want any sort of error checking here?
At the very least we may want to set error_list=PythonErrorList ?

>+class B2GBuild(MockMixin, BaseScript, VCSMixin, TooltoolMixin, TransferMixin, BuildbotMixin):

Instead of inheriting BaseScript and VCSMixin, you can inherit MercurialScript, which does some of this for you.

But this works, too.

>+    def _pre_config_lock(self, rw_config):
>+        super(B2GBuild, self)._pre_config_lock(rw_config)
>+
>+        if self.buildbot_config is None:
>+            self.read_buildbot_config()
>+
>+        if self.buildbot_config and 'properties' in self.buildbot_config:
>+            self.info("Using repo/branch config from build properties")
>+            self.config['repo'] = 'http://hg.mozilla.org/%s' % self.buildbot_config['properties']['repo_path']
>+            self.config['branch'] = self.buildbot_config['properties']['branch']

I'm not thrilled that we're starting to overwrite config, but it's good that we're logging it.  In the end, this may be less ugly than having to reference a bunch of config dictionaries and self.variables.

I was going to ask that you log the values, as well, but I suppose that will show up when we dump the config to the log.

>+    def query_branch(self):
>+        if self.buildbot_config is None:
>+            self.read_buildbot_config()

Hm, I'm not sure we'll ever hit this since you're calling read_buildbot_config() in _pre_config_lock().

Doesn't hurt though, other than a small coverage hit.

>+        if self.buildbot_config and 'sourcestamp' in self.buildbot_config:
>+            self.vcs_checkout(repo=c['repo'], dest=dirs['src'], revision=self.buildbot_config['sourcestamp']['revision'])
>+        else:
>+            self.vcs_checkout(repo=c['repo'], dest=dirs['src'])

You may need to also check out build/tools if you want these jobs to reboot at the end; we currently handle that by setting ScriptFactory's reboot_command to count_and_reboot.py.

>+        if 'mock_target' in gecko_config:
>+            # initialize mock
>+            self.setup_mock(gecko_config['mock_target'], gecko_config['mock_packages'])
>+            if self.config['ccache']:
>+                self.run_mock_command(gecko_config['mock_target'], 'ccache -z', cwd=dirs['work_dir'], env=env)
>+
>+            code = self.run_mock_command(gecko_config['mock_target'], cmd, cwd=dirs['work_dir'], env=env)
>+            if self.config['ccache']:
>+                self.run_mock_command(gecko_config['mock_target'], 'ccache -s', cwd=dirs['work_dir'], env=env)
>+        else:
>+            if self.config['ccache']:
>+                self.run_command('ccache -z', cwd=dirs['work_dir'], env=env)
>+            code = self.run_command(cmd, cwd=dirs['work_dir'], env=env)
>+            if self.config['ccache']:
>+                self.run_command('ccache -s', cwd=dirs['work_dir'], env=env)

It would be good to get error_lists for the above commands for easier log parsing.  Doesn't block landing, but I'd want a followup bug for that at least.

You may also want to use self.set_buildbot_property() so these builds have similar information for tbpl and downstream pulse listeners to reference.

Comment 16

[Aki Sasaki (not active)]

Hm, we could also change the fatal() to self.return_code = -1 or something.
That way the job will still turn red, but we'll upload the logs.

Comment 17

[Chris AtLee [:catlee]]

Attachment: mozharness_config support for generateBranchObjects

Comment 18

[Chris AtLee [:catlee]]

Attachment: in-tree configs for panda builds

Comment 19

[Chris AtLee [:catlee]]

Attachment: mozharness script for b2g builds

Comment 20

[Chris AtLee [:catlee]]

Attachment: mozharness_config support for generateBranchObjects

use base_name!

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 662231 [details] [diff] [review]
mozharness_config support for generateBranchObjects

Review of attachment 662231 [details] [diff] [review]:
-----------------------------------------------------------------

Much better, thanks!

Comment 22

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for panda builds

updated base_name for all b2g platforms

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 662233 [details] [diff] [review]
buildbot-configs for panda builds

Review of attachment 662233 [details] [diff] [review]:
-----------------------------------------------------------------

Watch out for TBPL display bustage when you land, but I _think_ it will be okay based on https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/default/js/Data.js#l511.

Comment 24

[Michael Wu [:mwu]]

Comment on attachment 662146 [details] [diff] [review]
in-tree configs for panda builds

One small thing worth noting - b2g builds shouldn't require java, though there are bogus java warnings complaining about it.

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 662146 [details] [diff] [review]
in-tree configs for panda builds

https://hg.mozilla.org/integration/mozilla-inbound/rev/576754febc78

Comment 26

[Rob Hawkes [:rob]]

Are these Panda builds going to be available anywhere for others to use? Like the daily B2G builds for the various phones.

Comment 27

[Chris AtLee [:catlee]]

They'll be available on ftp.

Comment 28

[Andrew Halberstadt [:ahal]]

Just keep in mind these can't be made publicly available.

Comment 29

[Chris AtLee [:catlee]]

(In reply to Andrew Halberstadt [:ahal] from comment #28)
> Just keep in mind these can't be made publicly available.

Not even the panda builds?

Comment 30

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://hg.mozilla.org/mozilla-central/rev/576754febc78

Comment 31

[(not currently active) Ted Mielczarek]

I think the question is about the binary driver blobs:
https://developers.google.com/android/nexus/drivers#panda

I had never actually read the license agreement for them, but (IANAL), it sounds like they do actually grant you a license to redistribute OS builds including their drivers for non-commercial use. We probably need a legal bug to determine if this is kosher.

Comment 32

[(not currently active) Ted Mielczarek]

I filed bug 792814 on that.

Comment 33

[Chris AtLee [:catlee]]

we're not done here yet!

still todo:
decide on final upload location
turn on builds

Comment 34

[Chris AtLee [:catlee]]

These builds don't work on try because they can't pull in the repo.

Comment 35

[Chris AtLee [:catlee]]

Attachment: add timeouts to mozharness jobs, make sure they're pretty

the pretty name stuff is to make sure they're available to try chooser (via -p all for now)

Comment 36

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for panda builds

pretty much the same as before, except refactoring some of the cmdline switches into mozharness configs, and adding a new config for try

Comment 37

[Chris AtLee [:catlee]]

Attachment: mozharness fixes for b2g

changes from before:

adds make_updates action
adds a releng-try config
adds 'default_actions' for the releng/releng-try configs
fixes subprocess line buffering
adds hgtool vcs provider
changes build targets to build .tar.bz2 files instead of .img files
adds support for try uploads
removes hgtool.py symlink

Comment 38

[Aki Sasaki (not active)]

Comment on attachment 664299 [details] [diff] [review]
mozharness fixes for b2g

You're not specifying hgtool.py in exes, so it needs to be in PATH.

Does this hgtool retry?  If so I may make an HGToolScript class and port most/all of the MercurialVCS jobs to it to solve bug 793642.

>diff --git a/scripts/b2g_build.py b/scripts/b2g_build.py
>--- a/scripts/b2g_build.py
>+++ b/scripts/b2g_build.py
<snip>
>+        c = self.config
>+        if c['enable_try_uploads']:
>+            try:
>+                user = self.buildbot_config['sourcestamp']['changes'][0]['who']
>+            except KeyError:
>+                user = "unkown"

Nit: sp unknown ?

Lgtm!

Comment 39

[Chris AtLee [:catlee]]

(In reply to Aki Sasaki [:aki] from comment #38)
> Comment on attachment 664299 [details] [diff] [review]
> mozharness fixes for b2g
> 
> You're not specifying hgtool.py in exes, so it needs to be in PATH.

It will be in /usr/bin or /usr/local/bin depending on how bug 764077 works out.

> Does this hgtool retry?  If so I may make an HGToolScript class and port
> most/all of the MercurialVCS jobs to it to solve bug 793642.

No, it doesn't retry. It has a bunch of fallbacks configured (bundle, mirrors), but if they all fail, it won't retry from the top. Do we want that?

> >diff --git a/scripts/b2g_build.py b/scripts/b2g_build.py
> >--- a/scripts/b2g_build.py
> >+++ b/scripts/b2g_build.py
> <snip>
> >+        c = self.config
> >+        if c['enable_try_uploads']:
> >+            try:
> >+                user = self.buildbot_config['sourcestamp']['changes'][0]['who']
> >+            except KeyError:
> >+                user = "unkown"
> 
> Nit: sp unknown ?

Good catch!

Comment 40

[Aki Sasaki (not active)]

(In reply to Chris AtLee [:catlee] from comment #39)
> > Does this hgtool retry?  If so I may make an HGToolScript class and port
> > most/all of the MercurialVCS jobs to it to solve bug 793642.
> 
> No, it doesn't retry. It has a bunch of fallbacks configured (bundle,
> mirrors), but if they all fail, it won't retry from the top. Do we want that?

Hm, bug 793642 is asking for it, but I wonder if your fallbacks will help enough to lower priority on that.

Either way, I think that's scope creep here.

Comment 41

[Chris AtLee [:catlee]]

Attachment: remove Android.mk files from breakpad; update b2g panda build reqs

turns out we don't need java to build, but we do need perl-Digest-SHA for the 'shasum' command which is used by the b2g build process.

the android.mk files in breakpad are breaking the b2g build system.

Comment 42

[Mounir Lamouri (:mounir)]

https://hg.mozilla.org/mozilla-central/rev/554edcc7f2ab

Comment 43

[Chris AtLee [:catlee]]

Looks like hg share isn't working quite right:

09:07:08     INFO -  Trying to share /builds/hg-shared/integration/mozilla-inbound to /builds/slave/b2g-m-in-panda-dep/build/gecko
09:07:08     INFO -  command: START
09:07:08     INFO -  command: hg share -U /builds/hg-shared/integration/mozilla-inbound /builds/slave/b2g-m-in-panda-dep/build/gecko
09:07:08     INFO -  command: cwd: /builds/slave/b2g-m-in-panda-dep
09:07:08     INFO -  command: output:
09:07:08    ERROR -  abort: No such file or directory: /builds/slave/b2g-m-in-panda-dep/build/gecko
09:07:08    ERROR -  Automation Error: hg not responding
09:07:08     INFO -  command: ERROR
09:07:08     INFO -  Traceback (most recent call last):
09:07:08     INFO -    File "<string>", line 42, in run_cmd
09:07:08     INFO -    File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
09:07:08     INFO -      raise CalledProcessError(retcode, cmd)
09:07:08     INFO -  CalledProcessError: Command '['hg', 'share', '-U', '/builds/hg-shared/integration/mozilla-inbound', '/builds/slave/b2g-m-in-panda-dep/build/gecko']' returned non-zero exit status 255
09:07:08     INFO -  command: END (0.39s elapsed)

probably need to create hte directory first

Comment 44

[Ed Morley [:emorley]]

Comment on attachment 664296 [details] [diff] [review]
buildbot-configs for panda builds

>diff --git a/mozilla/b2g_config.py b/mozilla/b2g_config.py
>-builder_prefix = "B2G "
>+builder_prefix = "b2g"

This broke TBPL's regexes for B2G, filed bug 794951.

Comment 45

[Ed Morley [:emorley]]

Please can someone CC me on bug 792814; it's been mentioned in a bug to which I am assigned, which is a little frustrating when one can't access it :-)

Comment 46

[(not currently active) Ted Mielczarek]

That's just a legal bug, "Determine if we can redistribute B2G OS builds for Pandaboard including binary drivers". The answer was "Yes, as long as we include a README that notifies downloaders of the terms of redistribution."

Comment 47

[Ed Morley [:emorley]]

(In reply to Ted Mielczarek [:ted] from comment #46)
> That's just a legal bug, "Determine if we can redistribute B2G OS builds for
> Pandaboard including binary drivers". The answer was "Yes, as long as we
> include a README that notifies downloaders of the terms of redistribution."

Ah, thank you :-)

Comment 48

[Chris AtLee [:catlee]]

Going to call this fixed.