Closed Bug 1344244 Opened 7 years ago Closed 7 years ago

mach bootstrap for Android (non-artifact) completes without installing correct Rust Android toolchain or build-tools version

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox56 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla56
Tracking Flags:
Tracking Status
firefox56 --- fixed

People

(Reporter: mconley, Assigned: nalexander)

References

Details

Attachments

(6 files)

Bug 1344244 - Pre: Support mobile/android in bootstrap on CentOS.
59 bytes, text/x-review-board-request
glandium
: review+
Details
Bug 1344244 - Part 1: Accept newer Android lint version.
59 bytes, text/x-review-board-request
glandium
: review+
Details
Bug 1344244 - Part 2: Merge similar code into android.py.
59 bytes, text/x-review-board-request
glandium
: review+
Details
Bug 1344244 - Part 3: Use sdkmanager rather than android.
59 bytes, text/x-review-board-request
glandium
: review+
Details
Bug 1344244 - Part 4: Homebrew includes |brew cask| directly now.
59 bytes, text/x-review-board-request
glandium
: review+
Details
Bug 1344244 - Post: Allow --application-choice to |mach bootstrap|.
59 bytes, text/x-review-board-request
glandium
: review+
Details 
Not sure if I was just in a funky state or something, but I ran ./mach bootstrap, and chose to build Firefox for Android (non-artifact). A bunch of stuff downloaded, and I agreed to some licenses, and then it spewed out some stuff that I needed to put into a .mozconfig.

I did that, and ran ./mach build.

It errored out with:

0:06.13 DEBUG: Executing: `/Users/mikeconley/.cargo/bin/rustc --crate-type staticlib --target=armv7-linux-androideabi -o /var/folders/bq/n5qs5f650bl7kmj4h_fqsw2m0000gn/T/conftestHuUo9z.rlib /var/folders/bq/n5qs5f650bl7kmj4h_fqsw2m0000gn/T/conftestYHjULo.rs`
 0:06.13 DEBUG: The command returned non-zero exit status 101.
 0:06.14 DEBUG: Its error output was:
 0:06.14 DEBUG: | error[E0463]: can't find crate for `std`
 0:06.14 DEBUG: |   |
 0:06.14 DEBUG: |   = note: the `armv7-linux-androideabi` target may not be installed
 0:06.14 DEBUG: |
 0:06.14 DEBUG: | error: aborting due to previous error
 0:06.14 DEBUG: |
 0:06.14 ERROR: Cannot compile for arm-unknown-linux-androideabi with /Users/mikeconley/.cargo/bin/rustc
 0:06.14 The target may be unsupported, or you may not have
 0:06.14 a rust std library for that target installed. Try:
 0:06.14
 0:06.14   rustup target add armv7-linux-androideabi

After running that rustup command, I re-ran ./mach build and got this:

0:08.06 checking if app-specific confvars.sh exists... /Users/mikeconley/Projects/mozilla-central/mobile/android/confvars.sh
 0:08.08 checking for Android SDK platform version 23... /Users/mikeconley/.mozbuild/android-sdk-macosx/platforms/android-23
 0:08.08 checking for Android build-tools... configure: error: You must install the Android build-tools version 23.0.3.  Try |mach bootstrap|.  (Looked for /Users/mikeconley/.mozbuild/android-sdk-macosx/build-tools/23.0.3)
 0:08.10 DEBUG: <truncated - see config.log for full output>
 0:08.10 DEBUG:     something starting with __ and the normal name is an alias.  */
 0:08.10 DEBUG: #if defined (__stub_malloc_usable_size) || defined (__stub___malloc_usable_size)
 0:08.10 DEBUG: choke me
 0:08.10 DEBUG: #else
 0:08.10 DEBUG: malloc_usable_size();
 0:08.10 DEBUG: #endif
 0:08.10 DEBUG:
 0:08.10 DEBUG: ; return 0; }
 0:08.10 DEBUG: configure:9556: checking whether malloc_usable_size definition can use const argument
 0:08.10 DEBUG: configure:9567: /Users/mikeconley/.mozbuild/android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -std=gnu99 -c -fno-short-enums -fno-exceptions  -march=armv7-a -mthumb -mfpu=vfp -mfloat-abi=softfp -mno-unaligned-access -fno-strict-aliasing -fno-math-errno -idirafter /Users/mikeconley/.mozbuild/android-ndk-r11b/platforms/android-9/arch-arm/usr/include  conftest.c 1>&5
 0:08.10 DEBUG: configure:9589: checking for valloc in malloc.h
 0:08.10 DEBUG: configure:9614: checking for valloc in unistd.h
 0:08.10 DEBUG: configure:9769: checking NSPR selection
 0:08.10 DEBUG: configure:10503: checking for gzread in -lz
 0:08.10 DEBUG: configure:10522: /Users/mikeconley/.mozbuild/android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -std=gnu99 -o conftest -fno-short-enums -fno-exceptions  -march=armv7-a -mthumb -mfpu=vfp -mfloat-abi=softfp -mno-unaligned-access -fno-strict-aliasing -fno-math-errno -idirafter /Users/mikeconley/.mozbuild/android-ndk-r11b/platforms/android-9/arch-arm/usr/include   -L/Users/mikeconley/.mozbuild/android-ndk-r11b/platforms/android-9/arch-arm/usr/lib -Wl,-rpath-link=/Users/mikeconley/.mozbuild/android-ndk-r11b/platforms/android-9/arch-arm/usr/lib --sysroot=/Users/mikeconley/.mozbuild/android-ndk-r11b/platforms/android-9/arch-arm -Wl,--allow-shlib-undefined  -mthumb -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -Wl,--hash-style=sysv conftest.c -lz  -llog  1>&5
 0:08.10 DEBUG: configure:10557: /Users/mikeconley/.mozbuild/android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-gcc -std=gnu99 -c -fno-short-enums -fno-exceptions  -march=armv7-a -mthumb -mfpu=vfp -mfloat-abi=softfp -mno-unaligned-access -fno-strict-aliasing -fno-math-errno -idirafter /Users/mikeconley/.mozbuild/android-ndk-r11b/platforms/android-9/arch-arm/usr/include  conftest.c 1>&5
 0:08.10 DEBUG: configure:10912: checking if app-specific confvars.sh exists
 0:08.10 DEBUG: configure:11077: checking for Android SDK platform version 23
 0:08.10 DEBUG: configure:11085: checking for Android build-tools
 0:08.10 DEBUG: configure: error: You must install the Android build-tools version 23.0.3.  Try |mach bootstrap|.  (Looked for /Users/mikeconley/.mozbuild/android-sdk-macosx/build-tools/23.0.3)
 0:08.10 ERROR: old-configure failed
 0:08.12 *** Fix above errors and then restart with               "/Applications/Xcode.app/Contents/Developer/usr/bin/make -f client.mk build"
 0:08.12 make: *** [configure] Error 1

I ended up having to go into my .mozbuild directory, finding the android package manager (~/.mozbuild/android-sdk-macosx/tools/android for me), and installing build-tools 23.0.3 manually.

The build appears to be working now, but bootstrap should probably do all of this stuff automatically.
I also had to use "~/.mozbuild/android-sdk-macosx/tools/android" to install "Android SDK Build-tools, revision 23.0.3" as described here:

http://stackoverflow.com/a/19416222

I'm using Mac OS X 10.12 and this was the first time I set up an Android build on this machine.
I got both errors bootstrapping for Firefox for Android today.
... except for some reason I don't have ~/.mozbuild/android-sdk-linux/tools/android.
(In reply to Mike Hommey [:glandium] from comment #3)
> ... except for some reason I don't have
> ~/.mozbuild/android-sdk-linux/tools/android.

So, for more completeness: I do have this file after I run mach bootstrap the first time (when there was no ~/.mozbuild/android-sdk-linux), and it disappears after running mach bootstrap a second time. The same also happens when I run android update sdk --no-ui manually after the first mach bootstrap.
One part of the problem is that the sdk we pull doesn't like that our list of sdk packages contains build-tools-23.0.3, when android list sdk only mentions build-tools-23.0.1... If I change the version mentioned in python/mozboot/mozboot/android.py, and rerun mach bootstrap after deleting ~/.mozbuild/android-sdk-linux, it works, and mach configure is happy again. That means effectively reverting half of bug 1247047.
This refactoring unifies similar code that has been copy-pasted and
subsequently diverged.

Review commit: https://reviewboard.mozilla.org/r/154708/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/154708/
Comment on attachment 8883767 [details]
Bug 1344244 - Pre: Support mobile/android in bootstrap on CentOS.

https://reviewboard.mozilla.org/r/154704/#review161088

::: python/mozboot/mozboot/centosfedora.py:25
(Diff revision 1)
>          self.group_packages = []
>  
>          self.packages = [
>              'autoconf213',
>              'mercurial',
> +            'which',

We use `which` from the shell in the build system?

::: python/mozboot/mozboot/centosfedora.py:48
(Diff revision 1)
>              'wireless-tools-devel',
>              'yasm',
>          ]
>  
> -        self.mobile_android_packages = []
> +        self.mobile_android_packages = [
> +            'java-1.8.0-openjdk-devel',

This will actively break on centos 6, but I guess it's actually better than the alternative of not having the required java stuff installed at all by bootstrap, which I guess is what happens today.

::: python/mozboot/mozboot/centosfedora.py:49
(Diff revision 1)
>              'yasm',
>          ]
>  
> -        self.mobile_android_packages = []
> +        self.mobile_android_packages = [
> +            'java-1.8.0-openjdk-devel',
> +            'ncurses-devel.i686',

I wonder why we needed ncurses-devel.i686 for... can you try without?

::: python/mozboot/mozboot/centosfedora.py:50
(Diff revision 1)
>          ]
>  
> -        self.mobile_android_packages = []
> +        self.mobile_android_packages = [
> +            'java-1.8.0-openjdk-devel',
> +            'ncurses-devel.i686',
> +            'libstdc++.i686',

I think we needed this because of the "host" tools in the android SDK/NDK being 32-bits. I think they're 64-bits nowadays... which means this shouldn't be required (and one more reason why ncurses-devel should be useless)

::: python/mozboot/mozboot/centosfedora.py:51
(Diff revision 1)
>  
> -        self.mobile_android_packages = []
> +        self.mobile_android_packages = [
> +            'java-1.8.0-openjdk-devel',
> +            'ncurses-devel.i686',
> +            'libstdc++.i686',
> +            'zlib-devel.i686',

This shouldn't be required anymore.
Attachment #8883767 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8883768 [details]
Bug 1344244 - Part 1: Accept newer Android lint version.

https://reviewboard.mozilla.org/r/154706/#review161090
Attachment #8883768 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8883769 [details]
Bug 1344244 - Part 2: Merge similar code into android.py.

https://reviewboard.mozilla.org/r/154708/#review161092

::: python/mozboot/mozboot/archlinux.py:70
(Diff revision 1)
>          'https://aur.archlinux.org/cgit/aur.git/snapshot/uuid.tar.gz',
>      ]
>  
>      MOBILE_ANDROID_COMMON_PACKAGES = [
> -        'zlib',  # mobile/android requires system zlib.
> -        'jdk7-openjdk', # It would be nice to handle alternative JDKs.  See https://wiki.archlinux.org/index.php/Java.
> +        # mobile/android requires system zlib.
> +        'zlib',

That's not a requirement anymore.

::: python/mozboot/mozboot/osx.py:419
(Diff revision 1)
>          ]
>          self._ensure_macports_packages(packages)
>  
>          # Verify the presence of java and javac.
>          if not self.which('java') or not self.which('javac'):
> -            raise Exception('You need to have Java version 1.7 or later installed. Please visit http://www.java.com/en/download/mac_download.jsp to get the latest version.')
> +            raise Exception('You need to have Java version 1.7 or later installed. ' +

You don't really need the +.
Attachment #8883769 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8883770 [details]
Bug 1344244 - Part 3: Use sdkmanager rather than android.

https://reviewboard.mozilla.org/r/154710/#review161098

::: python/mozboot/mozboot/android.py:45
(Diff revision 1)
>  '''
>  
> -NOT_INSTALLING_ANDROID_PACKAGES = '''
> -It looks like you already have the following Android packages:
> +ANDROID_SDK_TOO_OLD = '''
> +Looks like you have an outdated Android SDK installed at:
>  %s
> -No need to update!
> +I can't update outdated Android SDKs to have the required 'sdkmanger'

typo sdkmanager

::: python/mozboot/mozboot/android.py:226
(Diff revision 1)
> -        print(NOT_INSTALLING_ANDROID_PACKAGES % ', '.join(packages))
> -        return
> -
>      # This tries to install all the required Android packages.  The user
>      # may be prompted to agree to the Android license.
> -    print(INSTALLING_ANDROID_PACKAGES % ', '.join(missing))
> +    package_file_name = os.path.join(os.path.dirname(__file__), 'android-packages.txt')

The android-packages.txt file is missing.
Attachment #8883770 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8883771 [details]
Bug 1344244 - Part 4: Homebrew includes |brew cask| directly now.

https://reviewboard.mozilla.org/r/154712/#review161100
Attachment #8883771 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8883772 [details]
Bug 1344244 - Post: Allow --application-choice to |mach bootstrap|.

https://reviewboard.mozilla.org/r/154714/#review161102

::: python/mozboot/bin/bootstrap.py:144
(Diff revision 1)
>                        default='hgweb',
>                        help='The type of the repository. This defines how we fetch file '
>                        'content. Like --repo, you should not need to set this.')
>  
>      parser.add_option('--application-choice', dest='application_choice',
> -                      help='Pass in an application choice (desktop/android) instead of using the '
> +                      help='Pass in an application choice (see bootstrap.APPLICATIONS) instead of using the '

Can't we just import mozboot.bootstrap and feed the optionparser with the choices? If not, please add `mozboot.` prefix to the variable name in the help message, because there are two bootstrap.py files in the tree (three if you count servo's)
Attachment #8883772 - Flags: review?(mh+mozilla)
Comment on attachment 8883769 [details]
Bug 1344244 - Part 2: Merge similar code into android.py.

https://reviewboard.mozilla.org/r/154708/#review161842

::: python/mozboot/mozboot/archlinux.py:70
(Diff revision 1)
>          'https://aur.archlinux.org/cgit/aur.git/snapshot/uuid.tar.gz',
>      ]
>  
>      MOBILE_ANDROID_COMMON_PACKAGES = [
> -        'zlib',  # mobile/android requires system zlib.
> -        'jdk7-openjdk', # It would be nice to handle alternative JDKs.  See https://wiki.archlinux.org/index.php/Java.
> +        # mobile/android requires system zlib.
> +        'zlib',

OK, I've removed zlib, here and elsewhere.

::: python/mozboot/mozboot/osx.py:419
(Diff revision 1)
>          ]
>          self._ensure_macports_packages(packages)
>  
>          # Verify the presence of java and javac.
>          if not self.which('java') or not self.which('javac'):
> -            raise Exception('You need to have Java version 1.7 or later installed. Please visit http://www.java.com/en/download/mac_download.jsp to get the latest version.')
> +            raise Exception('You need to have Java version 1.7 or later installed. ' +

Dropped.
Comment on attachment 8883770 [details]
Bug 1344244 - Part 3: Use sdkmanager rather than android.

https://reviewboard.mozilla.org/r/154710/#review161846
Comment on attachment 8883772 [details]
Bug 1344244 - Post: Allow --application-choice to |mach bootstrap|.

https://reviewboard.mozilla.org/r/154714/#review161102

> Can't we just import mozboot.bootstrap and feed the optionparser with the choices? If not, please add `mozboot.` prefix to the variable name in the help message, because there are two bootstrap.py files in the tree (three if you count servo's)

I believe we *cannot* in this environment.  `bin/bootstrap.py` is usually cURLed down, and then run; it downloads the `mozboot` package afterwards.

I think we *can* in `mach_commands.py`, but I'd prefer to keep them uniform.

I've added `moboot.` prefixes in both places.
Comment on attachment 8883772 [details]
Bug 1344244 - Post: Allow --application-choice to |mach bootstrap|.

https://reviewboard.mozilla.org/r/154714/#review163196
Attachment #8883772 - Flags: review?(mh+mozilla) → review+
Comment on attachment 8883767 [details]
Bug 1344244 - Pre: Support mobile/android in bootstrap on CentOS.

https://reviewboard.mozilla.org/r/154704/#review161832

::: python/mozboot/mozboot/centosfedora.py:25
(Diff revision 1)
>          self.group_packages = []
>  
>          self.packages = [
>              'autoconf213',
>              'mercurial',
> +            'which',

We do: http://searchfox.org/mozilla-central/source/mach#9.  (Or at least, I think this is why I needed which.)

::: python/mozboot/mozboot/centosfedora.py:48
(Diff revision 1)
>              'wireless-tools-devel',
>              'yasm',
>          ]
>  
> -        self.mobile_android_packages = []
> +        self.mobile_android_packages = [
> +            'java-1.8.0-openjdk-devel',

Newer build tools have newer requirements: new Proguard requires Java 1.8 and new aapt requires glibc 2.14.  (I can't recall if Java 1.8 requires glibc 2.14; probably not.)  So CentOS 6 support won't be feasiable as soon as we use newer build tools, which we want yesterday.

::: python/mozboot/mozboot/centosfedora.py:49
(Diff revision 1)
>              'yasm',
>          ]
>  
> -        self.mobile_android_packages = []
> +        self.mobile_android_packages = [
> +            'java-1.8.0-openjdk-devel',
> +            'ncurses-devel.i686',

I'll just remove these.  Somebody else (possibly future me!) can patch this up.
Comment on attachment 8883767 [details]
Bug 1344244 - Pre: Support mobile/android in bootstrap on CentOS.

https://reviewboard.mozilla.org/r/154704/#review161088

> I think we needed this because of the "host" tools in the android SDK/NDK being 32-bits. I think they're 64-bits nowadays... which means this shouldn't be required (and one more reason why ncurses-devel should be useless)

I'm going to leave this to follow-up.
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s ff5acbafdb9d -d 18aeda1e84fb: rebasing 407597:ff5acbafdb9d "Bug 1344244 - Pre: Support mobile/android in bootstrap on CentOS. r=glandium"
rebasing 407598:e4b3750b3712 "Bug 1344244 - Part 1: Accept newer Android lint version. r=glandium"
merging build/autoconf/android.m4
merging old-configure.in
warning: conflicts while merging build/autoconf/android.m4! (edit, then use 'hg resolve --mark')
warning: conflicts while merging old-configure.in! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Assignee: nobody → nalexander
Status: NEW → ASSIGNED
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1515fd3c7003
Pre: Support mobile/android in bootstrap on CentOS. r=glandium
https://hg.mozilla.org/integration/autoland/rev/7c127a4f904a
Part 1: Accept newer Android lint version. r=glandium
https://hg.mozilla.org/integration/autoland/rev/01b0a01a38b1
Part 2: Merge similar code into android.py. r=glandium
https://hg.mozilla.org/integration/autoland/rev/0bae2cd79169
Part 3: Use sdkmanager rather than android. r=glandium
https://hg.mozilla.org/integration/autoland/rev/4f056c7650c7
Part 4: Homebrew includes |brew cask| directly now. r=glandium
https://hg.mozilla.org/integration/autoland/rev/0ee162368b58
Post: Allow --application-choice to |mach bootstrap|. r=glandium
Blocks: 1384199
Blocks: 1384231
Depends on: 1376306
Blocks: 1407872
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: