"./mach bootstrap" with Firefox for Android fails with java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema

RESOLVED FIXED in Firefox 66

Status

defect
RESOLVED FIXED
10 months ago
2 months ago

People

(Reporter: ato, Assigned: nalexander)

Tracking

(Blocks 2 bugs)

Trunk
mozilla66
Dependency tree / graph

Firefox Tracking Flags

(firefox66 fixed)

Details

Attachments

(6 attachments)

Reporter

Description

10 months ago
Running "./mach bootstrap" and selecting the fourth option, "Firefox
for Android", fails with a java.lang.NoClassDefFoundError:
javax/xml/bind/annotation/XmlSchema exception:

> % ./mach bootstrap
> 
> Note on Artifact Mode:
> 
> Artifact builds download prebuilt C++ components rather than building
> them locally.
> 
> Artifact builds are recommended for people working on Firefox or
> Firefox for Android frontends. They are unsuitable for those working
> on C++ code. For more information see:
> https://developer.mozilla.org/en-US/docs/Artifact_builds.
> 
> Please choose the version of Firefox you want to build:
> 1. Firefox for Desktop Artifact Mode
> 2. Firefox for Desktop
> 3. Firefox for Android Artifact Mode
> 4. Firefox for Android
> Your choice: 4
> Executing as root: sudo apt-get install autoconf2.13 build-essential nodejs python-dev python-pip python-setuptools unzip uuid zip
> 
> Reading package lists... 100%
> Reading package lists... Done
> Building dependency tree... 0%
> Building dependency tree... 50%
> Building dependency tree       
> Reading state information... Done
> autoconf2.13 is already the newest version (2.13-68).
> build-essential is already the newest version (12.5).
> uuid is already the newest version (1.6.2-1.5+b6).
> python-dev is already the newest version (2.7.15-3).
> python-pip is already the newest version (9.0.1-2.3).
> python-setuptools is already the newest version (39.2.0-1).
> unzip is already the newest version (6.0-21).
> zip is already the newest version (3.0-11+b1).
> nodejs is already the newest version (8.11.3-1nodesource1).
> 0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded.
> Executing as root: sudo dpkg --add-architecture i386
> Executing as root: sudo apt-get update
> 
> Hit:1 http://security.debian.org/debian-security buster/updates InRelease
>  buster/updates InRelease
> 0% [1 InRelease gpgv 38.3 kB] [Connecting to ftp.de.debian.org (141.76.2.4)]
> Hit:2 http://ftp.de.debian.org/debian buster InRelease
> 
> 0% [1 InRelease gpgv 38.3 kB]
> Hit:3 https://deb.nodesource.com/node_8.x buster InRelease
> 
> 0% [1 InRelease gpgv 38.3 kB]
> 0% [Working]
> 0% [2 InRelease gpgv 150 kB]
> 0% [Working]
> 0% [3 InRelease gpgv 4,643 B]
>              
> Reading package lists... 0%
> Reading package lists... 0%
> Reading package lists... 0%
> Reading package lists... 37%
> Reading package lists... 74%
> Reading package lists... 83%
> Reading package lists... 98%
> Reading package lists... 98%
> Reading package lists... 98%
> Reading package lists... Done
> Executing as root: sudo apt-get install default-jdk wget libncurses5:i386 libstdc++6:i386
> 
> Reading package lists... 100%
> Reading package lists... Done
> Building dependency tree... 0%
> Building dependency tree... 50%
> Building dependency tree       
> Reading state information... Done
> default-jdk is already the newest version (2:1.10-67).
> wget is already the newest version (1.19.5-1).
> libstdc++6:i386 is already the newest version (8.1.0-10).
> libncurses5:i386 is already the newest version (6.1+20180210-4).
> 0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded.
> 
> Looks like you have the Android NDK installed at:
> /home/ato/.mozbuild/android-ndk-r15c
> 
> 
> Looks like you have the Android SDK installed at:
> /home/ato/.mozbuild/android-sdk-linux
> We will install all required Android packages.
> 
> 
> We are now installing the following Android packages:
> platform-tools
> build-tools;26.0.2
> platforms;android-26
> extras;android;m2repository
> extras;google;m2repository
> emulator
> 
> You may be prompted to agree to the Android license. You may see some of
> output as packages are downloaded and installed.
> 
> Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema
> 	at com.android.repository.api.SchemaModule$SchemaModuleVersion.<init>(SchemaModule.java:156)
> 	at com.android.repository.api.SchemaModule.<init>(SchemaModule.java:75)
> 	at com.android.sdklib.repository.AndroidSdkHandler.<clinit>(AndroidSdkHandler.java:81)
> 	at com.android.sdklib.tool.SdkManagerCli.main(SdkManagerCli.java:117)
> 	at com.android.sdklib.tool.SdkManagerCli.main(SdkManagerCli.java:93)
> Caused by: java.lang.ClassNotFoundException: javax.xml.bind.annotation.XmlSchema
> 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
> 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190)
> 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499)
> 	... 5 more
> Error running mach:
> 
>     ['bootstrap']
> 
> The error occurred in code that was called by the mach command. This is either
> a bug in the called code itself or in the way that mach is calling it.
> 
> You should consider filing a bug for this issue.
> 
> If filing a bug, please include the full output of mach, including this error
> message.
> 
> The details of the failure are as follows:
> 
> CalledProcessError: Command '['/home/ato/.mozbuild/android-sdk-linux/tools/bin/sdkmanager', '--package_file=/home/ato/src/gecko/python/mozboot/mozboot/android-packages.txt']' returned non-zero exit status 1
> 
>   File "/home/ato/src/gecko/python/mozboot/mozboot/mach_commands.py", line 32, in bootstrap
>     bootstrapper.bootstrap()
>   File "/home/ato/src/gecko/python/mozboot/mozboot/bootstrap.py", line 281, in bootstrap
>     getattr(self.instance, 'install_%s_packages' % application)()
>   File "/home/ato/src/gecko/python/mozboot/mozboot/debian.py", line 127, in install_mobile_android_packages
>     self.ensure_mobile_android_packages()
>   File "/home/ato/src/gecko/python/mozboot/mozboot/debian.py", line 154, in ensure_mobile_android_packages
>     no_interactive=self.no_interactive)
>   File "/home/ato/src/gecko/python/mozboot/mozboot/android.py", line 193, in ensure_android
>     ensure_android_packages(sdkmanager_tool=sdkmanager_tool, no_interactive=no_interactive)
>   File "/home/ato/src/gecko/python/mozboot/mozboot/android.py", line 249, in ensure_android_packages
>     subprocess.check_call(args)
>   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
>     raise CalledProcessError(retcode, cmd)
Reporter

Comment 1

10 months ago
Not sure what to check for here.  Do you have any clues?

I forgot to mention in the bug that the target system is Debian testing.
Flags: needinfo?(nalexander)

Comment 2

10 months ago
I'm hitting the same problem in macos now.
(In reply to Andreas Tolfsen 「:ato」 from comment #1)
> Not sure what to check for here.  Do you have any clues?
> 
> I forgot to mention in the bug that the target system is Debian testing.

My guess is that we're not getting the right version of the JDK, although it would be odd to get it on both Linux and macOS.

ato, Pike: can you tell me what java version is being invoked?  (java -version, javac -version)  I'm pretty sure we require 1.8, and we might need to be more scrupulous about _checking_ the version at bootstrap time, since OS vendors seem to be moving it forward on us frequently.
Flags: needinfo?(nalexander)
Flags: needinfo?(l10n)
Flags: needinfo?(ato)

Updated

10 months ago
Component: Mach Core → Bootstrap Configuration

Comment 4

10 months ago
Fuchsia:src axelhecht$ which java 
/usr/bin/java
Fuchsia:src axelhecht$ java --version
java 9.0.4
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)
Fuchsia:src axelhecht$ javac --version
javac 9.0.4


and then there's 

Fuchsia:src axelhecht$ ls -l /Library/Java/
total 0
drwxr-xr-x  2 root  wheel   64 Jul 15  2017 Extensions
lrwxr-xr-x  1 root  wheel   64 Jul 23 11:12 Home -> /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home
drwxr-xr-x  5 root  wheel  160 Jul 23 11:12 JavaVirtualMachines
lrwxr-xr-x  1 root  wheel   62 Feb 12 20:47 MacOS -> /Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/MacOS


Fuchsia:src axelhecht$ /Library/Java/Home/bin/java --version
Unrecognized option: --version
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

but


Fuchsia:src axelhecht$ /Library/Java/Home/bin/java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Updated

10 months ago
Flags: needinfo?(l10n)
Reporter

Comment 5

10 months ago
> % java --version
> openjdk 10.0.2 2018-07-17
> OpenJDK Runtime Environment (build 10.0.2+13-Debian-1)
> OpenJDK 64-Bit Server VM (build 10.0.2+13-Debian-1, mixed mode)

> % javac --version
> javac 10.0.2

# update-java-alternatives -l
> java-1.10.0-openjdk-amd64      1101       /usr/lib/jvm/java-1.10.0-openjdk-amd64

I thought this initially had to do with multuple Java installations
on the system, but apparently not.
Flags: needinfo?(ato)
sudo update-java-alternatives --jre-headless --jre --set java-1.8.0-openjdk-amd64

helped me get past that point. We really ought to have some check for whether the correct jre is installed/active because it took me a long time to figure out what was going on here.
Duplicate of this bug: 1507601
These are downloaded from maven.google.com by Gradle now.
Just hygiene, no reason to lag, no reason to expect changes.

Depends on D16134
This is part of the larger shift toward GeckoView and away from
Firefox for Android.  We need GV to "show up" in these kind of
onboarding tools; here's a small first step.

Depends on D16135
What is happening is that distribution JRE and JDK packages roll
forward, installing a different version of Java than what is expected.
We don't check the version installed, so sadness ensues.

Right now, we require Java 1.8 to build, but in the near future, after
Android-Gradle plugin 3.2.1+, we'll be free to use later Java
versions.

However, Android's `sdkmanager` itself requires exactly Java 1.8.  We
only require `sdkmanager` to install `emulator`, really -- everything
else will be fetched by Gradle -- but I don't want to unravel that
right now.

So let's just provide decent error messages and try to prevent the
worst of the footguns.

Depends on D16136
We're seeing all sorts of problems with the NDK toolchain compilers
and the various versions of the NDK.  In automation we build with r17b
and our own toolchain clang; let's try to standardize on that.

This patch is ugly because of the evolution of mozboot.  Long ago, we
passed arguments around and interpolated strings into function names
rather than setting members with the application name and whether
we're in artifact mode.  The places I needed to modify didn't have the
right data at the right time so I added it to the bootstrap instance.
I don't have the time or energy to use the instance variables
through-out: that'll have to be future follow-up.

Depends on D16137

paul: on your macOS device, could you:

  • apply the patch stack from phab
  • invoke ./mach bootstrap --no-interactive --application-choice mobile_android
  • update your mozconfig as suggested, choosing a --target you care about
  • invoke ./mach build && ./mach package

and report back to me? I expect that these changes, which make using a Mozilla-produced clang that "supported configuration", will address your segfaults -- but I'd definitely like confirmation. Thank you!

Flags: needinfo?(paul)
Blocks: 1519180

It works! Thank you.

Flags: needinfo?(paul)

It worked for me also but ony when targeting arm-linux-androideabi

Otherwise when targeting
x86_64-linux-android I get

0:09.40 checking whether the C compiler (/Users/petru.lingurar/.mozbuild/clang/bin/clang -std=gnu99 --target=arm-linux-androideabi -fno-short-enums -fno-exceptions -L/Users/petru.lingurar/.mozbuild/android-ndk-r17b/platforms/android-21/arch-x86_64/usr/lib -Wl,-rpath-link=/Users/petru.lingurar/.mozbuild/android-ndk-r17b/platforms/android-21/arch-x86_64/usr/lib --sysroot=/Users/petru.lingurar/.mozbuild/android-ndk-r17b/platforms/android-21/arch-x86_64 -Wl,--allow-shlib-undefined -gcc-toolchain /Users/petru.lingurar/.mozbuild/android-ndk-r17b/toolchains/x86_64-4.9/prebuilt/darwin-x86_64 -fuse-ld=lld) works... no
0:09.40 configure: error: installation or configuration problem: C compiler cannot create executables.
0:09.40 DEBUG: <truncated - see config.log for full output>
0:09.40 DEBUG: configure:1360: /Users/petru.lingurar/.mozbuild/clang/bin/clang -std=gnu99 --target=arm-linux-androideabi -o conftest -fno-short-enums -fno-exceptions -isystem /Users/petru.lingurar/.mozbuild/android-ndk-r17b/sysroot/usr/include/x86_64-linux-android -isystem /Users/petru.lingurar/.mozbuild/android-ndk-r17b/sysroot/usr/include -gcc-toolchain /Users/petru.lingurar/.mozbuild/android-ndk-r17b/toolchains/x86_64-4.9/prebuilt/darwin-x86_64 -D__ANDROID_API__=21 -L/Users/petru.lingurar/.mozbuild/android-ndk-r17b/platforms/android-21/arch-x86_64/usr/lib -Wl,-rpath-link=/Users/petru.lingurar/.mozbuild/android-ndk-r17b/platforms/android-21/arch-x86_64/usr/lib --sysroot=/Users/petru.lingurar/.mozbuild/android-ndk-r17b/platforms/android-21/arch-x86_64 -Wl,--allow-shlib-undefined -gcc-toolchain /Users/petru.lingurar/.mozbuild/android-ndk-r17b/toolchains/x86_64-4.9/prebuilt/darwin-x86_64 -fuse-ld=lld conftest.c 1>&5
0:09.40 DEBUG: configure:1357:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
0:09.40 DEBUG: main(){return(0);}
0:09.40 DEBUG: ^
0:09.40 DEBUG: 1 warning generated.
0:09.40 DEBUG: ld.lld: error: cannot open crtbegin_dynamic.o: No such file or directory
0:09.40 DEBUG: ld.lld: error: unable to find library -lgcc
0:09.40 DEBUG: ld.lld: error: unable to find library -ldl
0:09.40 DEBUG: ld.lld: error: unable to find library -lc
0:09.40 DEBUG: ld.lld: error: unable to find library -lgcc
0:09.40 DEBUG: ld.lld: error: unable to find library -ldl
0:09.40 DEBUG: ld.lld: error: cannot open crtend_android.o: No such file or directory
0:09.40 DEBUG: clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
0:09.40 DEBUG: configure: failed program was:
0:09.40 DEBUG:
0:09.40 DEBUG: #line 1355 "configure"
0:09.40 DEBUG: #include "confdefs.h"
0:09.40 DEBUG:
0:09.40 DEBUG: main(){return(0);}
0:09.41 DEBUG: configure: error: installation or configuration problem: C compiler cannot create executables.
0:09.41 ERROR: old-configure failed
0:09.44 *** Fix above errors and then restart with "./mach build"
0:09.44 make: *** [configure] Error 1

i686-linux-android or
aarch64-linux-android
gives mainly the same error but from ld.lld just

0:09.91 DEBUG: ld.lld: error: unable to find library -lgcc
0:09.91 DEBUG: ld.lld: error: unable to find library -lgcc

Also when trying a robocop test I see

▶ ./mach robocop --serve
Error running mach:

['robocop', '--serve']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ADBTimeoutError: args: /Users/petru.lingurar/.mozbuild/android-sdk-macosx/platform-tools/adb -s 2259f4e8480d7ece wait-for-device shell su 0 id; echo adb_returncode=$?, exitcode: None, stdout:

File "/Users/petru.lingurar/Projects/Clients/mozilla-unified/testing/mochitest/mach_commands.py", line 572, in run_robocop
device_serial=device_serial)
File "/Users/petru.lingurar/Projects/Clients/mozilla-unified/testing/mozbase/mozrunner/mozrunner/devices/android_device.py", line 237, in verify_android_device
device = _get_device(build_obj.substs, device_serial)
File "/Users/petru.lingurar/Projects/Clients/mozilla-unified/testing/mozbase/mozrunner/mozrunner/devices/android_device.py", line 101, in _get_device
device = ADBDevice(adb=adb_path, verbose=verbose_logging, device=device_serial)
File "/Users/petru.lingurar/Projects/Clients/mozilla-unified/testing/mozbase/mozdevice/mozdevice/adb.py", line 642, in init
self.shell_output("su 0 id", timeout=timeout).find(uid) != -1):
File "/Users/petru.lingurar/Projects/Clients/mozilla-unified/testing/mozbase/mozdevice/mozdevice/adb.py", line 1443, in shell_output
raise ADBTimeoutError("%s" % adb_process)

Don't really know if it's because of this patch or not. Tested on a real device. com.android.shell was granted su.
For the ADBTimeout I see there's already a ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=1510059

Petru: the ADB issue is almost certainly unrelated.

The logs you post look very confused -- you say you're targeting x86_64-linux-android but the invocations look like /Users/petru.lingurar/.mozbuild/clang/bin/clang -std=gnu99 --target=arm-linux-androideabi ... (i.e., regular ARM).

Can you ensure you have exactly one --target line in your mozconfig, and then mach clobber before trying again?

Thanks!

Flags: needinfo?(petru.lingurar)
Assignee: nobody → nalexander
Status: NEW → ASSIGNED

Thanks Nick.

Indeed i686-linux-android worked now and the robocop tests also (on the emulator).
I deleted other options I had in my mozconfig, and kept only the output from bootstrap so maybe that helped.

Flags: needinfo?(petru.lingurar)

Comment 19

4 months ago
Pushed by ebalazs@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/b7b1e008be45
Pre: Don't require outdated packages. r=agi
https://hg.mozilla.org/mozilla-central/rev/9c33fda84724
Pre: Bump Android SDK version. r=agi
https://hg.mozilla.org/mozilla-central/rev/8b70ac3ef8d0
Part 0: Make GeckoView more prominent in |mach bootstrap|. r=agi
https://hg.mozilla.org/mozilla-central/rev/b1f74fc90f37
Part 1: Look for java and pin to Java 1.8 in |mach bootstrap|; r=agi
https://hg.mozilla.org/mozilla-central/rev/bbfe0cbb90b4
Part 2: Make |mach bootstrap| for Android recommend toolchain clang and lld. r=agi,froydnj,#firefox-build-system-reviewers
https://hg.mozilla.org/mozilla-central/rev/dc41aecedaac
Part 3: Make |mach bootstrap| for Android install Rust toolchains. r=agi
Blocks: 1503618
Duplicate of this bug: 1438167
Duplicate of this bug: 1418900
Duplicate of this bug: 1416383
You need to log in before you can comment on or make changes to this bug.