1650206 - Incompatible Java versions on path cause JAVA_HOME to not be checked

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

When bootstrapping for an Android build Java is searched for on the path before checking JAVA_HOME. This means that if the system has an incompatible version of Java on the path the search will fail early when it finds this version and will never check JAVA_HOME. This can be unintuitive, as the error message in this case suggests checking JAVA_HOME, despite that it is not the cause of the error. Example below

:~/projects/mozilla/mozilla-unified$ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ MOZCONFIG=.mozconfig-android-i686-opt ./mach bootstrap

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
Your choice: 4
Executing as root: sudo apt-get install autoconf2.13 build-essential nodejs python-setuptools unzip uuid zip python-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.4ubuntu1).
python-dev is already the newest version (2.7.15~rc1-1).
python-setuptools is already the newest version (39.0.1-2).
unzip is already the newest version (6.0-21ubuntu1).
zip is already the newest version (3.0-11build1).
autoconf2.13 is already the newest version (2.13-68).
uuid is already the newest version (1.6.2-1.5build4).
nodejs is already the newest version (8.10.0~dfsg-2ubuntu0.4).
The following packages were automatically installed and are no longer required:
  libclang-common-7-dev libclang1-7 libdumbnet1 libllvm7 linux-headers-4.15.0-46 linux-headers-4.15.0-46-generic linux-headers-4.15.0-55 linux-headers-4.15.0-55-generic linux-image-4.15.0-46-generic
  linux-image-4.15.0-55-generic linux-modules-4.15.0-46-generic linux-modules-4.15.0-55-generic linux-modules-extra-4.15.0-46-generic linux-modules-extra-4.15.0-55-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 129 not upgraded.
Executing as root: sudo apt-get install openjdk-8-jdk-headless wget
Reading package lists... Done
Building dependency tree       
Reading state information... Done
wget is already the newest version (1.19.4-1ubuntu2.2).
openjdk-8-jdk-headless is already the newest version (8u252-b09-1~18.04).
The following packages were automatically installed and are no longer required:
  libclang-common-7-dev libclang1-7 libdumbnet1 libllvm7 linux-headers-4.15.0-46 linux-headers-4.15.0-46-generic linux-headers-4.15.0-55 linux-headers-4.15.0-55-generic linux-image-4.15.0-46-generic
  linux-image-4.15.0-55-generic linux-modules-4.15.0-46-generic linux-modules-4.15.0-55-generic linux-modules-extra-4.15.0-46-generic linux-modules-extra-4.15.0-55-generic
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 129 not upgraded.
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: You need to have Java Development Kit version 1.8 installed (found /usr/bin/java but could not parse version "Property settings:
    awt.toolkit = sun.awt.X11.XToolkit
    file.encoding = UTF-8
    file.separator = /
    java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
    java.awt.printerjob = sun.print.PSPrinterJob
    java.class.path = 
    java.class.version = 55.0
    java.home = /usr/lib/jvm/java-11-openjdk-amd64
    java.io.tmpdir = /tmp
    java.library.path = /usr/java/packages/lib
        /usr/lib/x86_64-linux-gnu/jni
        /lib/x86_64-linux-gnu
        /usr/lib/x86_64-linux-gnu
        /usr/lib/jni
        /lib
        /usr/lib
    java.runtime.name = OpenJDK Runtime Environment
    java.runtime.version = 11.0.7+10-post-Ubuntu-2ubuntu218.04
    java.specification.name = Java Platform API Specification
    java.specification.vendor = Oracle Corporation
    java.specification.version = 11
    java.vendor = Ubuntu
    java.vendor.url = https://ubuntu.com/
    java.vendor.url.bug = https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
    java.version = 11.0.7
    java.version.date = 2020-04-14
    java.vm.compressedOopsMode = Zero based
    java.vm.info = mixed mode, sharing
    java.vm.name = OpenJDK 64-Bit Server VM
    java.vm.specification.name = Java Virtual Machine Specification
    java.vm.specification.vendor = Oracle Corporation
    java.vm.specification.version = 11
    java.vm.vendor = Ubuntu
    java.vm.version = 11.0.7+10-post-Ubuntu-2ubuntu218.04
    jdk.debug = release
    line.separator = \n 
    os.arch = amd64
    os.name = Linux
    os.version = 4.15.0-99-generic
    path.separator = :
    sun.arch.data.model = 64
    sun.boot.library.path = /usr/lib/jvm/java-11-openjdk-amd64/lib
    sun.cpu.endian = little
    sun.cpu.isalist = 
    sun.desktop = gnome
    sun.io.unicode.encoding = UnicodeLittle
    sun.java.launcher = SUN_STANDARD
    sun.jnu.encoding = UTF-8
    sun.management.compiler = HotSpot 64-Bit Tiered Compilers
    sun.os.patch.level = unknown
    user.country = US
    user.dir = /home/b/projects/mozilla/mozilla-unified
    user.home = /home/b
    user.language = en
    user.name = b
    user.timezone = 

openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04, mixed mode, sharing)"). Check the JAVA_HOME environment variable. Please install JDK 1.8 from https://adoptopenjdk.net/?variant=openjdk8.

  File "/home/b/projects/mozilla/mozilla-unified/python/mozboot/mozboot/mach_commands.py", line 46, in bootstrap
    bootstrapper.bootstrap()
  File "/home/b/projects/mozilla/mozilla-unified/python/mozboot/mozboot/bootstrap.py", line 486, in bootstrap
    getattr(self.instance, 'install_%s_packages' % application)()
  File "/home/b/projects/mozilla/mozilla-unified/python/mozboot/mozboot/debian.py", line 138, in install_mobile_android_packages
    self.ensure_mobile_android_packages()
  File "/home/b/projects/mozilla/mozilla-unified/python/mozboot/mozboot/debian.py", line 157, in ensure_mobile_android_packages
    self.ensure_java()
  File "/home/b/projects/mozilla/mozilla-unified/python/mozboot/mozboot/base.py", line 873, in ensure_java
    raise unknown_version_exception
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

JAVA_HOME is checked last as it is passed as an extra search dir to which and which checks such dirs after the path.

My expectation was that JAVA_HOME would be checked first to allow for flexibility in choice of Java used. I've locally changed this so JAVA_HOME is checked first to allow me to build while still having my system java on the path. I did this by patching ensure_java to check JAVA_HOME first, as I didn't want to make which check extra dirs first for all calls (I'm not sure if this would break other things).

Would it make sense to search JAVA_HOME before the rest of the path?

Comment 1

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Patch I've used locally below.

diff --git a/python/mozboot/mozboot/base.py b/python/mozboot/mozboot/base.py
--- a/python/mozboot/mozboot/base.py
+++ b/python/mozboot/mozboot/base.py
@@ -836,8 +836,13 @@ class BaseBootstrapper(object):
         "" "
 
         if 'JAVA_HOME' in os.environ:
-            extra_search_dirs += (os.path.join(os.environ['JAVA_HOME'], 'bin'),)
-        java = self.which('java', *extra_search_dirs)
+            #extra_search_dirs += (os.path.join(os.environ['JAVA_HOME'], 'bin'),)
+            possible_java_path = os.path.join(os.environ['JAVA_HOME'], 'bin', 'java')
+            if os.path.isfile(possible_java_path) and os.access(possible_java_path, os.X_OK):
+                java = possible_java_path
+
+        if not java:
+            java = self.which('java', *extra_search_dirs)
 
         if not java:
             raise Exception('You need to have Java Development Kit version 1.8 installed. '

if this a sane thing to do, let me know and I can push it for review (with some tidying).

Comment 2

[Nick Alexander :nalexander [he/him]]

bryce: yes, this seems sensible. JAVA_HOME is "finer grained" than PATH, so we may as well search for java there first. Post that patch and r?#build (or me). Thanks!

Comment 3

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Attachment: Bug 1650206 - If set, check JAVA_HOME for java instead of path. r?nalexander

Prior to this patch JAVA_HOME was appended to path and then the first instance
of java on those paths was used and checked for version compatibility. This
meant that if an incompatible version on java is on the path JAVA_HOME could not
be used to point to a different, compatible version.

Following this patch, JAVA_HOME can be used as a more fine grained selector of
java. Only if JAVA_HOME is not set will we check the path.

Comment 4

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9c6faae80158b6cb617c7b44b62d24e3799388c1

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/014541f7b152
If set, check JAVA_HOME for java instead of path. r=nalexander

Comment 6

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/014541f7b152