1795479 - 2.53.14 release build failure- regression

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect)

Description

[rezso]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0

Steps to reproduce:

I tried to compile 2.53.14 on same system and same configuration as 2.53.13 succesfully built.

Actual results:

• cd /var/uhubuild/work/object
• /var/uhubuild/work/compile/configure
  Traceback (most recent call last):
  File "/var/uhubuild/work/compile/configure.py", line 167, in <module>
  sys.exit(main(sys.argv))
  File "/var/uhubuild/work/compile/configure.py", line 40, in main
  sandbox.run(os.path.join(os.path.dirname(file), 'moz.configure'))
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 446, in run
  self.include_file(path)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 437, in include_file
  exec_(code, self)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 56, in exec_
  exec(object, globals, locals)
  File "/var/uhubuild/work/compile/moz.configure", line 7, in <module>
  include('build/moz.configure/init.configure')
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 746, in include_impl
  self.include_file(what)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 437, in include_file
  exec_(code, self)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 56, in exec_
  exec(object, globals, locals)
  File "/var/uhubuild/work/compile/build/moz.configure/init.configure", line 210, in <module>
  @imports(_from='mozbuild.virtualenv', _import='PY2')
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 729, in decorator
  depends = DependsFunction(self, func, dependencies, when=when)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 134, in init
  sandbox._value_for(self)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 521, in _value_for
  return self._value_for_depends(obj)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 981, in method_call
  cache[args] = self.func(instance, *args)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 530, in _value_for_depends
  value = obj.result()
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 981, in method_call
  cache[args] = self.func(instance, *args)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 160, in result
  return self._func(*resolved_args)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 1121, in wrapped
  return new_func(*args, **kwargs)
  File "/var/uhubuild/work/compile/build/moz.configure/init.configure", line 265, in virtualenv_python2
  found_python = find_program(python)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 1120, in wrapped
  self._apply_imports(func, glob)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 853, in _apply_imports
  glob[what] = self._get_one_import('%s%s' % (from, what))
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 931, in get_one_import
  exec(import_line, {}, glob)
  File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 56, in exec
  exec(object, globals, locals)
  File "<string>", line 1, in <module>
  ImportError: No module named mozfile

Expected results:

• cd /var/uhubuild/work/object
• /var/uhubuild/work/compile/configure
  Creating Python environment
  New python executable in /var/uhubuild/work/object/_virtualenvs/init/bin/python2
  Also creating executable in /var/uhubuild/work/object/_virtualenvs/init/bin/python
  Installing setuptools, pip, wheel...done.
  Error processing command. Ignoring because optional. (optional:setup.py:third_party/python/psutil:build_ext:--inplace)
  Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
  Reexecuting in the virtualenv
  Adding configure options from /var/uhubuild/work/compile/.mozconfig

Comment 1

[Frank-Rainer Grahl (:frg)]

Should be fixed in 2.53.15b1 pre already. Was a missing backport only affecting some systems.
Try this patch: https://www.wg9s.com/comm-253/patches/seamonkey-253-patches/mozilla-release/patches/1610774-74a1.patch

Comment 2

[rezso]

init.configure in 2.53.14 release doesn't contains " sys.path.append(os.path.join( topsrcdir, 'testing', 'mozbase', 'mozfile')) ".
Replacing existing ", os.path.join(topsrcdir, 'third_party', 'python', 'backports'))" line with 0, os.path.join(topsrcdir, 'testing', 'mozbase', 'mozfile')) causes build failure: "ImportError: No module named shutil_which"
I added "0, os.path.join(topsrcdir, 'testing', 'mozbase', 'mozfile')) " after "0, os.path.join(topsrcdir, 'third_party', 'python', 'backports'))":
NameError: global name 'Exception' is not defined
I added "@imports(_from='builtin', _import='Exception')".
0:03.33 File "/var/uhubuild/work/compile/build/moz.configure/init.configure", line 277, in virtualenv_python2
0:03.33 'could not determine version of given PYTHON: %s' % e)
0:03.33 /var/uhubuild/work/compile/build/moz.configure/checks.configure.FatalCheckError: could not determine version of given PYTHON: global name 'python_executable_version' is not defined

My patch currently:

diff -Naur a/build/moz.configure/init.configure b/build/moz.configure/init.configure
--- a/build/moz.configure/init.configure 2022-08-24 21:55:10.000000000 +0200
+++ b/build/moz.configure/init.configure 2022-10-16 23:51:44.270431685 +0200
@@ -200,6 +200,7 @@

@depends('PYTHON', check_build_environment, mozconfig, '--help')
+@imports(_from='builtin', _import='Exception')
@imports('os')
@imports('sys')
@imports('subprocess')
@@ -260,6 +261,8 @@
if normsep(sys.executable) != normsep(manager.python_path):
sys.path.insert(
0, os.path.join(topsrcdir, 'third_party', 'python', 'backports'))

•    sys.path.insert(
  

•        0, os.path.join(topsrcdir, 'testing', 'mozbase', 'mozfile'))
  

  if python:
  found_python = find_program(python)
  @@ -267,11 +270,11 @@
  die('The PYTHON environment variable does not contain '
  'a valid path. Cannot find %s', python)
  python = found_python

•    try:
  

•        version = python_executable_version(python).version
  

•    except Exception as e:
  

•        raise FatalCheckError(
  

•            'could not determine version of given PYTHON: %s' % e)
  

•    # try:
  

•    #    version = python_executable_version(python).version
  

•    # except Exception as e:
  

•    #    raise FatalCheckError(
  

•    #        'could not determine version of given PYTHON: %s' % e)
  

  else:
  python = sys.executable

Comment 3

[Frank-Rainer Grahl (:frg)]

Attachment: 1610774-74a1.patch

try the attached version.

Comment 4

[rezso]

I tried it. The topsrcdir, 'mozfile' path not exists, so the build fails with 'ImportError: No module named mozfile'.
Changing path to topsrcdir, 'testing', 'mozbase', 'mozfile': NameError: global name 'Exception' is not defined

Comment 5

[Myckel Habets]

The "Exception" issue is due to some mismatch between Python 2 and Python 3 in the way it handles this. The way to solve this is implement both versions (and have them called by whatever Python is running the code) or to delete the offending code (This code is called by both Py2 and Py3, both expecting different versions).

This is what we did on Gentoo: https://github.com/BioMike/gentoo-seamonkey-patches/blob/main/seamonkey/2020_fix_exeption_handling.patch
It's a quick and dirty hack, but in no means the best way to solve this.

Comment 6

[Myckel Habets]

@rezso, what does "env | grep PYTHON" (without the quotes) give on your system?

Also, do you have a .mozconfig? Does it set something Python related?

Comment 7

[rezso]

Yes, I have a .mozconfig:

mk_add_options MOZ_OBJDIR=$UB_OBJECTDIR
ac_add_options --prefix=/usr
ac_add_options --enable-application=comm/suite
ac_add_options --enable-release
ac_add_options --enable-official-branding
ac_add_options --enable-optimize="-O2"
ac_add_options --with-system-bz2
ac_add_options --with-system-icu
ac_add_options --with-system-jpeg
ac_add_options --with-system-libvpx
ac_add_options --with-system-nss
ac_add_options --with-system-nspr
ac_add_options --with-system-png
ac_add_options --with-system-zlib
ac_add_options --without-system-libevent
ac_add_options --enable-system-hunspell
ac_add_options --enable-startup-notification
ac_add_options --enable-default-toolkit=cairo-gtk3
ac_add_options --disable-crashreporter
ac_add_options --disable-clang-plugin
ac_add_options --disable-debug
ac_add_options --disable-necko-wifi
ac_add_options --disable-gconf
ac_add_options --disable-updater
ac_add_options --disable-tests
ac_add_options --disable-strip
ac_add_options --disable-install-strip
ac_add_options --disable-elf-hack
ac_add_options --host=x86_64-unknown-linux-gnu
ac_add_options --target=x86_64-unknown-linux-gnu

env | grep 'PYTHON'

PYTHON3=/usr/bin/python3
PYTHON=/usr/bin/python2

because I use
export PYTHON=/usr/bin/python2
export PYTHON3=/usr/bin/python3

Comment 8

[Myckel Habets]

Attachment: PY_build_fix.patch

Even with your environment settings I can't reproduce this issue. Could you give this patch a try?

Comment 9

[rezso]

The init.configure imports in 2.53.14 release and in your patch are different.
Adding @imports(_from='six', _import='PY3') and
if six.PY3:
try:
version = python_executable_version(python).version
except Exception as e:
raise FatalCheckError(
'could not determine version of given PYTHON: %s' % e)

causes an error:
NameError: global name 'six' is not defined

Comment 10

[Myckel Habets]

Ah... I see a mistake I made:

Replace the:

if six.PY3:

to

if PY3:

So, remove the "six." part.

Comment 11

[rezso]

I removed it, and works. Thanks.
So https://phabricator.services.mozilla.com/D60705 and your patch are needed.

Comment 12

[Frank-Rainer Grahl (:frg)]

Attachment: TOP-1795479-py3fix-25315.patch

Patch from Myckel. r/a from me for our 2.53 branch only. Interim fix. Will go away later when we are at python 3 level with the build system.
Thanks

Comment 13

[rezso]

I tried the new release (2.53.15) without previous patches.

The build fails again:

0:01.80 /var/uhubuild/work/compile/configure
0:02.20 Traceback (most recent call last):
0:02.20 File "/var/uhubuild/work/compile/configure.py", line 167, in <module>
0:02.20 sys.exit(main(sys.argv))
0:02.20 File "/var/uhubuild/work/compile/configure.py", line 40, in main
0:02.20 sandbox.run(os.path.join(os.path.dirname(file), 'moz.configure'))
0:02.20 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 446, in run
0:02.20 self.include_file(path)
0:02.20 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 437, in include_file
0:02.20 exec_(code, self)
0:02.20 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 60, in exec_
0:02.20 exec(object, globals, locals)
0:02.20 File "/var/uhubuild/work/compile/moz.configure", line 7, in <module>
0:02.21 include('build/moz.configure/init.configure')
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 746, in include_impl
0:02.21 self.include_file(what)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 437, in include_file
0:02.21 exec_(code, self)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 60, in exec_
0:02.21 exec(object, globals, locals)
0:02.21 File "/var/uhubuild/work/compile/build/moz.configure/init.configure", line 252, in <module>
0:02.21 @imports(_from='six', _import='PY3')
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 729, in decorator
0:02.21 depends = DependsFunction(self, func, dependencies, when=when)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 134, in init
0:02.21 sandbox._value_for(self)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 521, in _value_for
0:02.21 return self._value_for_depends(obj)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 1023, in method_call
0:02.21 cache[args] = self.func(instance, *args)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 530, in _value_for_depends
0:02.21 value = obj.result()
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/util.py", line 1023, in method_call
0:02.21 cache[args] = self.func(instance, *args)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 160, in result
0:02.21 return self._func(*resolved_args)
0:02.21 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/configure/init.py", line 1123, in wrapped
0:02.21 return new_func(*args, **kwargs)
0:02.21 File "/var/uhubuild/work/compile/build/moz.configure/init.configure", line 340, in virtualenv_python2
0:02.21 if version < (2, 7, 0):
0:02.21 UnboundLocalError: local variable 'version' referenced before assignment

It seems to me, that the version detect for python2 is not works.

Comment 14

[Frank-Rainer Grahl (:frg)]

Looking at the code can't say why it fails. 2.53.16 switches to python 3 for this part so hopefully better. Unable to reproduce with CentOS 7 and Rocky 8 Linux.

Comment 15

[Myckel Habets]

Yep, encountered it myself. I thought I send a fix for this, guess that slipped my mind. I'll have a look at it and prepare a patch for it.

Comment 16

[Myckel Habets]

Attachment: 2007_seamonkey-2.53.15_py_version_fix.patch

This patch fixes it in my case.

Comment 17

[rezso]

I tried to compile 2.53.16b1 with python 3.10.9, but fails.

0:56.70 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/action/process_define_files.py", line 66, in process_define_file
0:56.70 for name, val in config.defines['ALLDEFINES'].iteritems()))
0:56.70 AttributeError: 'dict' object has no attribute 'iteritems'

I changed 'iteritems' with 'items'.

The next error:

1:16.24 File "/var/uhubuild/work/compile/config/nsinstall.py", line 185, in <listcomp>
1:16.24 argv = [unicode(arg, sys.stdin.encoding) for arg in sys.argv]
1:16.24 argv = [unicode(arg, sys.stdin.encoding) for arg in sys.argv]
1:16.24 NameError: name 'unicode' is not defined
1:16.24 NameError: name 'unicode' is not defined

I solved this with https://hg.mozilla.org/mozilla-central/rev/b6bd082527123a713240e85ecc94062fd437a895

But I can't solve this error:

1:18.61 File "/var/uhubuild/work/compile/python/mozbuild/mozbuild/action/check_binary.py", line 82, in iter_readelf_symbols
1:18.61 if len(data) >= 8 and data[0].endswith(':') and data[0][:-1].isdigit():
1:18.61 TypeError: endswith first arg must be bytes or a tuple of bytes, not str

Comment 18

[rezso]

Attachment: python3-fixes.patch

I solved few python errors with this patch, but:

1:13.69 File "/var/uhubuild/work/compile/layout/style/GenerateCSSPropsGenerated.py", line 78
1:13.69 ps = sorted(ps, key=lambda (p, position): p["index"])
1:13.69 ^
1:13.69 SyntaxError: invalid syntax

The layout/style code comletely rewritten in mozilla-central.

Comment 19

[Myckel Habets]

@rezso, Python 3 isn't supported yet. I've been working on getting it in for the past year. The things you mention have been ported already, but not included yet, because they interfered with the dev team's build/testing systems (they needed to be updated for those to work). I'm not sure at what stage that is. The whole configure process should already support Python 3, but for the building a few scripts still need to be converted.

I think that when those systems are updated, Python3 support can be in quickly. Than we can finally ditch Python2 support.

Comment 20

[Frank-Rainer Grahl (:frg)]

Various backports landed in 2.53.17b1. If there is still a problem please open a new bug.