Build failure due to missing codegen.pp

RESOLVED FIXED in Firefox -esr52

Status

defect
RESOLVED FIXED
3 years ago
8 months ago

People

(Reporter: martin, Assigned: jbeich)

Tracking

({regression})

52 Branch
mozilla54
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr45 unaffected, firefox50 unaffected, firefox51 wontfix, firefox52 wontfix, firefox-esr52 fixed, firefox53 fixed, firefox54 fixed)

Details

Attachments

(1 attachment)

Trying to build from the 52.0a1 on NetBSD fails with:

gmake[3]: Entering directory '/usr/pkgobj/wip/firefox-hg/work/build/dom/bindings'
Makefile:52: codegen.pp: No such file or directory
CSS2Properties.webidl
/usr/pkgobj/wip/firefox-hg/work/build/_virtualenv/bin/python -m mozbuild.action.file_generate /usr/pkgobj/wip/firefox-hg/work/mozilla/dom/bindings/GenerateCSS2PropertiesWebIDL.py generate CSS2Properties.webidl .deps/CSS2Properties.webidl.pp /usr/pkgobj/wip/firefox-hg/work/mozilla/dom/webidl/CSS2Properties.webidl.in /usr/pkgobj/wip/firefox-hg/work/mozilla/layout/style/PythonCSSProps.h
Traceback (most recent call last):
  File "/usr/pkg/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/pkg/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/pkgobj/wip/firefox-hg/work/mozilla/python/mozbuild/mozbuild/action/file_generate.py", line 108, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/pkgobj/wip/firefox-hg/work/mozilla/python/mozbuild/mozbuild/action/file_generate.py", line 63, in main
    ret = module.__dict__[method](output, *args.additional_arguments)
  File "/usr/pkgobj/wip/firefox-hg/work/mozilla/dom/bindings/GenerateCSS2PropertiesWebIDL.py", line 22, in generate
    preprocessed = subprocess.check_output(cpp)
  File "/usr/pkg/lib/python2.7/subprocess.py", line 567, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/pkg/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/pkg/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied
backend.mk:8: recipe for target 'CSS2Properties.webidl' failed
gmake[3]: *** [CSS2Properties.webidl] Error 1
gmake[3]: *** Deleting file 'CSS2Properties.webidl'
gmake[3]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/build/dom/bindings'
/usr/pkgobj/wip/firefox-hg/work/mozilla/config/recurse.mk:79: recipe for target 'dom/bindings/export' failed
gmake[2]: *** [dom/bindings/export] Error 2
gmake[2]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/build'
/usr/pkgobj/wip/firefox-hg/work/mozilla/config/recurse.mk:32: recipe for target 'export' failed
gmake[1]: *** [export] Error 2

I guess codegen.pp should be generated from security/sandbox/chromium/sandbox/linux/bpf_dsl/codegen.cc and that is not build for NetBSD (not quite sure whether it should yet).

I am also not sure if the "Permission denied" error is refering to the same thing.
> '/usr/pkgobj/wip/firefox-hg/work/build/dom/bindings'

Can you try different MOZ_OBJDIR ? build/ is already occupied by some miscellaneous stuff.
And maybe try building as |./mach build|.
What fails with "Permission denied" is probably attempt to re-invoke python. codegen.pp is generated by https://dxr.mozilla.org/mozilla-central/rev/d38d06f85ef5/dom/bindings/Makefile.in#46
I see.
But the Makefile.in there includes codegen.pp, even before it exists. So that part is probably a spurious warning only, the question is more why the python call supposed to generate it fails.
Is there an easy way to make mozbuild show the child command it is trying to run?
Try adding -v flag e.g., 

  $ ./mach build -v |& fgrep mozbuild.action
  [...]
  objdir/_virtualenv/bin/python -m mozbuild.action.webidl dom/bindings
(In reply to Martin Husemann from comment #4)
> Is there an easy way to make mozbuild show the child command it is trying to
> run?

Maybe ask a build peer. CC'd glandium.
Building with "mach build -v" seems to work - I am confused big time now.

There is no codegen.pp generated, but the build continues:

 1:00.61 There are no private exports.
 1:00.61 gmake[7]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/security/nss/lib/
ssl'
 1:00.61 cd pkcs7; gmake private_export
 1:00.61 gmake[7]: Entering directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/security/nss/lib
/pkcs7'
 1:00.65 gmake[5]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/obj-x86_64-unknow
n-netbsd7.99/security/apps'
 1:00.65 gmake[5]: Entering directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/obj-x86_64-unkno
wn-netbsd7.99/dom/bindings'
 1:00.65 Makefile:52: codegen.pp: No such file or directory
 1:00.65 CSS2Properties.webidl
 1:00.65 /usr/pkgobj/wip/firefox-hg/work/mozilla/obj-x86_64-unknown-netbsd7.99/_virtualenv/bin/python -m mozbuild.action.file_generate /usr/pkgobj/wip/firefox-hg/work/mozilla/dom/bindings/GenerateCSS2PropertiesWebIDL.py generate CSS2Properties.webidl .deps/CSS2Properties.webidl.pp /usr/pkgobj/wip/firefox-hg/work/mozilla/dom/webidl/CSS2Properties.webidl.in /usr/pkgobj/wip/firefox-hg/work/mozilla/layout/style/PythonCSSProps.h
 1:00.67 /usr/pkgobj/wip/firefox-hg/work/mozilla/obj-x86_64-unknown-netbsd7.99/config/nsinstall -R -m 444 p7local.h /usr/pkgobj/wip/firefox-hg/work/mozilla/obj-x86_64-unknown-netbsd7.99/dist/private/nss
 1:00.67 gmake[7]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/security/nss/lib/pkcs7'
 1:00.67 cd pkcs12; gmake private_export
 1:00.67 gmake[7]: Entering directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/security/nss/lib/pkcs12'
 1:00.73 There are no private exports.
 1:00.73 gmake[7]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/security/nss/lib/pkcs12'
 1:00.73 cd smime; gmake private_export
 1:00.75 gmake[7]: Entering directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/security/nss/lib/smime'
 1:00.75 gmake[5]: Leaving directory '/usr/pkgobj/wip/firefox-hg/work/mozilla/obj-x86_64-unknown-netbsd7.99/gfx/thebes'
 1:00.75 rm -f Navigator.webidl


The missing codegen.pp is probably due to one of the config options:

 0:08.98 Adding configure options from /tmp/config.args
 0:08.98   --enable-application=browser
 0:08.98   --prefix=/usr/pkg
 0:08.98   --disable-tests
 0:08.98   --with-pthreads
 0:08.98   --enable-default-toolkit=cairo-gtk2
 0:08.98   --enable-system-cairo
 0:08.98   --enable-system-pixman
 0:08.98   --with-system-libvpx
 0:08.98   --enable-system-hunspell
 0:08.98   --enable-system-ffi
 0:08.98   --with-system-icu
 0:08.98   --with-system-jpeg
 0:08.98   --with-system-zlib
 0:08.98   --with-system-bz2
 0:08.98   --with-system-libevent=/usr
 0:08.98   --disable-crashreporter
 0:08.98   --disable-necko-wifi
 0:08.98   --enable-chrome-format=flat
 0:08.98   --disable-libjpeg-turbo
 0:08.98   --disable-elf-hack
 0:08.99   --disable-gconf
 0:08.99   --enable-gio
 0:08.99   --enable-extensions=gio
 0:08.99   --enable-url-classifier
 0:08.99   --disable-icf
 0:08.99   --disable-necko-wifi
 0:08.99   --disable-updater
 0:08.99   --disable-alsa
 0:08.99   --disable-dbus
 0:08.99   --disable-gnomeui
 0:08.99   --disable-jemalloc
 0:08.99   --enable-debug-symbols
 0:08.99   --disable-debug
 0:08.99   --enable-optimize=-O2
 0:09.00   --enable-install-strip
 0:09.00   --disable-pulseaudio
 0:09.00   --disable-webrtc

I have no idea why mach build proceeds while a simple make fails.
The errno 13 comes from trying to exec "/", so I guess some python var is accidentally undefined in my build configuration.
mozbuild in Firefox 51 no longer likes CPP passed from environment. Does this affect downstream builds on Arch, Gentoo, Debian or OpenBSD?

  $ rm .mozconfig
  $ ./mach clobber
  $ CPP=cpp ./mach build
  [...]
   0:18.59 Traceback (most recent call last):
   0:18.59   File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
   0:18.61     "__main__", fname, loader, pkg_name)
   0:18.61   File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
   0:18.61     exec code in run_globals
   0:18.61   File "python/mozbuild/mozbuild/action/file_generate.py", line 108, in <module>
   0:18.61     sys.exit(main(sys.argv[1:]))
   0:18.61   File "python/mozbuild/mozbuild/action/file_generate.py", line 63, inmain
   0:18.61     ret = module.__dict__[method](output, *args.additional_arguments)
   0:18.61   File "dom/bindings/GenerateCSS2PropertiesWebIDL.py", line 22, in generate
   0:18.61     preprocessed = subprocess.check_output(cpp)
   0:18.61   File "/usr/local/lib/python2.7/subprocess.py", line 212, in check_output
   0:18.61     process = Popen(stdout=PIPE, *popenargs, **kwargs)
   0:18.61   File "/usr/local/lib/python2.7/subprocess.py", line 390, in __init__
   0:18.61     errread, errwrite)
   0:18.61   File "/usr/local/lib/python2.7/subprocess.py", line 1024, in _execute_child
   0:18.61     raise child_exception
   0:18.61 OSError: [Errno 13] Permission denied
   0:18.61 Makefile:52: codegen.pp: No such file or directory
(In reply to Jan Beich from comment #9)
> mozbuild in Firefox 51 no longer likes CPP passed from environment. Does
> this affect downstream builds on Arch, Gentoo, Debian or OpenBSD?
> 
>   $ rm .mozconfig
>   $ ./mach clobber
>   $ CPP=cpp ./mach build
>   [...]
>    0:18.59 Traceback (most recent call last):
>    0:18.59   File "/usr/local/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
>    0:18.61     "__main__", fname, loader, pkg_name)
>    0:18.61   File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
>    0:18.61     exec code in run_globals
>    0:18.61   File "python/mozbuild/mozbuild/action/file_generate.py", line
> 108, in <module>
>    0:18.61     sys.exit(main(sys.argv[1:]))
>    0:18.61   File "python/mozbuild/mozbuild/action/file_generate.py", line
> 63, inmain
>    0:18.61     ret = module.__dict__[method](output,
> *args.additional_arguments)
>    0:18.61   File "dom/bindings/GenerateCSS2PropertiesWebIDL.py", line 22,
> in generate
>    0:18.61     preprocessed = subprocess.check_output(cpp)
>    0:18.61   File "/usr/local/lib/python2.7/subprocess.py", line 212, in
> check_output
>    0:18.61     process = Popen(stdout=PIPE, *popenargs, **kwargs)
>    0:18.61   File "/usr/local/lib/python2.7/subprocess.py", line 390, in
> __init__
>    0:18.61     errread, errwrite)
>    0:18.61   File "/usr/local/lib/python2.7/subprocess.py", line 1024, in
> _execute_child
>    0:18.61     raise child_exception
>    0:18.61 OSError: [Errno 13] Permission denied
>    0:18.61 Makefile:52: codegen.pp: No such file or directory

Gentoo does not define CPP we use CXX which has seen no problems in my local testing.

--host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-application=browser PKG_CONFIG=x86_64-pc-linux-gnu-pkg-config --enable-default-toolkit=cairo-gtk3 --with-google-api-keyfile=/var/tmp/portage/www-client/firefox-51.0_beta14/work/firefox-51.0b14/google-api-key --disable-tests CC=x86_64-pc-linux-gnu-gcc CXX=x86_64-pc-linux-gnu-g++ HOST_CC=x86_64-pc-linux-gnu-gcc HOST_CXX=x86_64-pc-linux-gnu-g++ --disable-debug-symbols LD=x86_64-pc-linux-gnu-ld MOZ_JEMALLOC4=1 --enable-replace-malloc --enable-system-ffi MAKE=make XARGS=/usr/bin/xargs --enable-system-hunspell --disable-crashreporter --enable-dbus --enable-extensions=default --disable-gconf --enable-gio --disable-gnomeui --disable-gold --disable-install-strip --enable-ion --disable-necko-wifi --enable-official-branding --enable-optimize=-O2 --enable-pie --enable-pulseaudio --disable-startup-notification --disable-strip --disable-system-cairo --enable-system-sqlite --disable-updater --libdir=/usr/lib64 --prefix=/usr --with-default-mozilla-five-home=/usr/lib64/firefox --with-intl-api --with-nspr-prefix=/usr --with-nss-prefix=/usr --with-system-bz2 --with-system-graphite2 --with-system-harfbuzz --with-system-icu --with-system-jpeg --with-system-libevent=/usr --with-system-libvpx --with-system-nspr --with-system-nss --with-system-png --with-system-zlib --x-includes=/usr/include --x-libraries=/usr/lib64
(In reply to Jan Beich from comment #9)
> mozbuild in Firefox 51 no longer likes CPP passed from environment. Does
> this affect downstream builds on Arch, Gentoo, Debian or OpenBSD?

We do not set CPP on Arch (no cross-compiling) so this doesn't affect us, either.
Per comment #11, mark 51 won't fix as it's too late for 51 and it won't affect Arch.
Comment on attachment 8840626 [details]
Bug 1316879 - Don't use CPP/CXXCPP from environment after bug 1299919.

https://reviewboard.mozilla.org/r/115078/#review117646
Attachment #8840626 - Flags: review?(mh+mozilla) → review+
Keywords: checkin-needed
Assignee: nobody → jbeich
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/0041f17d932f
Don't use CPP/CXXCPP from environment after bug 1299919. r=glandium
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/0041f17d932f
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 54
Too late for Fx52 at this point, but we could probably consider uplifting this to ESR52 and Fx53, though. Please request approval if you want to do so.
Comment on attachment 8840626 [details]
Bug 1316879 - Don't use CPP/CXXCPP from environment after bug 1299919.

Agree, I'd like this backported to Firefox 53.0 and 52.1.0 ESR. The merge date is close, so one of approval* flags maybe redundant.

Approval Request Comment
[ESR consideration]: Easier build downstream when overriding CC/CXX/CPP/CXXCPP for packages en masse
[Feature/Bug causing the regression]: Bug 1299919 regression
[User impact if declined]: Broken build in PkgSrc and FreeBSD Ports
[Is this code covered by automated tests?]: Only downstream, TaskCluster builds don't pass CPP/CXXCPP
[Has the fix been verified in Nightly?]: Yes, also downstream for Firefox 52.0
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Can only break build
[String changes made/needed]: None
Flags: needinfo?(jbeich)
Attachment #8840626 - Flags: approval-mozilla-esr52?
Attachment #8840626 - Flags: approval-mozilla-beta?
Attachment #8840626 - Flags: approval-mozilla-aurora?
Comment on attachment 8840626 [details]
Bug 1316879 - Don't use CPP/CXXCPP from environment after bug 1299919.

Let's take this for aurora 53 now. We can look at esr later (sorry this missed the cutoff for 52.0esr)
Attachment #8840626 - Flags: approval-mozilla-beta?
Attachment #8840626 - Flags: approval-mozilla-beta-
Attachment #8840626 - Flags: approval-mozilla-aurora?
Attachment #8840626 - Flags: approval-mozilla-aurora+
Comment on attachment 8840626 [details]
Bug 1316879 - Don't use CPP/CXXCPP from environment after bug 1299919.

build system fix for esr52, should be in 52.1.0
Attachment #8840626 - Flags: approval-mozilla-esr52? → approval-mozilla-esr52+
Component: Build Config → General
Product: Firefox → Firefox Build System
Keywords: regression
Target Milestone: Firefox 54 → mozilla54
You need to log in before you can comment on or make changes to this bug.