Closed Bug 1772676 Opened 3 years ago Closed 3 years ago

build fail with an error on isc

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Version:
Firefox 101
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: neoedmund, Unassigned)

Details

Attachments

(2 files, 2 obsolete files)

more detailed log.txt
4.28 KB, text/plain
Details
trace.out.7z
3.27 MB, application/x-7z-compressed
Details

Steps to reproduce:

./mach build

Actual results:

0:04.77 /usr/bin/gmake -f client.mk -j24 -s
0:04.78 If you think you have received this message due to a bug rather
0:04.78 than a configuration issue please read the section on submitting
0:04.78 bugs on either our web page at www.isc.org or in the README file
0:04.78 before submitting a bug. These pages explain the proper
0:04.78 process and the information we find helpful for debugging.
0:04.78 exiting.
0:04.78 gmake[1]: *** [Makefile:157: default] Error 1
0:04.78 gmake: *** [client.mk:63: build] Error 2
0:04.78 0 compiler warnings present.

Expected results:

start build

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::General' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → General
Product: Firefox → Firefox Build System

sorry but this isn't really actionable
you should have a real error message above

and you aren't giving enough information about your options, configuration, system, etc

I followed the steps on https://firefox-source-docs.mozilla.org/setup/linux_build.html
nothing customized setting, all default.

uname -a
Linux  5.18.0-gs #1 SMP PREEMPT_DYNAMIC Wed May 25 12:31:46 JST 2022 x86_64 GNU/Linux

neoe /ram/firefox/mozilla-unified > ./mach build
 0:00.44 Clobber not needed.
 0:00.46 Adding make options from /ram/firefox/mozilla-unified/mozconfig
    MOZ_OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    FOUND_MOZCONFIG=/ram/firefox/mozilla-unified/mozconfig
    export FOUND_MOZCONFIG
  Parallelism determined by memory: using 24 jobs for 24 cores based on 125.8 GiB RAM and estimated job size of 1.0 GiB
 0:00.46 /usr/bin/gmake -f client.mk -j24 -s
 0:00.48 If you think you have received this message due to a bug rather
 0:00.48 than a configuration issue please read the section on submitting
 0:00.48 bugs on either our web page at www.isc.org or in the README file
 0:00.48 before submitting a bug.  These pages explain the proper
 0:00.48 process and the information we find helpful for debugging.
 0:00.48 exiting.
 0:00.48 gmake[1]: *** [Makefile:157: default] Error 1
 0:00.48 gmake: *** [client.mk:63: build] Error 2
 0:00.50 0 compiler warnings present.

besides python3 bootstrap.py , also I tried source download from
wget "https://hg.mozilla.org/mozilla-central/archive/FIREFOX_NIGHTLY_102_END.zip"
and
git clone https://github.com/bolucat/Firefox

they all come to this exactly same error.

Which Distro and version do you use?
Uname isnt giving enough info

Debian 11 "bullseye" based it is. Is it related?

Attached file log (obsolete) —
more detailed log
Attached file log (obsolete) —
more detailed log
Attached file more detailed log.txt —

more detailed log

Attachment #9279820 - Attachment is obsolete: true
Attachment #9279821 - Attachment is obsolete: true

Makefile:157 is something that should only run when SCCACHE_VERBOSE_STATS is set, which it most probably isn't, and the error message is from isc-dhcpd, which has nothing to do with Firefox. Are you sure your source tree is not corrupted?

Makefile:157 is not Makefile.in:157 are you sure its about sccache?
Why build firefox need to use isc-dhcpd and why isc-dhcpd stopped the firefox building?
I clean the ~/.hgrc and bootstrap.py the source tree again, but nothing changed.
mach seems smart but because of that I cannot find where the Makefile is and where the isc-dhcpd message comes from...

Indeed, Makefile:157 is looking more normal, but the build system doesn't use isc-dhcpd. Try mach build -v, it might give more information.

More information, but I'm afraid still not enough informative.

./mach -v build
 0:00.44 Clobber not needed.
 0:00.45 Adding make options from /ram/firefox/mozilla-unified/mozconfig
    MOZ_OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    FOUND_MOZCONFIG=/ram/firefox/mozilla-unified/mozconfig
    export FOUND_MOZCONFIG
  Parallelism determined by memory: using 24 jobs for 24 cores based on 125.8 GiB RAM and estimated job size of 1.0 GiB
 0:00.45 /usr/bin/gmake -f client.mk -j24 -s
 0:00.45 Environment: {'SHELL': '/bin/bash', 'COLORTERM': 'truecolor', 'JAVA_HOME': '/opt/jdk-19', 'PWD': '/ram/firefox/mozilla-unified', 'LOGNAME': 'neoe', 'XAUTHORITY': '/home/neoe/.Xauthority', 'WINDOWPATH': '7', 'HOME': '/home/neoe', 'VULKAN_SDK': '/opt/vulkansdk/1.3.211.0/x86_64', 'LANG': 'C.UTF-8', 'VTE_VERSION': '6203', 'neoebuild_base': '/home/neoe/oss/lab', 'TERM': 'xterm-256color', 'USER': 'neoe', 'DISPLAY': ':0.0', 'SHLVL': '6', 'ANDROID_SDK_ROOT': '/opt/androidsdk', 'LD_LIBRARY_PATH': '/opt/vulkansdk/1.3.211.0/x86_64/lib:/opt/vulkansdk/1.3.211.0/x86_64/lib', 'TILIX_ID': '2d6a0b44-9c97-493f-b0b3-9add7ec6a5cb', 'PS1': '\\e[1;32m\\u \\e[0;94m\\w \\e[0;35m\\!>\\e[0m ', 'CONSOLES_DEBUG': '1', 'GDK_BACKEND': 'x11', 'PATH': '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin:/home/neoe/.mozbuild/srcdirs/mozilla-unified-06bb8a315fee/_virtualenvs/mach/bin:/home/neoe/.cargo/bin:/opt/vulkansdk/1.3.211.0/x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/jdk-19/bin:/opt/androidsdk/platform-tools:/home/neoe/oss/lab/bins:/home/neoe/.local/bin', 'VK_ADD_LAYER_PATH': '/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d:/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d:/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d:/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d', 'OLDPWD': '/ram/firefox', '_': './mach', 'VIRTUAL_ENV': '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build', 'MACH_MAIN_PID': '13975', 'MACH_STDOUT_ISATTY': '1', 'TOPSRCDIR': '/ram/firefox/mozilla-unified', 'CONFIG_GUESS': 'x86_64-pc-linux-gnu', 'OBJDIR': '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu', 'MACH': '1', 'PYTHONUNBUFFERED': '1'}
 0:00.46 If you think you have received this message due to a bug rather
 0:00.46 than a configuration issue please read the section on submitting
 0:00.47 bugs on either our web page at www.isc.org or in the README file
 0:00.47 before submitting a bug.  These pages explain the proper
 0:00.47 process and the information we find helpful for debugging.
 0:00.47 exiting.
 0:00.47 gmake[1]: *** [Makefile:157: default] Error 1
 0:00.47 gmake: *** [client.mk:63: build] Error 2
 0:00.47 0 compiler warnings present.
 0:00.47 Unable to run active build backend's post-build step; failing the build due to exception: [Errno 2] No such file or directory: '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/dist/bin/browser/.purgecaches'.

./mach -v clobber
neoe /ram/firefox/mozilla-unified 510> ./mach -v build
created virtual environment CPython3.9.2.final.0-64 in 23ms
  creator CPython3Posix(dest=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
 0:00.70 Clobber not needed.
 Config object not found by mach.
 0:00.82 Using Python 3.9.2 from /ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python
 0:00.82 Adding configure options from /ram/firefox/mozilla-unified/mozconfig
 0:00.82   --enable-artifact-builds
 0:00.82 checking for vcs source checkout... hg
 0:00.98 checking for a shell... /bin/sh
 0:01.00 checking for host system type... x86_64-pc-linux-gnu
 0:01.00 checking for target system type... x86_64-pc-linux-gnu
 0:01.09 checking whether cross compiling... no
 0:01.18 checking for Python 3... /ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python (3.9.2)
 0:01.18 checking for hg... /home/neoe/.local/bin/hg
 0:01.27 checking for Mercurial version... 6.1.3
 0:01.40 checking for sparse checkout... no
 0:03.30 checking for wget... /usr/bin/wget
 0:03.34 checking for nodejs... /home/neoe/.mozbuild/node/bin/node (12.22.1)
 0:03.34 checking for tar... /bin/tar
 0:03.34 checking for unzip... /usr/bin/unzip
 0:03.34 checking for gn... not found
 0:03.34 checking for the Mozilla API key... no
 0:03.34 checking for the Google Location Service API key... no
 0:03.34 checking for the Google Safebrowsing API key... no
 0:03.34 checking for the Bing API key... no
 0:03.34 checking for the Adjust SDK key... no
 0:03.34 checking for the Leanplum SDK key... no
 0:03.34 checking for the Pocket API key... no
 0:03.35 checking for awk... /usr/bin/gawk
 0:03.35 checking for perl... /usr/bin/perl
 0:03.35 checking for minimum required perl version >= 5.006... 5.032001
 0:03.36 checking for full perl installation... yes
 0:03.36 checking for gmake... /usr/bin/gmake
 0:03.36 checking for watchman... not found
 0:03.36 checking for xargs... /usr/bin/xargs
 0:03.36 checking for rpmbuild... not found
 0:03.36 checking for m4... /usr/bin/m4
 0:03.37 creating cache ./config.cache
 0:03.38 checking host system type... x86_64-pc-linux-gnu
 0:03.39 checking target system type... x86_64-pc-linux-gnu
 0:03.39 checking build system type... x86_64-pc-linux-gnu
 0:03.40 checking if app-specific confvars.sh exists... /ram/firefox/mozilla-unified/browser/confvars.sh
 0:03.41 updating cache ./config.cache
 0:03.41 creating ./config.data
 0:03.42 Creating config.status
 0:03.50 Reticulating splines...
 0:03.91  0:00.43 File already read. Skipping: /ram/firefox/mozilla-unified/gfx/angle/targets/angle_common/moz.build
 0:07.51 Finished reading 1917 moz.build files in 1.52s
 0:07.51 Read 0 gyp files in parallel contributing 0.00s to total wall time
 0:07.51 Processed into 10366 build config descriptors in 1.15s
 0:07.51 FasterMake+RecursiveMake backend executed in 0.71s
 0:07.51   3088 total backend files; 3088 created; 0 updated; 0 unchanged; 0 deleted
 0:07.51 Total wall time: 4.03s; CPU time: 4.03s; Efficiency: 100%; Untracked: 0.64s
Configure complete!
Be sure to run |mach build| to pick up any changes
 0:07.83 Adding make options from /ram/firefox/mozilla-unified/mozconfig
    MOZ_OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    FOUND_MOZCONFIG=/ram/firefox/mozilla-unified/mozconfig
    export FOUND_MOZCONFIG
  Parallelism determined by memory: using 24 jobs for 24 cores based on 125.8 GiB RAM and estimated job size of 1.0 GiB
 0:07.83 /usr/bin/gmake -f client.mk -j24 -s
 0:07.83 Environment: {'SHELL': '/bin/bash', 'COLORTERM': 'truecolor', 'JAVA_HOME': '/opt/jdk-19', 'PWD': '/ram/firefox/mozilla-unified', 'LOGNAME': 'neoe', 'XAUTHORITY': '/home/neoe/.Xauthority', 'WINDOWPATH': '7', 'HOME': '/home/neoe', 'VULKAN_SDK': '/opt/vulkansdk/1.3.211.0/x86_64', 'LANG': 'C.UTF-8', 'VTE_VERSION': '6203', 'neoebuild_base': '/home/neoe/oss/lab', 'TERM': 'xterm-256color', 'USER': 'neoe', 'DISPLAY': ':0.0', 'SHLVL': '6', 'ANDROID_SDK_ROOT': '/opt/androidsdk', 'LD_LIBRARY_PATH': '/opt/vulkansdk/1.3.211.0/x86_64/lib:/opt/vulkansdk/1.3.211.0/x86_64/lib', 'TILIX_ID': '2d6a0b44-9c97-493f-b0b3-9add7ec6a5cb', 'PS1': '\\e[1;32m\\u \\e[0;94m\\w \\e[0;35m\\!>\\e[0m ', 'CONSOLES_DEBUG': '1', 'GDK_BACKEND': 'x11', 'PATH': '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin:/home/neoe/.mozbuild/srcdirs/mozilla-unified-06bb8a315fee/_virtualenvs/mach/bin:/home/neoe/.cargo/bin:/opt/vulkansdk/1.3.211.0/x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/jdk-19/bin:/opt/androidsdk/platform-tools:/home/neoe/oss/lab/bins:/home/neoe/.local/bin', 'VK_ADD_LAYER_PATH': '/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d:/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d:/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d:/opt/vulkansdk/1.3.211.0/x86_64/etc/vulkan/explicit_layer.d', 'OLDPWD': '/ram/firefox', '_': './mach', 'VIRTUAL_ENV': '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/_virtualenvs/build', 'MACH_MAIN_PID': '18806', 'MACH_STDOUT_ISATTY': '1', 'TOPSRCDIR': '/ram/firefox/mozilla-unified', 'CONFIG_GUESS': 'x86_64-pc-linux-gnu', 'OBJDIR': '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu', 'MACH': '1', 'PYTHONUNBUFFERED': '1'}
 0:07.84 If you think you have received this message due to a bug rather
 0:07.84 than a configuration issue please read the section on submitting
 0:07.84 bugs on either our web page at www.isc.org or in the README file
 0:07.84 before submitting a bug.  These pages explain the proper
 0:07.84 process and the information we find helpful for debugging.
 0:07.84 exiting.
 0:07.84 gmake[1]: *** [Makefile:157: default] Error 1
 0:07.84 gmake: *** [client.mk:63: build] Error 2
 0:07.84 0 compiler warnings present.
 0:07.85 Unable to run active build backend's post-build step; failing the build due to exception: [Errno 2] No such file or directory: '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/dist/bin/browser/.purgecaches'.

Try mach build -v

./mach build -v
os.nice(10)
 0:00.44 Clobber not needed.
 0:00.46 Adding make options from /ram/firefox/mozilla-unified/mozconfig
    MOZ_OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    OBJDIR=/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
    FOUND_MOZCONFIG=/ram/firefox/mozilla-unified/mozconfig
    export FOUND_MOZCONFIG
  Parallelism determined by memory: using 24 jobs for 24 cores based on 125.8 GiB RAM and estimated job size of 1.0 GiB
 0:00.46 /usr/bin/gmake -f client.mk -j24
 0:00.46 /usr/bin/gmake  -C /ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu
 0:00.47 gmake[1]: Entering directory '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu'
 0:00.47 If you think you have received this message due to a bug rather
 0:00.47 than a configuration issue please read the section on submitting
 0:00.47 bugs on either our web page at www.isc.org or in the README file
 0:00.47 before submitting a bug.  These pages explain the proper
 0:00.47 process and the information we find helpful for debugging.
 0:00.47 exiting.
 0:00.47 gmake[1]: *** [Makefile:157: default] Error 1
 0:00.47 gmake[1]: Leaving directory '/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu'
 0:00.47 gmake: *** [client.mk:63: build] Error 2
 0:00.49 0 compiler warnings present.

That didn't add any verbosity unfortunately (it did remove the -s option passed to gmake).

The line at Makefile:157 is this one: https://searchfox.org/mozilla-central/rev/1c6bb1476e86b6d428b26d04e6ea3f4367528c77/Makefile.in#149
BUILDSTATUS expands to this: https://searchfox.org/mozilla-central/rev/1c6bb1476e86b6d428b26d04e6ea3f4367528c77/config/rules.mk#226-229

IOW, this runs echo BUILDSTATUS TIERS ...

The error message that ensues suggests that you have a echo command in your PATH that is not what shells normally expect echo to be.

/ram/firefox/mozilla-unified/obj-x86_64-pc-linux-gnu/Makefile

LINE 156   default all::
LINE 157   	$(call BUILDSTATUS,TIERS $(TIERS) $(if $(MOZ_AUTOMATION),$(MOZ_AUTOMATION_TIERS)))

neoe /ram/firefox/mozilla-unified 514> echo ok
ok
neoe /ram/firefox/mozilla-unified 515> which echo
/bin/echo
neoe /ram/firefox/mozilla-unified 516> ls -l `which echo`
-rwxr-xr-x 1 neoe neoe 516816 Jun  2 14:36 /bin/echo

what do you mean about echo?

Run strace -f -o trace.out ./mach build and attach the trace.out file.

Attached file trace.out.7z — 
7380  execve("/bin/echo", ["echo", "BUILDSTATUS TIERS artifact pre-e"...], 0x562abeedc7b0 /* 39 vars */ <unfinished ...>

pid 7380 executes echo

7380  clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD <unfinished ...>
7380  <... clone resumed>, child_tidptr=0x7f4db8cd3a10) = 7381

echo forks!?!

7381  write(2, "If you think you have received t"..., 63 <unfinished ...>

That's that forked process that prints the error message.

Both processes try to connect to a socket at /dev/log

A normal echo doesn't do that.

From the log, it seems like /bin/echo might be linked to at least libdns-export.so.1110 and libisc-export.so.1105. That doesn't seem right.

You don't seem to have LD_PRELOAD set, I suspect you're rootkit'ed. I would advise to stop using this machine immediately and reinstall it from scratch.

In any case, I don't think there's anything wrong in the Firefox build system.

Status: UNCONFIRMED → RESOLVED
Closed: 3 years ago
Resolution: --- → INVALID
Summary: build fail → build fail with an error on isc
ldd /bin/echo
	linux-vdso.so.1 (0x00007fff2bdd7000)
	libdns-export.so.1110 => /lib/x86_64-linux-gnu/libdns-export.so.1110 (0x00007fcba2000000)
	libisc-export.so.1105 => /lib/x86_64-linux-gnu/libisc-export.so.1105 (0x00007fcba1f89000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcba1dc4000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcba2287000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcba2265000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fcba1a00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcba2345000)
ldd /bin/echo
	linux-vdso.so.1 (0x00007ffcfdffd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fec54509000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fec54707000)

-rwxr-xr-x 1 root root   39712 Jun  7 16:46 echo
-rwxr-xr-x 1 neoe neoe  516816 Jun  2 14:36 echo.bad

-rwxr-xr-x 1 neoe neoe 516816 May 27  2021 dhclient

lrwxrwxrwx 1 root root 9 Jun  2 13:14 /sbin/dhclient -> /bin/echo

That is mind-blowing.
There is no rootkit.
Thank you for your technically tracking down.

In short, I hate dhclient and replace it with /bin/echo, but by accident, reversed.

and echo is also a bash build-in command, so I type echo in bash gives no clue. problem only shows when execute /bin/echo

After I replace /bin/echo with an original one,
compiled smoothly.

Thank you firefox, Thank you Mike.
Best regards.

I mean thank you Ledru.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: