Frequent Panda build failure with "ERROR - Return code: 2" following "cannot create debug link section `out/target/product/panda/symbols/system/lib/libdl.so': Invalid operation"

RESOLVED FIXED

Status

Firefox OS
GonkIntegration
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: philor, Assigned: ted)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=17081223&tree=Mozilla-Inbound
b2g_mozilla-inbound_panda_dep on 2012-11-15 14:58:39 PST for push dc61a415f3e9
slave: bld-linux64-ec2-044

15:01:22     INFO -  HOST_BUILD_TYPE=release
15:01:22     INFO -  BUILD_ID=OPENMASTER
15:01:22     INFO -  OUT_DIR=out
15:01:22     INFO -  ============================================
15:02:28  WARNING -  external/busybox/Android.mk:33: warning: overriding commands for target `out/target/product/panda/obj/EXECUTABLES/busybox_intermediates/busybox'
15:02:28  WARNING -  build/core/prebuilt.mk:117: warning: ignoring old commands for target `out/target/product/panda/obj/EXECUTABLES/busybox_intermediates/busybox'
15:02:37  WARNING -  external/screencap-gonk/Android.mk:35: warning: overriding commands for target `out/target/product/panda/system/bin/screencap-gonk'
15:02:37  WARNING -  build/core/base_rules.mk:532: warning: ignoring old commands for target `out/target/product/panda/system/bin/screencap-gonk'
15:02:41     INFO -  find: `../../external/nist-sip/java': No such file or directory
15:02:51  WARNING -  gaia/Android.mk:32: warning: overriding commands for target `out/target/product/panda/data/local/gaia'
15:02:51  WARNING -  build/core/base_rules.mk:532: warning: ignoring old commands for target `out/target/product/panda/data/local/gaia'
15:02:51  WARNING -  gonk-misc/Android.mk:26: warning: overriding commands for target `out/target/product/panda/obj/ETC/init.rc_intermediates/init.rc'
15:02:51  WARNING -  build/core/prebuilt.mk:117: warning: ignoring old commands for target `out/target/product/panda/obj/ETC/init.rc_intermediates/init.rc'
15:02:51  WARNING -  gonk-misc/Android.mk:155: warning: overriding commands for target `out/target/product/panda/system/gecko'
15:02:51  WARNING -  build/core/base_rules.mk:532: warning: ignoring old commands for target `out/target/product/panda/system/gecko'
15:02:51  WARNING -  gonk-misc/Android.mk:214: warning: overriding commands for target `out/target/product/panda/obj/DATA/gecko_intermediates/gecko'
15:02:51  WARNING -  build/core/prebuilt.mk:117: warning: ignoring old commands for target `out/target/product/panda/obj/DATA/gecko_intermediates/gecko'
15:02:58     INFO -  No private recovery resources for TARGET_DEVICE panda
15:03:01     INFO -  target Strip: libdl (out/target/product/panda/obj/lib/libdl.so)
15:03:01     INFO -  target Strip: libz (out/target/product/panda/obj/lib/libz.so)
15:03:01     INFO -  prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-objcopy:out/target/product/panda/obj/lib/stjsaA4o: cannot create debug link section `out/target/product/panda/symbols/system/lib/libdl.so': Invalid operation
15:03:01     INFO -  make: *** [out/target/product/panda/obj/lib/libdl.so] Error 1
15:03:01     INFO -  make: *** Deleting file `out/target/product/panda/obj/lib/libdl.so'
15:03:01     INFO -  make: *** Waiting for unfinished jobs....
15:03:01     INFO -  target Strip: libstlport (out/target/product/panda/obj/lib/libstlport.so)
15:03:01     INFO -  prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-objcopy:out/target/product/panda/obj/lib/stDNKEFx: cannot create debug link section `out/target/product/panda/symbols/system/lib/libz.so': Invalid operation
15:03:01     INFO -  make: *** [out/target/product/panda/obj/lib/libz.so] Error 1
15:03:01     INFO -  make: *** Deleting file `out/target/product/panda/obj/lib/libz.so'
15:03:01     INFO -  prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/arm-linux-androideabi-objcopy:out/target/product/panda/obj/lib/stugRsDy: cannot create debug link section `out/target/product/panda/symbols/system/lib/libstlport.so': Invalid operation
15:03:01     INFO -  make: *** [out/target/product/panda/obj/lib/libstlport.so] Error 1
15:03:01     INFO -  make: *** Deleting file `out/target/product/panda/obj/lib/libstlport.so'
15:03:01     INFO -  real	1m40.497s
15:03:01     INFO -  user	0m10.095s
15:03:01     INFO -  sys	0m4.577s
15:03:01     INFO -  
15:03:01     INFO -  > Build failed! <
15:03:01     INFO -  Build with |./build.sh -j1| for better messages
15:03:01     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
15:03:01    ERROR - Return code: 2
15:03:01     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g-m-in-panda-dep/build', '--unpriv', '--shell', '/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com MAIL=/var/spool/mail/cltbld CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-044.build.aws-us-west-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json CCACHE_DIR=/builds/ccache ccache -s'] in /builds/slave/b2g-m-in-panda-dep/build
15:03:01     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g-m-in-panda-dep/build --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com MAIL=/var/spool/mail/cltbld CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-044.build.aws-us-west-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json CCACHE_DIR=/builds/ccache ccache -s"
15:03:02     INFO -  cache directory                     /builds/ccache
15:03:02     INFO -  cache hit (direct)                     0
15:03:02     INFO -  cache hit (preprocessed)               0
15:03:02     INFO -  cache miss                             0
15:03:02     INFO -  files in cache                    228951
15:03:02     INFO -  cache size                           9.3 Gbytes
15:03:02     INFO -  max cache size                      10.0 Gbytes
15:03:02     INFO - Return code: 0
15:03:02    FATAL - failed to build
15:03:02    FATAL - Exiting 2
Not sure who is on the hook for B2G build stuff; Ted/Mike, can you help or else suggest who might be able to? :-)
(In reply to Ed Morley [:edmorley UTC+0] from comment #17)
> Not sure who is on the hook for B2G build stuff; Ted/Mike, can you help or
> else suggest who might be able to? :-)

That would be mwu.

Comment 21

6 years ago
I've never seen this in a build. Maybe some sort of build env issue?

tzimmermann, have you ever seen this error?
It's possible this is related to my symbol dumping changes. We're probably running strip on these binaries now. Would that break things?
You can test this theory by running "build.sh buildsymbols" locally.
mwu thinks the symbol stripping might be breaking things, so I filed a gonk PR to stop doing that:
https://github.com/mozilla-b2g/gonk-misc/pull/56
Sorry, I've never seen this problem during builds. It looks like you first strip the libraries and later try to access the (now removed) debugging symbols.
We're using GNU gold 2.20 and related tools. Without deeper research, I'd say you trigger this bug:

  http://sourceware.org/bugzilla/show_bug.cgi?id=11072
Assignee: nobody → ted
Depends on: 810994
Keywords: intermittent-failure
The pull request mentioned in comment 28 was closed 4 days ago. I lose track of how the emulator builds work - someone has to manually update it, right?

https://tbpl.mozilla.org/php/getParsedLog.php?id=17280075&tree=Mozilla-Inbound
(In reply to Ed Morley [:edmorley, UTC+0, email: emorley at moco] from comment #84)
> The pull request mentioned in comment 28 was closed 4 days ago. I lose track
> of how the emulator builds work - someone has to manually update it, right?

Oh bug 810994 comment 13.
Ted, this is another fail even after updating gonk-misc to aafa9ca4d2e9e755fe9964018a9797eac4ecc7de:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17292460&tree=Mozilla-Inbound

In the few builds after I landed the snapshot update we had green on both clobber and dep builds, but only had 6 builds in total until this fail.
Whiteboard: [orange]
Is this fixed by bug 810994 or not? The most recent comment there seems to indicate it is, but we still have failure logs here on inbound. Does inbound just need a clobber?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #96)
> Is this fixed by bug 810994 or not? The most recent comment there seems to
> indicate it is, but we still have failure logs here on inbound. Does inbound
> just need a clobber?

I don't believe this is fixed.

However, I can't re-clobber just to be sure; due to bug 815556.
I clobbered inbound for this at 2012-11-24 16:44:55, so would love to know if there were any fails after that which are not recorded here. I'll have a look in the buildbot db too.
There has been one failure on inbound since the clobber, which was an unrelated code bustage. There has been one failure on mozilla-beta since I landed+clobbered there, also unrelated code error. No failures on central since the 20th, and can't see any other branches with issues. I've just landed on Aurora to complete the set.

---> This bug is fixed.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.