1324210 - Permanent false-positive "OSError: [Errno 2] No such file or directory" in artifact builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=8044661&repo=autoland

[task 2016-12-17T00:15:38.612302Z] 00:15:38     INFO -  Refreshing /home/worker/workspace/build/src/old-configure with /usr/bin/autoconf-2.13
[task 2016-12-17T00:15:38.794935Z] 00:15:38     INFO -  creating cache ./config.cache
[task 2016-12-17T00:15:38.822560Z] 00:15:38     INFO -  checking host system type... x86_64-pc-linux-gnu
[task 2016-12-17T00:15:38.838128Z] 00:15:38     INFO -  checking target system type... x86_64-pc-linux-gnu
[task 2016-12-17T00:15:38.845636Z] 00:15:38     INFO -  checking build system type... x86_64-pc-linux-gnu
[task 2016-12-17T00:15:38.845812Z] 00:15:38     INFO -  checking for objcopy... /home/worker/workspace/build/src/gcc/bin/objcopy
[task 2016-12-17T00:15:39.030296Z] 00:15:39     INFO -  checking NSPR selection... source-tree
[task 2016-12-17T00:15:39.033638Z] 00:15:39     INFO -  checking if app-specific confvars.sh exists... /home/worker/workspace/build/src/browser/confvars.sh
[task 2016-12-17T00:15:39.053917Z] 00:15:39     INFO -  checking for gio-2.0 >= 2.22... yes
[task 2016-12-17T00:15:39.054113Z] 00:15:39     INFO -  checking MOZ_GIO_CFLAGS... -pthread -I/home/worker/workspace/build/src/gtk3/usr/local/include/glib-2.0 -I/home/worker/workspace/build/src/gtk3/usr/local/lib/glib-2.0/include
[task 2016-12-17T00:15:39.057450Z] 00:15:39     INFO -  checking MOZ_GIO_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/local/lib -lgio-2.0 -lgobject-2.0 -lglib-2.0
[task 2016-12-17T00:15:39.064917Z] 00:15:39     INFO -  checking for gconf-2.0 >= 1.2.1 gobject-2.0 ... yes
[task 2016-12-17T00:15:39.066375Z] 00:15:39     INFO -  checking MOZ_GCONF_CFLAGS... -DORBIT2=1 -pthread -I/home/worker/workspace/build/src/gtk3/usr/local/include/glib-2.0 -I/home/worker/workspace/build/src/gtk3/usr/local/lib/glib-2.0/include -I/home/worker/workspace/build/src/gtk3/usr/include/gconf/2 -I/home/worker/workspace/build/src/gtk3/usr/include/orbit-2.0 -I/home/worker/workspace/build/src/gtk3/usr/include/dbus-1.0 -I/home/worker/workspace/build/src/gtk3/usr/lib64/dbus-1.0/include
[task 2016-12-17T00:15:39.069681Z] 00:15:39     INFO -  checking MOZ_GCONF_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/local/lib -L/home/worker/workspace/build/src/gtk3/usr/lib64 -lgconf-2 -lgobject-2.0 -lglib-2.0
[task 2016-12-17T00:15:39.077107Z] 00:15:39     INFO -  checking for dbus-1 >= 0.60... yes
[task 2016-12-17T00:15:39.080588Z] 00:15:39     INFO -  checking MOZ_DBUS_CFLAGS... -I/home/worker/workspace/build/src/gtk3/usr/include/dbus-1.0 -I/home/worker/workspace/build/src/gtk3/usr/lib64/dbus-1.0/include
[task 2016-12-17T00:15:39.081812Z] 00:15:39     INFO -  checking MOZ_DBUS_LIBS... -L/home/worker/workspace/build/src/gtk3/lib64 -ldbus-1 -lpthread -lrt
[task 2016-12-17T00:15:39.085141Z] 00:15:39     INFO -  checking for dbus-glib-1 >= 0.60... yes
[task 2016-12-17T00:15:39.088648Z] 00:15:39     INFO -  checking MOZ_DBUS_GLIB_CFLAGS... -I/home/worker/workspace/build/src/gtk3/usr/local/include/glib-2.0 -I/home/worker/workspace/build/src/gtk3/usr/local/lib/glib-2.0/include -I/home/worker/workspace/build/src/gtk3/usr/include/dbus-1.0 -I/home/worker/workspace/build/src/gtk3/usr/lib64/dbus-1.0/include
[task 2016-12-17T00:15:39.089898Z] 00:15:39     INFO -  checking MOZ_DBUS_GLIB_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/local/lib -L/home/worker/workspace/build/src/gtk3/usr/lib64 -L/home/worker/workspace/build/src/gtk3/lib64 -ldbus-glib-1 -ldbus-1 -lpthread -lrt -lgobject-2.0 -lglib-2.0
[task 2016-12-17T00:15:39.131377Z] 00:15:39     INFO -  /home/worker/workspace/build/src/old-configure: line 14428: -E: command not found
[task 2016-12-17T00:15:39.132584Z] 00:15:39     INFO -  configure: warning: No assembler or assembly support for libvpx. Using unoptimized C routines.
[task 2016-12-17T00:15:39.148162Z] 00:15:39     INFO -  checking for libpulse... yes
[task 2016-12-17T00:15:39.148317Z] 00:15:39     INFO -  checking MOZ_PULSEAUDIO_CFLAGS... -D_REENTRANT
[task 2016-12-17T00:15:39.148464Z] 00:15:39     INFO -  checking MOZ_PULSEAUDIO_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/lib64 -lpulse
[task 2016-12-17T00:15:39.184061Z] 00:15:39     INFO -  checking for linux/joystick.h... yes
[task 2016-12-17T00:15:39.185288Z] 00:15:39     INFO -  configure: warning: No assembler or assembly support for libav-fft.  Using unoptimized C routines.
[task 2016-12-17T00:15:39.192765Z] 00:15:39     INFO -  checking for wget... wget
[task 2016-12-17T00:15:39.196130Z] 00:15:39     INFO -  configure: warning: Cannot --enable-signmar with --disable-compile-environment
[task 2016-12-17T00:15:39.284152Z] 00:15:39     INFO -  Traceback (most recent call last):
[task 2016-12-17T00:15:39.284324Z] 00:15:39     INFO -    File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
[task 2016-12-17T00:15:39.284449Z] 00:15:39     INFO -      "__main__", fname, loader, pkg_name)
[task 2016-12-17T00:15:39.284580Z] 00:15:39     INFO -    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2016-12-17T00:15:39.284729Z] 00:15:39     INFO -      exec code in run_globals
[task 2016-12-17T00:15:39.284883Z] 00:15:39     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 79, in <module>
[task 2016-12-17T00:15:39.284993Z] 00:15:39     INFO -      print 'MOZ_LIBSTDCXX_TARGET_VERSION=%s' % find_version(cxx_env)
[task 2016-12-17T00:15:39.285131Z] 00:15:39     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 64, in find_version
[task 2016-12-17T00:15:39.285248Z] 00:15:39     INFO -      p = subprocess.Popen(args, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
[task 2016-12-17T00:15:39.285374Z] 00:15:39     INFO -    File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
[task 2016-12-17T00:15:39.285489Z] 00:15:39     INFO -      errread, errwrite)
[task 2016-12-17T00:15:39.285638Z] 00:15:39     INFO -    File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
[task 2016-12-17T00:15:39.285753Z] 00:15:39     INFO -      raise child_exception
[task 2016-12-17T00:15:39.285881Z] 00:15:39     INFO -  OSError: [Errno 2] No such file or directory

Comment 1

[Intermittent Failures Robot]

12 failures in 722 pushes (0.017 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 4
* try: 3
* mozilla-inbound: 2
* mozilla-central: 2
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 9
* linux64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324210&startday=2017-01-09&endday=2017-01-15&tree=all

Comment 2

[Geoff Brown [:gbrown]]

Beginning Jan 12, there are intermittent Android failures caused by failures to run netstat; I'll open a separate bug for that.

Comment 3

[Phil Ringnalda (:philor)]

And if anyone wants to see artifact build failures correctly starred and filed, getting rid of this permanent "error" which appears in every single log, green or orange or red, would be a good idea.

Comment 4

[Intermittent Failures Robot]

21 failures in 115 pushes (0.183 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 21

Platform breakdown:
* linux64: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324210&startday=2017-01-19&endday=2017-01-19&tree=all

Comment 5

[Intermittent Failures Robot]

33 failures in 690 pushes (0.048 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 25
* autoland: 5
* mozilla-central: 2
* try: 1

Platform breakdown:
* linux64: 23
* android-4-3-armv7-api15: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324210&startday=2017-01-16&endday=2017-01-22&tree=all

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Phil Ringnalda (:philor) from comment #3)
> And if anyone wants to see artifact build failures correctly starred and
> filed, getting rid of this permanent "error" which appears in every single
> log, green or orange or red, would be a good idea.

I see: 

[task 2017-01-23T20:25:04.326179Z] 20:25:04     INFO -  Traceback (most recent call last):
[task 2017-01-23T20:25:04.326379Z] 20:25:04     INFO -    File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
[task 2017-01-23T20:25:04.326521Z] 20:25:04     INFO -      "__main__", fname, loader, pkg_name)
[task 2017-01-23T20:25:04.326682Z] 20:25:04     INFO -    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2017-01-23T20:25:04.326829Z] 20:25:04     INFO -      exec code in run_globals
[task 2017-01-23T20:25:04.326983Z] 20:25:04     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 79, in <module>
[task 2017-01-23T20:25:04.327137Z] 20:25:04     INFO -      print 'MOZ_LIBSTDCXX_TARGET_VERSION=%s' % find_version(cxx_env)
[task 2017-01-23T20:25:04.327287Z] 20:25:04     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 64, in find_version
[task 2017-01-23T20:25:04.327430Z] 20:25:04     INFO -      p = subprocess.Popen(args, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
[task 2017-01-23T20:25:04.327564Z] 20:25:04     INFO -    File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
[task 2017-01-23T20:25:04.327710Z] 20:25:04     INFO -      errread, errwrite)
[task 2017-01-23T20:25:04.327856Z] 20:25:04     INFO -    File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
[task 2017-01-23T20:25:04.327984Z] 20:25:04     INFO -      raise child_exception
[task 2017-01-23T20:25:04.328120Z] 20:25:04     INFO -  OSError: [Errno 2] No such file or directory

It looks like libstdcxx.py is trying to launch os.environ['CXX']. I don't see CXX defined in the log, but I think it must be defined in order for libstdcxx.py to get as far as the Popen().

Comment 7

[Chris Manchester (limited bugmail, email directly)]

We just need to avoid doing that check in artifact builds, it was probably missed initially because it doesn't cause the build to fail.

I remember a bug where this was discussed, but I can't seem to find it now, so we can just fix it here I suppose!

Comment 8

[Chris Manchester (limited bugmail, email directly)]

Attachment: Bug 1324210 - Don't attempt to check libstdcxx version in artifact builds.

Review commit: https://reviewboard.mozilla.org/r/106654/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/106654/

Comment 9

[Chris Manchester (limited bugmail, email directly)]

As seen in https://treeherder.mozilla.org/#/jobs?repo=try&revision=06e8fdaa2d2a61ce34ab37de4bae8951d818ede7

Comment 10

[Michael Shal [:mshal]]

Comment on attachment 8829646 [details]
Bug 1324210 - Don't attempt to check libstdcxx version in artifact builds.

https://reviewboard.mozilla.org/r/106654/#review107980

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3e10a2796cce
Don't attempt to check libstdcxx version in artifact builds. r=mshal

Comment 12

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/3e10a2796cce