Permanent false-positive "OSError: [Errno 2] No such file or directory" in artifact builds

RESOLVED FIXED in Firefox 54

Status

()

Core
Build Config
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: aryx, Assigned: chmanchester)

Tracking

(Blocks: 1 bug)

unspecified
mozilla54
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox54 fixed)

Details

(Whiteboard: [stockwell fixed])

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

https://treeherder.mozilla.org/logviewer.html#?job_id=8044661&repo=autoland

[task 2016-12-17T00:15:38.612302Z] 00:15:38     INFO -  Refreshing /home/worker/workspace/build/src/old-configure with /usr/bin/autoconf-2.13
[task 2016-12-17T00:15:38.794935Z] 00:15:38     INFO -  creating cache ./config.cache
[task 2016-12-17T00:15:38.822560Z] 00:15:38     INFO -  checking host system type... x86_64-pc-linux-gnu
[task 2016-12-17T00:15:38.838128Z] 00:15:38     INFO -  checking target system type... x86_64-pc-linux-gnu
[task 2016-12-17T00:15:38.845636Z] 00:15:38     INFO -  checking build system type... x86_64-pc-linux-gnu
[task 2016-12-17T00:15:38.845812Z] 00:15:38     INFO -  checking for objcopy... /home/worker/workspace/build/src/gcc/bin/objcopy
[task 2016-12-17T00:15:39.030296Z] 00:15:39     INFO -  checking NSPR selection... source-tree
[task 2016-12-17T00:15:39.033638Z] 00:15:39     INFO -  checking if app-specific confvars.sh exists... /home/worker/workspace/build/src/browser/confvars.sh
[task 2016-12-17T00:15:39.053917Z] 00:15:39     INFO -  checking for gio-2.0 >= 2.22... yes
[task 2016-12-17T00:15:39.054113Z] 00:15:39     INFO -  checking MOZ_GIO_CFLAGS... -pthread -I/home/worker/workspace/build/src/gtk3/usr/local/include/glib-2.0 -I/home/worker/workspace/build/src/gtk3/usr/local/lib/glib-2.0/include
[task 2016-12-17T00:15:39.057450Z] 00:15:39     INFO -  checking MOZ_GIO_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/local/lib -lgio-2.0 -lgobject-2.0 -lglib-2.0
[task 2016-12-17T00:15:39.064917Z] 00:15:39     INFO -  checking for gconf-2.0 >= 1.2.1 gobject-2.0 ... yes
[task 2016-12-17T00:15:39.066375Z] 00:15:39     INFO -  checking MOZ_GCONF_CFLAGS... -DORBIT2=1 -pthread -I/home/worker/workspace/build/src/gtk3/usr/local/include/glib-2.0 -I/home/worker/workspace/build/src/gtk3/usr/local/lib/glib-2.0/include -I/home/worker/workspace/build/src/gtk3/usr/include/gconf/2 -I/home/worker/workspace/build/src/gtk3/usr/include/orbit-2.0 -I/home/worker/workspace/build/src/gtk3/usr/include/dbus-1.0 -I/home/worker/workspace/build/src/gtk3/usr/lib64/dbus-1.0/include
[task 2016-12-17T00:15:39.069681Z] 00:15:39     INFO -  checking MOZ_GCONF_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/local/lib -L/home/worker/workspace/build/src/gtk3/usr/lib64 -lgconf-2 -lgobject-2.0 -lglib-2.0
[task 2016-12-17T00:15:39.077107Z] 00:15:39     INFO -  checking for dbus-1 >= 0.60... yes
[task 2016-12-17T00:15:39.080588Z] 00:15:39     INFO -  checking MOZ_DBUS_CFLAGS... -I/home/worker/workspace/build/src/gtk3/usr/include/dbus-1.0 -I/home/worker/workspace/build/src/gtk3/usr/lib64/dbus-1.0/include
[task 2016-12-17T00:15:39.081812Z] 00:15:39     INFO -  checking MOZ_DBUS_LIBS... -L/home/worker/workspace/build/src/gtk3/lib64 -ldbus-1 -lpthread -lrt
[task 2016-12-17T00:15:39.085141Z] 00:15:39     INFO -  checking for dbus-glib-1 >= 0.60... yes
[task 2016-12-17T00:15:39.088648Z] 00:15:39     INFO -  checking MOZ_DBUS_GLIB_CFLAGS... -I/home/worker/workspace/build/src/gtk3/usr/local/include/glib-2.0 -I/home/worker/workspace/build/src/gtk3/usr/local/lib/glib-2.0/include -I/home/worker/workspace/build/src/gtk3/usr/include/dbus-1.0 -I/home/worker/workspace/build/src/gtk3/usr/lib64/dbus-1.0/include
[task 2016-12-17T00:15:39.089898Z] 00:15:39     INFO -  checking MOZ_DBUS_GLIB_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/local/lib -L/home/worker/workspace/build/src/gtk3/usr/lib64 -L/home/worker/workspace/build/src/gtk3/lib64 -ldbus-glib-1 -ldbus-1 -lpthread -lrt -lgobject-2.0 -lglib-2.0
[task 2016-12-17T00:15:39.131377Z] 00:15:39     INFO -  /home/worker/workspace/build/src/old-configure: line 14428: -E: command not found
[task 2016-12-17T00:15:39.132584Z] 00:15:39     INFO -  configure: warning: No assembler or assembly support for libvpx. Using unoptimized C routines.
[task 2016-12-17T00:15:39.148162Z] 00:15:39     INFO -  checking for libpulse... yes
[task 2016-12-17T00:15:39.148317Z] 00:15:39     INFO -  checking MOZ_PULSEAUDIO_CFLAGS... -D_REENTRANT
[task 2016-12-17T00:15:39.148464Z] 00:15:39     INFO -  checking MOZ_PULSEAUDIO_LIBS... -L/home/worker/workspace/build/src/gtk3/usr/lib64 -lpulse
[task 2016-12-17T00:15:39.184061Z] 00:15:39     INFO -  checking for linux/joystick.h... yes
[task 2016-12-17T00:15:39.185288Z] 00:15:39     INFO -  configure: warning: No assembler or assembly support for libav-fft.  Using unoptimized C routines.
[task 2016-12-17T00:15:39.192765Z] 00:15:39     INFO -  checking for wget... wget
[task 2016-12-17T00:15:39.196130Z] 00:15:39     INFO -  configure: warning: Cannot --enable-signmar with --disable-compile-environment
[task 2016-12-17T00:15:39.284152Z] 00:15:39     INFO -  Traceback (most recent call last):
[task 2016-12-17T00:15:39.284324Z] 00:15:39     INFO -    File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
[task 2016-12-17T00:15:39.284449Z] 00:15:39     INFO -      "__main__", fname, loader, pkg_name)
[task 2016-12-17T00:15:39.284580Z] 00:15:39     INFO -    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2016-12-17T00:15:39.284729Z] 00:15:39     INFO -      exec code in run_globals
[task 2016-12-17T00:15:39.284883Z] 00:15:39     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 79, in <module>
[task 2016-12-17T00:15:39.284993Z] 00:15:39     INFO -      print 'MOZ_LIBSTDCXX_TARGET_VERSION=%s' % find_version(cxx_env)
[task 2016-12-17T00:15:39.285131Z] 00:15:39     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 64, in find_version
[task 2016-12-17T00:15:39.285248Z] 00:15:39     INFO -      p = subprocess.Popen(args, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
[task 2016-12-17T00:15:39.285374Z] 00:15:39     INFO -    File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
[task 2016-12-17T00:15:39.285489Z] 00:15:39     INFO -      errread, errwrite)
[task 2016-12-17T00:15:39.285638Z] 00:15:39     INFO -    File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
[task 2016-12-17T00:15:39.285753Z] 00:15:39     INFO -      raise child_exception
[task 2016-12-17T00:15:39.285881Z] 00:15:39     INFO -  OSError: [Errno 2] No such file or directory

Comment 1

a year ago
12 failures in 722 pushes (0.017 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 4
* try: 3
* mozilla-inbound: 2
* mozilla-central: 2
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 9
* linux64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324210&startday=2017-01-09&endday=2017-01-15&tree=all
Beginning Jan 12, there are intermittent Android failures caused by failures to run netstat; I'll open a separate bug for that.
Depends on: 1331455
And if anyone wants to see artifact build failures correctly starred and filed, getting rid of this permanent "error" which appears in every single log, green or orange or red, would be a good idea.
Keywords: intermittent-failure
Summary: Intermittent artifact OSError: [Errno 2] No such file or directory → Permanent false-positive "OSError: [Errno 2] No such file or directory" in artifact builds
(Assignee)

Updated

a year ago
Blocks: 901840

Comment 4

a year ago
21 failures in 115 pushes (0.183 failures/push) were associated with this bug yesterday.  

Repository breakdown:
* mozilla-inbound: 21

Platform breakdown:
* linux64: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324210&startday=2017-01-19&endday=2017-01-19&tree=all

Comment 5

a year ago
33 failures in 690 pushes (0.048 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 25
* autoland: 5
* mozilla-central: 2
* try: 1

Platform breakdown:
* linux64: 23
* android-4-3-armv7-api15: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1324210&startday=2017-01-16&endday=2017-01-22&tree=all
(In reply to Phil Ringnalda (:philor) from comment #3)
> And if anyone wants to see artifact build failures correctly starred and
> filed, getting rid of this permanent "error" which appears in every single
> log, green or orange or red, would be a good idea.

I see: 

[task 2017-01-23T20:25:04.326179Z] 20:25:04     INFO -  Traceback (most recent call last):
[task 2017-01-23T20:25:04.326379Z] 20:25:04     INFO -    File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
[task 2017-01-23T20:25:04.326521Z] 20:25:04     INFO -      "__main__", fname, loader, pkg_name)
[task 2017-01-23T20:25:04.326682Z] 20:25:04     INFO -    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2017-01-23T20:25:04.326829Z] 20:25:04     INFO -      exec code in run_globals
[task 2017-01-23T20:25:04.326983Z] 20:25:04     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 79, in <module>
[task 2017-01-23T20:25:04.327137Z] 20:25:04     INFO -      print 'MOZ_LIBSTDCXX_TARGET_VERSION=%s' % find_version(cxx_env)
[task 2017-01-23T20:25:04.327287Z] 20:25:04     INFO -    File "/home/worker/workspace/build/src/python/mozbuild/mozbuild/configure/libstdcxx.py", line 64, in find_version
[task 2017-01-23T20:25:04.327430Z] 20:25:04     INFO -      p = subprocess.Popen(args, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
[task 2017-01-23T20:25:04.327564Z] 20:25:04     INFO -    File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
[task 2017-01-23T20:25:04.327710Z] 20:25:04     INFO -      errread, errwrite)
[task 2017-01-23T20:25:04.327856Z] 20:25:04     INFO -    File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
[task 2017-01-23T20:25:04.327984Z] 20:25:04     INFO -      raise child_exception
[task 2017-01-23T20:25:04.328120Z] 20:25:04     INFO -  OSError: [Errno 2] No such file or directory

It looks like libstdcxx.py is trying to launch os.environ['CXX']. I don't see CXX defined in the log, but I think it must be defined in order for libstdcxx.py to get as far as the Popen().
(Assignee)

Comment 7

a year ago
We just need to avoid doing that check in artifact builds, it was probably missed initially because it doesn't cause the build to fail.

I remember a bug where this was discussed, but I can't seem to find it now, so we can just fix it here I suppose!
Assignee: nobody → cmanchester
Comment hidden (mozreview-request)

Comment 10

a year ago
mozreview-review
Comment on attachment 8829646 [details]
Bug 1324210 - Don't attempt to check libstdcxx version in artifact builds.

https://reviewboard.mozilla.org/r/106654/#review107980
Attachment #8829646 - Flags: review?(mshal) → review+

Comment 11

a year ago
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3e10a2796cce
Don't attempt to check libstdcxx version in artifact builds. r=mshal

Comment 12

a year ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/3e10a2796cce
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox54: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
Whiteboard: [stockwell fixed]
You need to log in before you can comment on or make changes to this bug.