Closed Bug 1280365 Opened 8 years ago Closed 6 years ago

Intermittent tp6 IndexError: list index out of range for Collecting cffi==1.10.0 in mitmproxy_requirements.txt

Categories

(Testing :: Talos, defect)

Product:
Testing
Component:
Talos
Version:
Version 3
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox60 fixed, firefox61 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla61
Tracking Flags:
Tracking Status
firefox60 --- fixed
firefox61 --- fixed

People

(Reporter: aryx, Assigned: jmaher)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell fixed:other])

Attachments

(1 file)

reorder mitmproxy_requirements.txt
1.10 KB, patch
rwood
: review+
Details | Diff | Splinter Review 
https://treeherder.mozilla.org/logviewer.html#?job_id=30209859&repo=mozilla-inbound

01:12:03     INFO -  NightlyDebug.app/Contents/Resources/res/cursors/zoomOut@2x.png
01:12:03     INFO -  NightlyDebug.app/Contents/_CodeSignature/
01:12:03     INFO -  NightlyDebug.app/Contents/_CodeSignature/CodeResources
01:12:04     INFO -  sent 200190985 bytes  received 2808 bytes  8518884.81 bytes/sec
01:12:04     INFO -  total size is 200157211  speedup is 1.00
01:12:04     INFO -  "disk3" unmounted.
01:12:04     INFO -  "disk3" ejected.
01:12:05     INFO -  Process Process-1:
01:12:05     INFO -  Traceback (most recent call last):
01:12:05     INFO -    File "/tools/python27/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
01:12:05     INFO -      self.run()
01:12:05     INFO -    File "/tools/python27/lib/python2.7/multiprocessing/process.py", line 114, in run
01:12:05     INFO -      self._target(*self._args, **self._kwargs)
01:12:05     INFO -    File "/builds/slave/m-in-m64-d-0000000000000000000/build/src/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 100, in _collect
01:12:05     INFO -      io_diff = [v - io_last[i] for i, v in enumerate(io)]
01:12:05     INFO -  IndexError: list index out of range
...
01:16:52     INFO -  Error running mach:
01:16:52     INFO -      ['--log-no-times', 'build', '-v']
01:16:52     INFO -  The error occurred in code that was called by the mach command. This is either
01:16:52     INFO -  a bug in the called code itself or in the way that mach is calling it.
01:16:52     INFO -  You should consider filing a bug for this issue.
01:16:52     INFO -  If filing a bug, please include the full output of mach, including this error
01:16:52     INFO -  message.
01:16:52     INFO -  The details of the failure are as follows:
01:16:52     INFO -  AssertionError
01:16:52     INFO -    File "/builds/slave/m-in-m64-d-0000000000000000000/build/src/python/mozbuild/mozbuild/mach_commands.py", line 441, in build
01:16:52     INFO -      monitor.finish(record_usage=status==0)
01:16:52     INFO -    File "/builds/slave/m-in-m64-d-0000000000000000000/build/src/python/mozbuild/mozbuild/controller/building.py", line 246, in finish
01:16:52     INFO -      self.resources.stop()
01:16:52     INFO -    File "/builds/slave/m-in-m64-d-0000000000000000000/build/src/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 325, in stop
Summary: Intermittent list index out of range in resourcemonitor.py's _collect → Intermittent IndexError: list index out of range in resourcemonitor.py's _collect
In the last 7 days there are 30 failures.

This occurs mainly on Windows 10-64 talos-h2-e10s (opt and pgo) with a few occurrences on Linux 64 and Windows 7 (debug).

:aryx can you take a look at this?
Flags: needinfo?(aryx.bugmail)
Whiteboard: [stockwell needswork]
Recent failures look different from comment 0. This seems typical:

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=151761929&lineNumber=1578

17:36:06     INFO -      IndexError: list index out of range
17:36:06     INFO -      During handling of the above exception, another exception occurred:
17:36:06     INFO -      Traceback (most recent call last):
17:36:06     INFO -        File "<string>", line 1, in <module>
17:36:06     INFO -        File "C:\Users\cltbld\AppData\Local\Temp\pip-build-fwtzutc3\cffi\setup.py", line 113, in <module>
17:36:06     INFO -          if sys.platform == 'win32' and uses_msvc():
17:36:06     INFO -        File "C:\Users\cltbld\AppData\Local\Temp\pip-build-fwtzutc3\cffi\setup.py", line 91, in uses_msvc
17:36:06     INFO -          return config.try_compile('#ifndef _MSC_VER\n#error "not MSVC"\n#endif')
17:36:06     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\command\config.py", line 227, in try_compile
17:36:06     INFO -          self._compile(body, headers, include_dirs, lang)
17:36:06     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\command\config.py", line 133, in _compile
17:36:06     INFO -          self.compiler.compile([src], include_dirs=include_dirs)
17:36:06     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\_msvccompiler.py", line 304, in compile
17:36:06     INFO -          self.initialize()
17:36:06     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\_msvccompiler.py", line 197, in initialize
17:36:06     INFO -          vc_env = _get_vc_env(plat_spec)
17:36:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\msvc.py", line 172, in msvc14_get_vc_env
17:36:06     INFO -          return EnvironmentInfo(plat_spec, vc_min_ver=14.0).return_env()
17:36:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\msvc.py", line 771, in __init__
17:36:06     INFO -          self.si = SystemInfo(self.ri, vc_ver)
17:36:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\msvc.py", line 480, in __init__
17:36:06     INFO -          raise distutils.errors.DistutilsPlatformError(err)
17:36:06     INFO -      distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
17:36:06     INFO -      ----------------------------------------
17:36:06     INFO -  Command "python setup.py egg_info" failed with error code 1 in C:\Users\cltbld\AppData\Local\Temp\pip-build-fwtzutc3\cffi\
17:36:06    ERROR - Return code: 1
17:36:06     INFO - Installing mitmproxy

Since this is happening almost entirely in Talos jobs, let's update the component accordingly.

:aryx reported this bug, but I see no other connection -- let's see if someone on Talos can look at this.
Component: Mozbase → Talos
Flags: needinfo?(aryx.bugmail) → needinfo?(jmaher)
Flags: needinfo?(jmaher) → needinfo?(rwood)
Hmm it's like some mitmproxy package requirement is dependent upon MS VC++ 14.0. Something must have changed in the test env...

17:36:06     INFO -      distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
17:36:06     INFO -      ----------------------------------------
17:36:06     INFO -  Command "python setup.py egg_info" failed with error code 1 in C:\Users\cltbld\AppData\Local\Temp\pip-build-fwtzutc3\cffi\
17:36:06    ERROR - Return code: 1
I assume we need to fix this in the OCC:
https://github.com/mozilla-releng/OpenCloudConfig/blob/master/userdata/Manifest/gecko-t-win10-64-hw.json

as a note, this failure rate will drop dramatically as we have disabled the 'h2' job on windows10- we still should fix this though :)

but I don't know if that is used on our IX machines, :markco, can you help us figure out how to install MS VC++ 14.0 on the IX machines?
Flags: needinfo?(mcornmesser)
I can deploy it to the IX machines through the domain, as well as update the OCC manifest. Will this be only on hardware?
Flags: needinfo?(mcornmesser)
Flags: needinfo?(rwood)
Summary: Intermittent IndexError: list index out of range in resourcemonitor.py's _collect → Intermittent tp6 IndexError: list index out of range for Collecting cffi==1.10.0 in mitmproxy_requirements.txt
In the last 7 days we have 40 failures. They occur on windows7-32 (opt, pgo), windows10-64 (opt).

Recent failure log example: https://treeherder.mozilla.org/logviewer.html#?job_id=170563153&repo=autoland&lineNumber=1445

and a relevant part of it:

07:14:12     INFO -      IndexError: list index out of range
07:14:12     INFO -      During handling of the above exception, another exception occurred:
07:14:12     INFO -      Traceback (most recent call last):
07:14:12     INFO -        File "<string>", line 1, in <module>
07:14:12     INFO -        File "C:\Users\CLTBLD~1.T-W\AppData\Local\Temp\pip-build-kpgzkgpt\cffi\setup.py", line 113, in <module>
07:14:12     INFO -          if sys.platform == 'win32' and uses_msvc():
07:14:12     INFO -        File "C:\Users\CLTBLD~1.T-W\AppData\Local\Temp\pip-build-kpgzkgpt\cffi\setup.py", line 91, in uses_msvc
07:14:12     INFO -          return config.try_compile('#ifndef _MSC_VER\n#error "not MSVC"\n#endif')
07:14:12     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\command\config.py", line 227, in try_compile
07:14:12     INFO -          self._compile(body, headers, include_dirs, lang)
07:14:12     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\command\config.py", line 133, in _compile
07:14:12     INFO -          self.compiler.compile([src], include_dirs=include_dirs)
07:14:12     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\_msvccompiler.py", line 304, in compile
07:14:12     INFO -          self.initialize()
07:14:12     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\_msvccompiler.py", line 197, in initialize
07:14:12     INFO -          vc_env = _get_vc_env(plat_spec)
07:14:12     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\msvc.py", line 172, in msvc14_get_vc_env
07:14:12     INFO -          return EnvironmentInfo(plat_spec, vc_min_ver=14.0).return_env()
07:14:12     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\msvc.py", line 771, in __init__
07:14:12     INFO -          self.si = SystemInfo(self.ri, vc_ver)
07:14:12     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\msvc.py", line 480, in __init__
07:14:12     INFO -          raise distutils.errors.DistutilsPlatformError(err)
07:14:12     INFO -      distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
07:14:12     INFO -      ----------------------------------------
07:14:14     INFO -  Command "python setup.py egg_info" failed with error code 1 in C:\Users\CLTBLD~1.T-W\AppData\Local\Temp\pip-build-kpgzkgpt\cffi\
07:14:14    ERROR - Return code: 1
thanks for mentioning this Eliza!  We are planning to switch this week to new machines and OS installations which will fix this issue in bug 1431161
Depends on: 1431161
I found that moving some dependencies around I resolved the issues on try pushes.
a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=555864dad0452a6bdc7abae9322aeed0b0e67c42

this will not solve this bug 100%, but should be 95%+ solved
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8963128 - Flags: review?(rwood)
Comment on attachment 8963128 [details] [diff] [review]
reorder mitmproxy_requirements.txt

Review of attachment 8963128 [details] [diff] [review]:
-----------------------------------------------------------------

Cool!
Attachment #8963128 - Flags: review?(rwood) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9d08798cee62
Intermittent tp6 IndexError: list index out of range for Collecting cffi==1.10.0 in mitmproxy_requirements.txt - reorder installation of dependencies. r=rwood
https://hg.mozilla.org/mozilla-central/rev/9d08798cee62
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox61: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Whiteboard: [stockwell unknown] → [stockwell fixed:other][checkin-needed-beta]
https://hg.mozilla.org/releases/mozilla-beta/rev/525f37e3a399
status-firefox60: --- → fixed
Whiteboard: [stockwell fixed:other][checkin-needed-beta] → [stockwell fixed:other]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: