Closed Bug 1611139 Opened 6 years ago Closed 3 years ago

Intermittent SystemError: unknown opcode

Categories

(Release Engineering :: Applications: MozharnessCore, defect, P5)

Product:
Release Engineering
Component:
Applications: MozharnessCore
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=286068686&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PS0lSRr9QKGN3GTSFQBazw/runs/0/artifacts/public/logs/live_backing.log

[task 2020-01-23T12:24:08.362Z] 12:24:08 INFO - Running setup.py bdist_wheel for idna: started
[task 2020-01-23T12:24:08.762Z] 12:24:08 INFO - Running setup.py bdist_wheel for idna: finished with status 'done'
[task 2020-01-23T12:24:08.763Z] 12:24:08 INFO - Stored in directory: C:\Users\task_1579779479\AppData\Local\pip\Cache\wheels\c2\75\0c\641d27d584aef4b02784f962777f146879d970f4cfdb8e7ec0
[task 2020-01-23T12:24:08.782Z] 12:24:08 INFO - Successfully built distro requests redo idna
[task 2020-01-23T12:24:08.785Z] 12:24:08 INFO - Installing collected packages: six, distro, chardet, idna, urllib3, certifi, requests, blessings, redo
[task 2020-01-23T12:24:10.048Z] 12:24:10 INFO - Successfully installed blessings-1.7 certifi-2019.11.28 chardet-3.0.4 distro-1.4.0 idna-2.8 redo-2.0.3 requests-2.22.0 six-1.13.0 urllib3-1.25.7
[task 2020-01-23T12:24:10.279Z] 12:24:10 INFO - Return code: 0
[task 2020-01-23T12:24:10.279Z] 12:24:10 INFO - Done creating virtualenv C:\Users\task_1579779479\build/venv.
[task 2020-01-23T12:24:10.279Z] 12:24:10 INFO - Getting output from command: ['C:\Users\task_1579779479\build\venv\Scripts\pip', 'freeze']
[task 2020-01-23T12:24:10.279Z] 12:24:10 INFO - Copy/paste: C:\Users\task_1579779479\build\venv\Scripts\pip freeze
[task 2020-01-23T12:24:10.280Z] XXX lineno: 1638, opcode: 36
[task 2020-01-23T12:24:10.280Z] 12:24:10 INFO - Running post-action listener: _resource_record_post_action
[task 2020-01-23T12:24:10.280Z] 12:24:10 INFO - Running post-action listener: _start_resource_monitoring
[task 2020-01-23T12:24:10.455Z] 12:24:10 INFO - Starting resource monitoring.
[task 2020-01-23T12:24:10.513Z] 12:24:10 INFO - [mozharness: 2020-01-23 12:24:10.513000Z] Finished create-virtualenv step (failed)
[task 2020-01-23T12:24:10.631Z] 12:24:10 FATAL - Uncaught exception: Traceback (most recent call last):
[task 2020-01-23T12:24:10.631Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\base\script.py", line 2132, in run
[task 2020-01-23T12:24:10.631Z] 12:24:10 FATAL - self.run_action(action)
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\base\script.py", line 2071, in run_action
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - self._possibly_run_method(method_name, error_if_missing=True)
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\base\script.py", line 2026, in _possibly_run_method
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - return getattr(self, method_name)()
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\mozilla\testing\raptor.py", line 635, in create_virtualenv
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - modules=['pip>=1.5']
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\base\python.py", line 436, in create_virtualenv
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - self.package_versions(log_output=True)
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\base\python.py", line 169, in package_versions
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - [pip, "freeze"], silent=True, ignore_errors=True)
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - File "C:\Users\task_1579779479\mozharness\mozharness\base\script.py", line 1638, in get_output_from_command
[task 2020-01-23T12:24:10.632Z] 12:24:10 FATAL - p = subprocess.Popen(command, shell=shell, stdout=tmp_stdout,
[task 2020-01-23T12:24:10.633Z] 12:24:10 FATAL - SystemError: unknown opcode
[task 2020-01-23T12:24:10.633Z] 12:24:10 FATAL - Running post_fatal callback...
[task 2020-01-23T12:24:10.633Z] 12:24:10 FATAL - Exiting -1
[task 2020-01-23T12:24:10.633Z] 12:24:10 INFO - Running post-run listener: _resource_record_post_run
[taskcluster 2020-01-23T12:24:12.019Z] Exit Code: 4294967295
[taskcluster 2020-01-23T12:24:12.019Z] User Time: 15.625ms
[taskcluster 2020-01-23T12:24:12.019Z] Kernel Time: 15.625ms
[taskcluster 2020-01-23T12:24:12.019Z] Wall Time: 3m23.8963849s
[taskcluster 2020-01-23T12:24:12.019Z] Result: FAILED
[taskcluster 2020-01-23T12:24:12.033Z] === Task Finished ===
[taskcluster 2020-01-23T12:24:12.033Z] Task Duration: 3m23.9326605s
[taskcluster 2020-01-23T12:24:12.490Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2021-01-21T22:00:22.876Z
[taskcluster 2020-01-23T12:24:12.892Z] Uploading artifact public/test_info/resource-usage.json from file build\blobber_upload_dir\resource-usage.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2021-01-21T22:00:22.876Z
[taskcluster 2020-01-23T12:24:13.121Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PS0lSRr9QKGN3GTSFQBazw/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2021-01-21T22:00:22.876Z
[taskcluster:error] exit status 4294967295

I think the relevant part is

[task 2020-01-23T12:24:10.279Z] 12:24:10 INFO - Getting output from command: ['C:\Users\task_1579779479\build\venv\Scripts\pip', 'freeze']
[task 2020-01-23T12:24:10.279Z] 12:24:10 INFO - Copy/paste: C:\Users\task_1579779479\build\venv\Scripts\pip freeze
[task 2020-01-23T12:24:10.280Z] XXX lineno: 1638, opcode: 36

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE

The last intermittent failure shows:

[task 2020-03-25T04:40:04.383Z] XXX lineno: 436, opcode: 110
[task 2020-03-25T04:40:04.384Z] 04:40:04     INFO - Running post-action listener: _resource_record_post_action
[task 2020-03-25T04:40:04.384Z] 04:40:04     INFO - Running post-action listener: _start_resource_monitoring
[task 2020-03-25T04:40:04.406Z] 04:40:04     INFO - Starting resource monitoring.
[task 2020-03-25T04:40:04.411Z] 04:40:04     INFO - [mozharness: 2020-03-25 04:40:04.410766Z] Finished create-virtualenv step (failed)

The same applies to all the other already classified failures. So it should have been filed into mozharness core.

Status: RESOLVED → REOPENED
Component: Mozbase → Applications: MozharnessCore
Product: Testing → Release Engineering
QA Contact: jlund
Resolution: INCOMPLETE → ---
Version: Version 3 → unspecified
Status: REOPENED → RESOLVED
Closed: 6 years ago → 3 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.