Closed Bug 895388 Opened 11 years ago Closed 10 years ago

Intermittent RuntimeError: Unable to get the disk's parent. | ERROR - Exception during post-run listener: Traceback (most recent call last): | AssertionError | FATAL - Aborting due to failure in post-run listener.

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: RyanVM, Assigned: gps)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Trap exception during resource monitor reporting
1.16 KB, patch
mozilla
: review+
Details | Diff | Splinter Review 
Looks related to the resource monitoring stuff.

https://tbpl.mozilla.org/php/getParsedLog.php?id=25385766&tree=Birch

Rev5 MacOSX Mountain Lion 10.8 birch opt test jsreftest on 2013-07-17 10:14:06 PDT for push 7acaefad07a2
slave: talos-mtnlion-r5-020

Process Process-1:
Traceback (most recent call last):
  File "/tools/python27/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
10:17:06     INFO - Reading from file tmpfile_stdout
    self.run()
  File "/tools/python27/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/mozsystemmonitor/resourcemonitor.py", line 47, in _collect
    io = psutil.disk_io_counters()
  File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/psutil/__init__.py", line 1211, in disk_io_counters
    rawdict = _psplatform.disk_io_counters()
RuntimeError: Unable to get the disk's parent.

10:23:19     INFO -  REFTEST TEST-START | Shutdown
10:23:19     INFO -  INFO | automation.py | Application ran for: 0:06:12.888239
10:23:19     INFO -  INFO | zombiecheck | Reading PID log: /var/folders/k1/llk26x893t14rvtt5b6c5rf000000w/T/tmpTIyJEPpidlog
10:23:19     INFO -  WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
10:23:19     INFO -  REFTEST INFO | runreftest.py | Running tests: end.
10:23:19     INFO - Return code: 0
10:23:19     INFO - TinderboxPrint: reftest-jsreftest<br/>57140/0/1351
10:23:19     INFO - # TBPL SUCCESS #
10:23:19     INFO - The reftest suite: jsreftest ran with return status: SUCCESS
10:23:19     INFO - Running post-action listener: _resource_record_post_action
10:23:19     INFO - Running post-run listener: _resource_record_post_run
10:23:20    ERROR - Exception during post-run listener: Traceback (most recent call last):
10:23:20    ERROR -   File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 1014, in run
10:23:20    ERROR -     method()
10:23:20    ERROR -   File "/builds/slave/talos-slave/test/scripts/mozharness/base/python.py", line 418, in _resource_record_post_run
10:23:20    ERROR -     self._resource_monitor.stop()
10:23:20    ERROR -   File "/builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages/mozsystemmonitor/resourcemonitor.py", line 255, in stop
10:23:20    ERROR -     assert done
10:23:20    ERROR - AssertionError
10:23:20    FATAL - Aborting due to failure in post-run listener.
10:23:20    FATAL - Exiting -1
program finished with exit code 255
Definitely a regression from resource monitoring.

Is this actually turning jobs non-green, or is it just log clutter?

It should be easy enough to trap this error. Although, I would like to get to the root cause - this seems like something that shouldn't be happening!

The underling cause appears to be a psutil issue. I wonder if it's fixed in a more modern psutil release.
Depends on: 859573
Flags: needinfo?(ryanvm)
It turns them red.
Flags: needinfo?(ryanvm)
This is the quickest fix. I'll open a bug against mozsystemmonitor to
dig into this.
Attachment #778648 - Flags: review?(aki)
Assignee: nobody → gps
Blocks: 896037
Attachment #778648 - Flags: review?(aki) → review+
Merged to production:

  778  hg pull -u
  779  hg up -r production
  780  hg merge -r default
  781  hg commit
  782  hg push
(In reply to Gregory Szorc [:gps] from comment #9)
> https://hg.mozilla.org/build/mozharness/rev/d3c72907b73c

See bug 900070 for the output this produced,.
Product: mozilla.org → Release Engineering
I have similiar issue as well. It happens after 1 build and hg pull -u

  File "e:\mozilla-central\python/mozbuild/mozbuild/mach_commands.py", line 305,
 in build
    monitor.init(warnings_path)
  File "e:\mozilla-central\python/mozbuild\mozbuild\controller\building.py", lin
e 278, in init
    self.resources = SystemResourceMonitor(poll_interval=1.0)
  File "e:\mozilla-central\testing/mozbase/mozsystemmonitor\mozsystemmonitor\res
ourcemonitor.py", line 179, in __init__
    io = psutil.disk_io_counters()
  File "e:\mozilla-central\python/psutil\psutil\__init__.py", line 1229, in disk
_io_counters
    raise RuntimeError("couldn't find any physical disk")
Closing inactive keywords:intermittent-failure bugs where the TBPLbot has previously commented and the test isn't marked as disabled; filter on orange-cleanup-201401.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
See Also: → 1800055
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: