Intermittent ImportError: No module named mozsystemmonitor.resourcemonitor

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
11 months ago
a month ago

People

(Reporter: aryx, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [stockwell unknown])

https://treeherder.mozilla.org/logviewer.html#?job_id=102550183&repo=autoland

09:22:03     INFO - Fetch https://queue.taskcluster.net/v1/task/KZ5Et2QORNCOm3ERAAePdA/artifacts/public/build/target.crashreporter-symbols.zip into memory
09:22:04     INFO - Content-Length response header: 67759832
09:22:04     INFO - Bytes received: 67759832
09:22:09     INFO - Running post-action listener: _resource_record_post_action
09:22:09     INFO - Running post-action listener: set_extra_try_arguments
09:22:09     INFO - [mozharness: 2017-05-27 09:22:09.824000Z] Finished download-and-extract step (success)
09:22:09     INFO - [mozharness: 2017-05-27 09:22:09.824000Z] Running create-virtualenv step.
09:22:09     INFO - Running pre-action listener: _install_mozbase
09:22:09     INFO - Running pre-action listener: _pre_create_virtualenv
09:22:09     INFO - Running pre-action listener: _resource_record_pre_action
09:22:09     INFO - Running main action method: create_virtualenv
09:22:09     INFO - Creating virtualenv Z:\task_1495876836\build\venv
09:22:09     INFO - mkdir: Z:\task_1495876836\build\venv\Scripts
09:22:09     INFO - Copying c:\mozilla-build\python\python27.dll to Z:\task_1495876836\build\venv\Scripts\python27.dll
09:22:09     INFO - Running command: ['c:\\mozilla-build\\python\\python.exe', 'c:\\mozilla-build\\python\\Lib\\site-packages\\virtualenv.py', '--no-site-packages', '--distribute', 'Z:\\task_1495876836\\build\\venv'] in Z:\task_1495876836\build
09:22:09     INFO - Copy/paste: c:\mozilla-build\python\python.exe c:\mozilla-build\python\Lib\site-packages\virtualenv.py --no-site-packages --distribute Z:\task_1495876836\build\venv
09:22:13     INFO -  New python executable in Z:\task_1495876836\build\venv\Scripts\python.exe
09:22:23     INFO -  Installing setuptools, pip, wheel...done.
09:22:23     INFO - Return code: 0
09:22:23     INFO - Getting output from command: ['Z:\\task_1495876836\\build\\venv\\Scripts\\pip', '--version']
09:22:23     INFO - Copy/paste: Z:\task_1495876836\build\venv\Scripts\pip --version
09:22:23     INFO - Running post-action listener: _resource_record_post_action
09:22:23     INFO - Running post-action listener: _start_resource_monitoring
09:22:23  WARNING - Unable to start resource monitor: Traceback (most recent call last):
09:22:23  WARNING -   File "Z:\task_1495876836\mozharness\mozharness\base\python.py", line 583, in _start_resource_monitoring
09:22:23  WARNING -     from mozsystemmonitor.resourcemonitor import SystemResourceMonitor
09:22:23  WARNING - ImportError: No module named mozsystemmonitor.resourcemonitor
09:22:23     INFO - [mozharness: 2017-05-27 09:22:23.380000Z] Finished create-virtualenv step (failed)
09:22:23    FATAL - Uncaught exception: Traceback (most recent call last):
09:22:23    FATAL -   File "Z:\task_1495876836\mozharness\mozharness\base\script.py", line 2066, in run
09:22:23    FATAL -     self.run_action(action)
09:22:23    FATAL -   File "Z:\task_1495876836\mozharness\mozharness\base\script.py", line 2005, in run_action
09:22:23    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
09:22:23    FATAL -   File "Z:\task_1495876836\mozharness\mozharness\base\script.py", line 1945, in _possibly_run_method
09:22:23    FATAL -     return getattr(self, method_name)()
09:22:23    FATAL -   File "Z:\task_1495876836\mozharness\mozharness\base\python.py", line 436, in create_virtualenv
09:22:23    FATAL -     halt_on_failure=True)
09:22:23    FATAL -   File "Z:\task_1495876836\mozharness\mozharness\base\script.py", line 1563, in get_output_from_command
09:22:23    FATAL -     cwd=cwd, stderr=tmp_stderr, env=env)
09:22:23    FATAL -   File "c:\mozilla-build\python\lib\subprocess.py", line 710, in __init__
09:22:23    FATAL -     errread, errwrite)
09:22:23    FATAL -   File "c:\mozilla-build\python\lib\subprocess.py", line 958, in _execute_child
09:22:23    FATAL -     startupinfo)
09:22:23    FATAL - WindowsError: [Error 5] Access is denied
09:22:23    FATAL - Running post_fatal callback...

Comment 1

11 months ago
7 failures in 891 pushes (0.008 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-central: 2
* mozilla-inbound: 1

Platform breakdown:
* windows7-32-vm: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-05-22&endday=2017-05-28&tree=all

Comment 2

11 months ago
12 failures in 820 pushes (0.015 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 6
* mozilla-central: 5
* mozilla-inbound: 1

Platform breakdown:
* windows7-32-vm: 10
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-05-29&endday=2017-06-04&tree=all

Comment 3

10 months ago
7 failures in 864 pushes (0.008 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 3
* autoland: 2
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows7-32-vm: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 4

10 months ago
8 failures in 814 pushes (0.01 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 2
* mozilla-central: 2

Platform breakdown:
* windows7-32-vm: 7
* linux64-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 5

10 months ago
2 failures in 892 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windows7-32-vm: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 6

10 months ago
10 failures in 718 pushes (0.014 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 5
* autoland: 3
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows7-32-vm: 9
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 7

9 months ago
3 failures in 656 pushes (0.005 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 2
* autoland: 1

Platform breakdown:
* windows7-32-vm: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 8

9 months ago
5 failures in 720 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* oak: 1
* mozilla-esr52: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32-vm: 2
* windows7-32: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 9

9 months ago
7 failures in 822 pushes (0.009 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 6
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 10

9 months ago
3 failures in 1008 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* windows7-32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-07-24&endday=2017-07-30&tree=all

Comment 11

9 months ago
1 failures in 888 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 12

8 months ago
5 failures in 901 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 3
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-08-07&endday=2017-08-13&tree=all

Comment 13

8 months ago
7 failures in 949 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 4
* mozilla-inbound: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 5
* windows7-32-nightly: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-08-14&endday=2017-08-20&tree=all

Comment 14

8 months ago
3 failures in 908 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 1
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windows7-32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-08-21&endday=2017-08-27&tree=all

Comment 15

8 months ago
14 failures in 939 pushes (0.015 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 7
* mozilla-central: 5
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 5
* windows7-32-stylo: 4
* osx-10-10: 2
* macosx64-stylo: 1
* linux64-stylo: 1
* linux32-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 16

7 months ago
10 failures in 924 pushes (0.011 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 5
* try: 2
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 3
* osx-10-10: 3
* linux32: 2
* windows7-32-stylo: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 17

7 months ago
6 failures in 1032 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 2
* autoland: 2
* mozilla-inbound: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 3
* osx-10-10: 2
* windows7-32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-09-11&endday=2017-09-17&tree=all

Comment 18

7 months ago
6 failures in 943 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* mozilla-central: 2
* try: 1
* autoland: 1

Platform breakdown:
* osx-10-10: 3
* windows7-32-stylo-disabled: 2
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-09-18&endday=2017-09-24&tree=all

Comment 19

7 months ago
26 failures in 885 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 17
* mozilla-inbound: 6
* oak: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 15
* windows7-32-stylo-disabled: 10
* windows7-32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-09-25&endday=2017-10-01&tree=all
we have 44 instances in the last week,almost all on windows 7.

:catlee, can you help find someone to look at this high frequency failure?
Flags: needinfo?(catlee)
Whiteboard: [stockwell needswork]
glandium, you're more familiar with mozsystemmonitor. any ideas?
Flags: needinfo?(catlee) → needinfo?(mh+mozilla)
Whatever is causing the job to fail, it's not mozsystemmonitor.

The code is:
        try:
            from mozsystemmonitor.resourcemonitor import SystemResourceMonitor

            self.info("Starting resource monitoring.")
            self._resource_monitor = SystemResourceMonitor(poll_interval=1.0)
            self._resource_monitor.start()
        except Exception:
            self.warning("Unable to start resource monitor: %s" %
                         traceback.format_exc())

IOW, the WARNING lines are all output from that except branch. Everything that follows is normal execution with that exception ignored. That's what fails, and what the traceback says is that it's:

        output = self.get_output_from_command([pip, '--version'],
                                              halt_on_failure=True)

that fails.

09:22:23    FATAL - WindowsError: [Error 5] Access is denied

suggests pip is not readable/executable/whatever.
Flags: needinfo?(mh+mozilla)

Comment 23

6 months ago
46 failures in 824 pushes (0.056 failures/push) were associated with this bug in the last 7 days. 

This is the #47 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 22
* mozilla-inbound: 15
* mozilla-beta: 6
* try: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 29
* windows7-32-stylo-disabled: 11
* windows7-32-nightly: 2
* windows7-32-devedition: 1
* osx-10-10: 1
* macosx64-stylo-disabled: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-10-02&endday=2017-10-08&tree=all
:catlee, is this something that the relops team would look at or maybe the taskcluster team?  The failure rate is remaining high.
Flags: needinfo?(catlee)
Greg, any ideas? Looks like this is happening on the TC win7 instances.
Flags: needinfo?(catlee) → needinfo?(garndt)

Comment 26

6 months ago
Rob, do you know what could cause pip to not be readable/executable? (see comment 22)

Could this be related to the access denied messages we saw before with hg-shared?
Flags: needinfo?(garndt) → needinfo?(rthijssen)
sounds very similar to the hg problem. investigating now...

Comment 28

6 months ago
50 failures in 947 pushes (0.053 failures/push) were associated with this bug in the last 7 days. 

This is the #44 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 23
* autoland: 18
* try: 3
* mozilla-central: 3
* mozilla-beta: 2
* mozilla-esr52: 1

Platform breakdown:
* windows7-32: 33
* windows7-32-stylo-disabled: 12
* osx-10-10: 2
* windows7-32-nightly: 1
* macosx64-stylo-disabled: 1
* android-5-0-aarch64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-10-09&endday=2017-10-15&tree=all
i believe the problem (on Windows 7) was caused by the missing Y: drive. On tc win instances the PIP_DOWNLOAD_CACHE environment variable is set to Y:\pip-cache and appropriate permissions are set on that directory (but only if creation of that directory was successful). On windows 7 we had a problem with the script which mounts the Y: drive (now fixed), while the Y: drive didn't exist, the env var wasn't being set causing pip calls to fall back to the buildbot default pip directory at c:\builds\pip_cache. Since this directory didn't have any permissions set, i think this was the cause of the access errors.

the errors should have already stopped occuring on win7 since the y: drive fix last week.
https://github.com/mozilla-releng/OpenCloudConfig/commit/28dcf8c9d4370aae5b138382da085f48b7ad8469
Flags: needinfo?(rthijssen)
they errors have gone way down, thanks for getting this fix.  Since we do still have errors and they are not all on win7, we either need to keep this bug open, or duplicate it so we can track the new failures.

Comment 31

6 months ago
14 failures in 864 pushes (0.016 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 8
* autoland: 3
* try: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 5
* windows10-64: 3
* windows10-64-stylo-disabled: 2
* osx-10-10: 2
* windows7-32-stylo-disabled: 1
* macosx64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-10-16&endday=2017-10-22&tree=all
Whiteboard: [stockwell needswork] → [stockwell unknown]

Comment 32

6 months ago
3 failures in 857 pushes (0.004 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* windows10-64-stylo-disabled: 1
* windows10-64: 1
* linux32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 33

5 months ago
27 failures in 849 pushes (0.032 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 12
* try: 10
* mozilla-inbound: 4
* autoland: 1

Platform breakdown:
* osx-10-10: 15
* macosx64-stylo-disabled: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 34

4 months ago
1 failures in 590 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 35

3 months ago
1 failures in 701 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2018-01-22&endday=2018-01-28&tree=all
Depends on: 1433851

Comment 36

2 months ago
1 failures in 735 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2018-01-29&endday=2018-02-04&tree=all
should now be fixed with the same patch as was used for bug 1343049 where we now prevent generic worker from starting when the user environment is incomplete
Status: NEW → RESOLVED
Last Resolved: 2 months ago
Resolution: --- → FIXED

Comment 38

a month ago
1 failures in 814 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* try: 1

Platform breakdown:
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1368241&startday=2018-03-12&endday=2018-03-18&tree=all
You need to log in before you can comment on or make changes to this bug.