Closed Bug 789520 Opened 12 years ago Closed 12 years ago

mozharness: "Directory is not empty" in clobber self.rmtree step

Categories

(Release Engineering :: Applications: MozharnessCore, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: k0scist, Assigned: mozilla)

References

Details

(Whiteboard: [mozharness])

Attachments

(1 file, 1 obsolete file)

I don't even know how I actually got to this point.  For some reason,
running `python scripts\jetperf.py --config-file
configs\jetperf\test_config.py` in the same way as I've done many
times now gives me the error: "The directory is not empty". I have no
idea why now and not before.  Will post the output following.
Whiteboard: [mozharness]
(mozharness) C:\Users\jhammel\mozharness\src\mozharness>python scripts\jetperf.p
y --config-file configs\jetperf\test_config.py
On windows: {'default_actions': ['clobber', 'pull', 'build', 'create-virtualenv'
, 'test', 'baseline'], 'pywin32_url': 'http://superb-sea2.dl.sourceforge.net/pro
ject/pywin32/pywin32/Build216/pywin32-216.win32-py2.7.exe', 'virtualenv_modules'
: ['pywin32', 'talos', 'mozinstall'], 'datazilla_urls': ['file://C:\\Users\\jham
mel\\mozharness\\src\\mozharness\\20120907_1041.json']}
10:41:09     INFO - MultiFileLogger online at 20120907 10:41:09 in C:\Users\jham
mel\mozharness\src\mozharness
10:41:09     INFO - Run as scripts\jetperf.py --config-file configs\jetperf\test
_config.py
10:41:09     INFO - Dumping config to C:\Users\jhammel\mozharness\src\mozharness
\logs\localconfig.json.
10:41:09     INFO - {'addon-directories': [],
10:41:09     INFO -  'append_to_log': False,
10:41:09     INFO -  'base_work_dir': 'C:\\Users\\jhammel\\mozharness\\src\\mozh
arness',
10:41:09     INFO -  'config_file': 'configs\\jetperf\\test_config.py',
10:41:09     INFO -  'datazilla_urls': ['file://C:\\Users\\jhammel\\mozharness\\
src\\mozharness\\20120907_1041.json'],
10:41:09     INFO -  'default_actions': ['clobber',
10:41:09     INFO -                      'pull',
10:41:09     INFO -                      'build',
10:41:09     INFO -                      'create-virtualenv',
10:41:09     INFO -                      'test',
10:41:09     INFO -                      'baseline'],
10:41:09     INFO -  'log_level': 'info',
10:41:09     INFO -  'log_to_console': True,
10:41:09     INFO -  'pywin32_url': 'http://superb-sea2.dl.sourceforge.net/proje
ct/pywin32/pywin32/Build216/pywin32-216.win32-py2.7.exe',
10:41:09     INFO -  'repos': [{'dest': 'addon-sdk',
10:41:09     INFO -             'repo': 'http://hg.mozilla.org/projects/addon-sd
k',
10:41:09     INFO -             'revision': 'default'},
10:41:09     INFO -            {'dest': 'addons_clone',
10:41:09     INFO -             'repo': 'http://hg.mozilla.org/projects/addon-sd
k-jetperf-tests/',
10:41:09     INFO -             'revision': 'default'}],
10:41:09     INFO -  'talos_url': 'http://hg.mozilla.org/build/talos/archive/tip
.tar.gz',
10:41:09     INFO -  'tests': ['ts'],
10:41:09     INFO -  'virtualenv_modules': ['pywin32', 'talos', 'mozinstall'],
10:41:09     INFO -  'virtualenv_path': 'venv',
10:41:09     INFO -  'volatile_config': {'actions': None, 'add_actions': None, '
no_actions': None},
10:41:09     INFO -  'work_dir': 'build'}
10:41:09     INFO - #####
10:41:09     INFO - ##### Running clobber step.
10:41:09     INFO - #####
10:41:09     INFO - rmtree: C:\Users\jhammel\mozharness\src\mozharness\build
Traceback (most recent call last):
  File "scripts\jetperf.py", line 254, in <module>
    main()
  File "scripts\jetperf.py", line 251, in main
    jetperf.run()
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 679, in run
    self._possibly_run_method(method_name, error_if_missing=True)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 650, in _possibly_run_method
    return getattr(self, method_name)()
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 700, in clobber
    self.rmtree(dirs['abs_work_dir'])
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 64, in rmtree
    self._rmdir_recursive(path)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 105, in _rmdir_recursive
    self._rmdir_recursive(full_name)
  File "C:\Users\jhammel\mozharness\src\mozharness\mozharness\base\script.py", l
ine 110, in _rmdir_recursive
    os.rmdir(path)
WindowsError: [Error 145] The directory is not empty: 'C:\\Users\\jhammel\\mozha
rness\\src\\mozharness\\build\\addon-sdk\\.hg\\store\\data\\python-lib\\cuddlefi
sh\\tests\\bug-614712-files\\packages\\original-naming'
Do you have a job running or a command prompt / explorer window open in that directory?

There are reasons why we reboot our build+test slaves regularly; that may be the only fix if this persists.
No I don't on both counts. This is also right after a fresh boot.

I'll look at this in a bit.  I just wanted to note the issue while the error message was still in console.
`rmdir /S build` certainly fixed the issue, although I can't really say what the issue was.  Not sure if I should leave open?  I don't have STR though I'm guessing I could hit it again if I tried really hard
FWIW, I still see this issue on occasion, though I've been unable to isolate a pattern.  As per comment 4, rmdir /S "fixes"
We could potentially replace _rmdir_recursive() with a self.run_command([rmdir, "/S", ...]), or try the _rmdir_recursive() first and then pave over with the rmdir /S, or something along those lines.
Assignee: nobody → aki
Priority: -- → P3
Attached patch switch to rmdir /s (obsolete) — Splinter Review
Attachment #663193 - Flags: review?(jhammel)
Comment on attachment 663193 [details] [diff] [review]
switch to rmdir /s

This is fine though I can't say I'm a fan.  Eventually we will need a rmtree that works reliably on windows
Attachment #663193 - Flags: review?(jhammel) → review+
I'd love that too, but I think we're in a position to be pragmatic, not idealistic at this point.  Windows rm has been a pain throughout our production infrastructure for years.

http://hg.mozilla.org/build/mozharness/rev/fc115f1ac2b7
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Backed out due to bustage.  I need to add /Q, and it seems like windows doesn't like the list format.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attached patch try that againSplinter Review
rmdir /s /q "path", as a string instead of a list.

I was able to get this to work in staging.
Attachment #663193 - Attachment is obsolete: true
Attachment #663231 - Flags: review?(jhammel)
Attachment #663231 - Flags: review?(jhammel) → review+
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
See Also: → 888519
Product: mozilla.org → Release Engineering
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: