965270 - [mozversion] All unit tests fail on Windows because tempfile.mkstemp() keeps file open

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

test_binary (test_binary.BinaryTest) ... Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpfofy_x". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpfofy_x". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpfofy_x". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpfofy_x". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpfofy_x". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_binary_in_current_path (test_binary.BinaryTest) ... ok
test_invalid_binary_path (test_binary.BinaryTest) ... ok
test_missing_ini_files (test_binary.BinaryTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmplhgbck\application.ini
mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmplhgbck\platform.ini
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmplhgbck". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmplhgbck". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmplhgbck". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmplhgbck". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmplhgbck". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_invalid_sources_path (test_sources.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpyjlmvf\application.ini
mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpyjlmvf\platform.ini
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpyjlmvf". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpyjlmvf". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpyjlmvf". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpyjlmvf". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpyjlmvf". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_missing_sources_file (test_sources.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpzh0dip\application.ini
mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpzh0dip\platform.ini
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpzh0dip". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpzh0dip". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpzh0dip". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpzh0dip". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpzh0dip". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_sources (test_sources.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpjsogzz\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpjsogzz\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Error pulling gaia file
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpjsogzz". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpjsogzz". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpjsogzz". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpjsogzz". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpjsogzz". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_sources_in_current_directory (test_sources.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpnjse3p\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpnjse3p\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Error pulling gaia file
ok
test_gaia_commit (test_b2g.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpm52o2n\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpm52o2n\platform.ini
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpm52o2n". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpm52o2n". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpm52o2n". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpm52o2n". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpm52o2n". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_invalid_gaia_commit (test_b2g.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpubuiyt\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpubuiyt\platform.ini
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpubuiyt". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpubuiyt". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpubuiyt". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpubuiyt". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpubuiyt". Reason: The process cannot access the file because it is being used by another process (13)
ERROR
test_missing_gaia_commit (test_b2g.SourcesTest) ... mozversion.mozversion.LocalVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpqtikii\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Unable to find c:\users\mozauto\appdata\local\temp\tmpqtikii\platform.ini
mozversion.mozversion.LocalB2GVersion WARNING | Error pulling gaia file
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpqtikii". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpqtikii". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpqtikii". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpqtikii". Reason: The process cannot access the file because it is being used by another process (13)
Retrying to remove "c:\users\mozauto\appdata\local\temp\tmpqtikii". Reason: The process cannot access the file because it is being used by another process (13)
ERROR

======================================================================
ERROR: test_binary (test_binary.BinaryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_binary.py", line 37, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpfofy_x\\tmpmdbgd_'

======================================================================
ERROR: test_missing_ini_files (test_binary.BinaryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_binary.py", line 37, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmplhgbck\\tmpk_md9k'

======================================================================
ERROR: test_invalid_sources_path (test_sources.SourcesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_sources.py", line 32, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpyjlmvf\\tmplyxmdi'

======================================================================
ERROR: test_missing_sources_file (test_sources.SourcesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_sources.py", line 32, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpzh0dip\\tmpqc9wkv'

======================================================================
ERROR: test_sources (test_sources.SourcesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_sources.py", line 32, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpjsogzz\\tmpnr0c_i'

======================================================================
ERROR: test_gaia_commit (test_b2g.SourcesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_b2g.py", line 26, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpm52o2n\\tmplozzos'

======================================================================
ERROR: test_invalid_gaia_commit (test_b2g.SourcesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_b2g.py", line 26, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpubuiyt\\tmpmiqhdj'

======================================================================
ERROR: test_missing_gaia_commit (test_b2g.SourcesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\mozilla\code\mozbase\mozversion\tests\test_b2g.py", line 26, in tearDown
    mozfile.remove(self.tempdir)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 200, in remove
    _call_with_windows_retry(shutil.rmtree, path)
  File "c:\mozilla\code\mozbase\mozfile\mozfile\mozfile.py", line 154, in _call_with_windows_retry
    func(path)
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 250, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "C:\mozilla\mozmill-env\python\Lib\shutil.py", line 248, in rmtree
    os.remove(fullname)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\mozauto\\appdata\\local\\temp\\tmpqtikii\\tmp0ndtbc'

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As it looks like we are trying a couple of times but then bail out because the process has still not been quit.

Comment 2

[Andrew Halberstadt [:ahal]]

Why is this marked as blocking the mozbase merge? I don't see it anywhere in the try run:
https://tbpl.mozilla.org/?tree=Try&rev=842b31a7fa8d

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It was failing for me. If it works fine now, we should be good to remove it from the dependency list.

Comment 4

[Andrew Halberstadt [:ahal]]

Actually the new mozversion tests seem to crash on windows and then they hit this bug afterwards:
https://tbpl.mozilla.org/php/getParsedLog.php?id=34580469&tree=Try&full=1

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The problem here is actually the following:
http://www.logilab.org/blogentry/17873

So if you request a temporary file via mkstemp() you will have to close its handle via os.close().

I think instead of mkstemp() we could create another file on our own in the tempfolder. That wouldn't hurt us at all.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I can fix that tomorrow morning if it's not a pressure.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I'm going to fix that. Looks like the mozbase unittests never have been run on Windows during their implementation.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1

This fixes our problems...

Comment 10

[Andrew Halberstadt [:ahal]]

Comment on attachment 8375428 [details] [diff] [review]
Patch v1

Review of attachment 8375428 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed on master:
https://github.com/mozilla/mozbase/commit/62ca95898daeb67551580f4b0b26400578f911a9

I hope that was the last blocker for the merge! :)