1626581 - Use mozfile.remove in test_refresh_firefox.py and add the "\\?\" prefix for file path on Windows for mozfile.remove

Status: RESOLVED FIXED

(Core :: Storage: Quota Manager, task, P3)

Comment 1

[Tom Tung [:tt, :ttung]]

Attachment: Bug 1626581 - P1 - Use mozfile.remove on test_refresh_firefox;

Comment 2

[Tom Tung [:tt, :ttung]]

Attachment: Bug 1626581 - P2 - Use the prefix for mozfile.remove;

Comment 3

[Tom Tung [:tt, :ttung]]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=11dac849d269ad2864aa0d7b54116ce408028352&selectedJob=295731818

I think I am close to do the job but I'm not familir with Python and the code around. Therefore, move patches here and fix remaining issues for safty.

Comment 4

[Tom Tung [:tt, :ttung]]

I printed more log: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2a737f353a1ec37c87ec774eee4f4b476378a31f&selectedJob=295751969
I quote part of them below:

[task 2020-04-01T13:05:13.851Z] 13:05:13     INFO -  [TT]3 \\?\Z:\task_1585742307\build\application
[task 2020-04-01T13:05:13.851Z] 13:05:13    ERROR - Errors received:
[task 2020-04-01T13:05:13.851Z] 13:05:13     INFO - Reading from file tmpfile_stderr
[task 2020-04-01T13:05:13.851Z] 13:05:13    ERROR -  Traceback (most recent call last):
[task 2020-04-01T13:05:13.851Z] 13:05:13    ERROR -    File "Z:\task_1585742307\build\venv\Scripts\mozuninstall-script.py", line 9, in <module>
[task 2020-04-01T13:05:13.851Z] 13:05:13    ERROR -      load_entry_point('mozInstall==2.0.0', 'console_scripts', 'mozuninstall')()
[task 2020-04-01T13:05:13.851Z] 13:05:13    ERROR -    File "z:\task_1585742307\build\venv\lib\site-packages\mozinstall\mozinstall.py", line 374, in uninstall_cli
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      uninstall(argv[0])
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -    File "z:\task_1585742307\build\venv\lib\site-packages\mozinstall\mozinstall.py", line 252, in uninstall
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      mozfile.remove(install_folder)
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -    File "z:\task_1585742307\build\venv\lib\site-packages\mozfile\mozfile.py", line 250, in remove
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      _call_with_windows_retry(shutil.rmtree, (tmp_path,))
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -    File "z:\task_1585742307\build\venv\lib\site-packages\mozfile\mozfile.py", line 191, in _call_with_windows_retry
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      _call_windows_retry(*args, **kwargs)
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -    File "z:\task_1585742307\build\venv\lib\site-packages\mozfile\mozfile.py", line 149, in _call_windows_retry
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      func(*args)
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -    File "c:\mozilla-build\python\Lib\shutil.py", line 239, in rmtree
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      onerror(os.listdir, path, sys.exc_info())
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -    File "c:\mozilla-build\python\Lib\shutil.py", line 237, in rmtree
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -      names = os.listdir(path)
[task 2020-04-01T13:05:13.852Z] 13:05:13    ERROR -  WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\Z:\\task_1585742307\\build\\application/*.*'
[task 2020-04-01T13:05:13.852Z] 13:05:13     INFO - Using _rmtree_windows ...

I will need to find where /* was appended to the path.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hey Tom, will you be able to continue on this bug?

Comment 6

[Tom Tung [:tt, :ttung]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #5)

Hey Tom, will you be able to continue on this bug?

Hi Henrik,

It's still in my queue, but I might not be able to deal with that really soon. If you want to fix that soon, please feel free to take it.

The prefix was disabled in bug 1626846 and we are working on turning it on and then I will fix the follow-up bugs for bug 1536796 one by one (including this bug). And, currently, I am in bug 1626514.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks for your feedback. No, it's not pressing. It can wait for a bit.

Comment 8

[Tom Tung [:tt, :ttung]]

Attachment: Bug 1626581 - Use mozfile.remove in test_refresh_firefox.py and add the "\\?\" prefix for file path on Windows for mozfile.remove;

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/64456c63779a
Use mozfile.remove in test_refresh_firefox.py and add the "\\?\" prefix for file path on Windows for mozfile.remove; r=whimboo,gbrown

Comment 10

[Cristina Coroiu [:ccoroiu]]

Backed out changeset 64456c63779a (bug 1626581) for xpcshell task aborted failures

Backout: https://hg.mozilla.org/integration/autoland/rev/01e827a31a9f3c69d1f9be812192ad33a84aace5

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=64456c63779abeb5e391144dfcdbeea94143db66

Failure logs:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308486671&repo=autoland&lineNumber=81894
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308496197&repo=autoland&lineNumber=2039
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=308493503&repo=autoland&lineNumber=851

Comment 11

[Tom Tung [:tt, :ttung]]

Sorry, I will check xpcshell tests (I only checked the marrionette tests https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb018398a2caf95a38f52dac0b71b70360681be1)

Comment 12

[Tom Tung [:tt, :ttung]]

Hmm, I saw file paths like \\\\?\\c:\\users\\task_1593784908\\appdata\\local\\temp\\tmpsa4vpz/*.* after prepending the \\\\?\\.
The file path in this case should be \\\\?\\c:\\users\\task_1593784908\\appdata\\local\\temp\\tmpsa4vpz\\*.*.

I will check if the file path is like c:\\users\\task_1593784908\\appdata\\local\\temp\\tmpsa4vpz/*.* before the prepending or not first.

Comment 13

[Tom Tung [:tt, :ttung]]

I found https://stackoverflow.com/questions/29557760/long-paths-in-python-on-windows.

It seems that it's related to shutil.rmtree.
If the file path for a directory doesn't end with \\ for non-unicode version, it will append /*.*. For unicode version, append \*.*.

Comment 14

[Tom Tung [:tt, :ttung]]

making file path end with \\ doesn't seem to work for me. I use unicode instead.

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c2e2e76049f7f5d2996e4bac2b72c0f93edb4ae&selectedTaskRun=VUpt6iNgTVOzrubYfjfbrw.0

Comment 15

[Tom Tung [:tt, :ttung]]

One last try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ba82a23814695ff5a9e250d016b560b631e2ab2a

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/caa6d07ab019
Use mozfile.remove in test_refresh_firefox.py and add the "\\?\" prefix for file path on Windows for mozfile.remove; r=whimboo,gbrown

Comment 17

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/caa6d07ab019