1948871 - mach clobber fails because too long paths in obj-dir/gradle on Windows

Status: ASSIGNED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

> ./mach clobber;
Error running mach:

    mach clobber

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file clobber`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

FileNotFoundError: [WinError 3] The system cannot find the path specified: 'D:/gecko/obj-x86_64-unknown-linux-android/gradle\\build\\mobile\\android\\exoplayer2\\intermediates\\javac\\debug\\compileDebugJavaWithJavac\\classes\\org\\mozilla\\thirdparty\\com\\google\\android\\exoplayer2\\audio\\AudioCapabilitiesReceiver$ExternalSurroundSoundSettingObserver.class'

  File "D:\gecko\python\mozbuild\mozbuild\mach_commands.py", line 529, in clobber
    ).remove_objdir(full)
      ^^^^^^^^^^^^^^^^^^^
  File "d:\gecko\python\mozbuild\mozbuild\controller\clobber.py", line 162, in remove_objdir
    self.delete_dirs(self.topobjdir, paths)
  File "d:\gecko\python\mozbuild\mozbuild\controller\clobber.py", line 130, in delete_dirs
    mozfileremove(path)
  File "d:\gecko\testing\mozbase\mozfile\mozfile\mozfile.py", line 305, in remove
    _update_permissions(os.path.join(root, entry))
  File "d:\gecko\testing\mozbase\mozfile\mozfile\mozfile.py", line 265, in _update_permissions
    stats = os.stat(path)
            ^^^^^^^^^^^^^

Sentry event ID: 8eacbfae23f642e69241dbe24dad8b6e
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-Break to quit

That file certainly exists but fails to be removed, because of the default Windows path length limit. The solution is to change the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem's LongPathsEnabled value to 1, maybe the script can suggest that?

Comment 1

[Nick Alexander :nalexander [he/him]]

Adding a message somewhere sounds fine; not sure it should be in mozfile. It sounds like this might want to be in mach bootstrap on Windows as well.

Can this be worked around by deleting via relative paths or similar?

Comment 2

[Kagami Rosylight [:saschanaz] (they/them)]

I never tried that, although Microsoft docs say in https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-deletefilew :

By default, the name is limited to MAX_PATH characters. To extend this limit to 32,767 wide characters, prepend "\\?\" to the path.

Maybe worth trying?

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:polly, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Nick Alexander :nalexander [he/him]]

I'm pretty sure that I'm seeing this with a browser/ build --

nalexander@dama ~/Mozilla/gecko.git
$ ./mach clobber
Error running mach:

    mach clobber

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file clobber`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:/Users/nalexander/Mozilla/objdirs/objdir-browser-artifact/dist\\test-stage\\jsreftest\\tests\\js\\src\\tests\\test262\\built-ins\\Object\\seal\\object-seal-configurable-attribute-of-own-data-property-of-o-is-set-from-true-to-false-and-other-attributes-of-the-property-are-unaltered.js'

  File "C:\Users\nalexander\Mozilla\gecko.git\python\mozbuild\mozbuild\mach_commands.py", line 527, in clobber
    Clobberer(
  File "c:\users\nalexander\mozilla\gecko.git\python\mozbuild\mozbuild\controller\clobber.py", line 162, in remove_objdir
    self.delete_dirs(self.topobjdir, paths)
  File "c:\users\nalexander\mozilla\gecko.git\python\mozbuild\mozbuild\controller\clobber.py", line 130, in delete_dirs
    mozfileremove(path)
  File "c:\users\nalexander\mozilla\gecko.git\testing\mozbase\mozfile\mozfile\mozfile.py", line 305, in remove
    _update_permissions(os.path.join(root, entry))
  File "c:\users\nalexander\mozilla\gecko.git\testing\mozbase\mozfile\mozfile\mozfile.py", line 265, in _update_permissions
    stats = os.stat(path)

and that file does exist, it's just got a long name.

Comment 5

[Nick Alexander :nalexander [he/him]]

This is an oversight in which the long path accommodation expects c:\ but not c:/. It's easily fixed. See https://searchfox.org/mozilla-central/rev/cb46268bc26b0cd9e91e625aa92aaa5a6f047b9d/testing/mozbase/mozfile/mozfile/mozfile.py#280-291.

ahal: this looks busted in (vendored) taskgraph, too -- can you address or explain why it's not needed?

Comment 6

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1948871 - Fix `mach clobber` deleting long path names like `c:/...`. r?#build

Comment 7

[Andrew Halberstadt [:ahal]]

Thanks! The vendored run-task isn't used anywhere in Gecko, but I'll port the fix there anyway for other consumers.

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Taskgraph PR

Link to Taskgraph PR for posterity.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

There is an r+ patch which didn't land and no activity in this bug for 2 weeks.
:nalexander, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 10

[Nick Alexander :nalexander [he/him]]

I've landed this without a test; I'm not going to spend the time right now.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/82796a3084f2
https://hg.mozilla.org/integration/autoland/rev/740923aface1
Fix `mach clobber` deleting long path names like `c:/...`. r=mozbase-reviewers,ahal

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9e35893aea22
https://hg.mozilla.org/integration/autoland/rev/259542db5ae8
Revert "Bug 1948871 - Fix `mach clobber` deleting long path names like `c:/...`. r=mozbase-reviewers,ahal" for causing py3 failures @ test_clobber.py

Comment 13

[amarc]

Backed out for causing py3 fails @ test_clobber

• Backout link
• Push with failures
• Failure Log