1119905 - hg purge hitting issues on windows

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

I've seen this a few times on Try when running hg purge:

09:28:50    ERROR -  abort: obj-firefox/dist/test-stage/web-platform/meta/webvtt/rendering/cues-with-video/processing-model/selectors/cue_function/underline_object/underline_with_class_object_specific_selector.html.ini cannot be removed

I suspect that there's some maximum path length problem being hit.

:gps suggests adding --traceback to all our hg invocations so that at least we get a stack trace when hg blows up.

Comment 1

[Chris AtLee [:catlee]]

This is not fatal; the build clobbers and re-clones from the local share, but it's still a waste of time.

Comment 2

[Gregory Szorc [:gps]]

Mercurial doesn't handle long paths properly. A proper fix is non-trivial because it dabbles into Unicode path handling, which is "fun" in the context of a program with cross-platform compatibility concerns.

There is an extension that claims to add largefile support: https://bitbucket.org/remleduff/win32lfn

The source is a bit scary (monkeypatching os.* functions). But it might be worth testing. If you install it, I would only activate it on hg invocations that can run into the file length issue. hg --config extensions.win32lfn=/path/to/ext/... purge

CCing a few Mercurial maintainers for an opinion here.

Comment 3

[Pierre-Yves David]

Relevant upstream bug: http://bz.selenic.com/show_bug.cgi?id=4488

Comment 4

[James Graham [:jgraham]]

So upstream here we impose a limit of 150 characters on the path length (excluding the .ini extension). This was required to get all the tests to run at all under Windows, and I thought that everything was therefore under the 260 character limit. How far off are we in this case (i.e. what's the full path here)? It is of course possible to rename some things upstream but doing that is a) ugly since it means that upstream is even more constrained by quirks of our environment and b) unfortunate since we lose history.

I don't suppose it's at all possible to make the prefixes shorter here? I mean we *know* that Windows is a length-constrained environment, so making mandatory parts of the path as short as possible should be encouraged.

Comment 5

[Chris AtLee [:catlee]]

I gave the win32lfn extension a shot on one of the machines that's failing. My first attempt was this code:
http://hg.mozilla.org/users/catlee_mozilla.com/tools/rev/7743d1c32b94

it tries the regular purge first, and then falls back to using purge with the win32lfn extension enabled.

This doesn't work out of the box because of the way we have hg deployed on windows. We have compiled version of hg, which resets the python module search path, so the win32lfn extension can't find the pywin32 extensions.

A few routes we could go:
- modify the win32lfn code to re-add the pywin32 modules to the search path
- deploy hg as a real python package, rather than a self-contained exe

Comment 6

[Gregory Szorc [:gps]]

I think you should deploy hg as a real Python package, not a py2exe executable. This will enable us to run more extensions, if needed. (Currently, we can't deploy extensions that use modules like json because they aren't present in the py2exe's stripped Python standard library.) This also saves you from having to modify upstream source code.