1391424 - WindowsError: [Error 206] The filename or extension is too long: 'y:\\hg-shared\\8ba995b74e18334ab3707f27e9eb8f4e37ba3d29\\.hg\\data/mobile/android/tests/browser/chrome/tp5/amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/%5C%2

Status: RESOLVED FIXED

(Developer Services :: Mercurial: robustcheckout, defect)

Description

[Gregory Szorc [:gps]]

Seeing this in automation.

Z:\task_1503001569>"c:\Program Files\Mercurial\hg.exe" robustcheckout --sharebase y:\hg-shared --purge --upstream https://hg.mozilla.org/mozilla-unified --revision fc199105bfbe0d7ff774bf11c5150fae4e1b7c8e https://hg.mozilla.org/try .\src 

clone [                                         ]    2049359/2099589350 34m53s
...
ensuring https://hg.mozilla.org/try@fc199105bfbe0d7ff774bf11c5150fae4e1b7c8e is available at .\src
(cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
(sharing from existing pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/0cf94aae28707b28ff1fdd27448843af23136e0d.packed1.hg
381232 files to transfer, 1.96 GB of data
Traceback (most recent call last):
  File "mercurial\dispatch.pyc", line 204, in _runcatch
  File "mercurial\dispatch.pyc", line 880, in _dispatch
  File "mercurial\dispatch.pyc", line 637, in runcommand
  File "mercurial\dispatch.pyc", line 1010, in _runcommand
  File "mercurial\dispatch.pyc", line 971, in checkargs
  File "mercurial\dispatch.pyc", line 877, in <lambda>
  File "mercurial\util.pyc", line 1036, in check
  File "C:/mozilla-build/robustcheckout.py", line 176, in robustcheckout
  File "C:/mozilla-build/robustcheckout.py", line 315, in _docheckout
  File "mercurial\hg.pyc", line 496, in clone
  File "mercurial\hg.pyc", line 385, in clonewithshare
  File "mercurial\exchange.pyc", line 1183, in pull
  File "mercurial\exchange.pyc", line 1777, in _maybeapplyclonebundle
  File "mercurial\exchange.pyc", line 1919, in trypullbundlefromurl
  File "mercurial\streamclone.pyc", line 383, in apply
  File "mercurial\streamclone.pyc", line 371, in applybundlev1
  File "mercurial\streamclone.pyc", line 318, in consumev1
  File "mercurial\scmutil.pyc", line 649, in __call__
  File "mercurial\scmutil.pyc", line 582, in __call__
  File "mercurial\util.pyc", line 1580, in makedirs
  File "mercurial\win32.pyc", line 499, in makedir
WindowsError: [Error 206] The filename or extension is too long: 'y:\\hg-shared\\8ba995b74e18334ab3707f27e9eb8f4e37ba3d29\\.hg\\data/mobile/android/tests/browser/chrome/tp5/amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/%5C%22http%3A/g-ecx.images-amazon.com/images/G/01/kindle/shasta/photos'
abort: The filename or extension is too long: 'y:\hg-shared\8ba995b74e18334ab3707f27e9eb8f4e37ba3d29\.hg\data/mobile/android/tests/browser/chrome/tp5/amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/%5C%22http%3A/g-ecx.images-amazon.com/images/G/01/kindle/shasta/photos'
[taskcluster 2017-08-17T20:45:52.557Z]    Exit Code: 255
[taskcluster 2017-08-17T20:45:52.557Z] Success Code: 0x0
[taskcluster 2017-08-17T20:45:52.557Z]    User Time: 0s
[taskcluster 2017-08-17T20:45:52.557Z]  Kernel Time: 15.625ms
[taskcluster 2017-08-17T20:45:52.557Z]    Wall Time: 18m15.2639853s
[taskcluster 2017-08-17T20:45:52.557Z]  Peak Memory: 1699840
[taskcluster 2017-08-17T20:45:52.557Z]       Result: FAILED

Comment 1

[Gregory Szorc [:gps]]

What's wonky about this is that knows about long path problems and is supposed to use .hg/data/dh/ abbreviated paths. For whatever reason, those aren't being used here.

Mercurial out of the box doesn't have this problem. I suspect it is specific to robustcheckout.

Comment 2

[Gregory Szorc [:gps]]

Bonus wonkiness: it prints "sharing from existing pooled repository" but then applies a clone bundle. So the existing pooled repository must be empty???

I'm really curious how things got in this state.

Comment 3

[Gregory Szorc [:gps]]

Mercurial needs to use the fncachestore to get those .hg/store/dh abbreviated paths. The code for using that store is:

def store(requirements, path, vfstype):
    if 'store' in requirements:
        if 'fncache' in requirements:
            return fncachestore(path, vfstype, 'dotencode' in requirements)
        return encodedstore(path, vfstype)
    return basicstore(path, vfstype)

What I think may be happening is the "empty" repo that somehow comes into existence doesn't have a requirements file and we end up using basicstore. We can detect and work around this in robustcheckout.

Comment 4

[Gregory Szorc [:gps]]

Attachment: robustcheckout: add test for store lock and missing working directory (bug 1391424);

And it shows we time out waiting for the lock :(

Review commit: https://reviewboard.mozilla.org/r/169872/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/169872/

Comment 5

[Gregory Szorc [:gps]]

Attachment: robustcheckout: more robustly detect local store (bug 1391424);

As part of implementing more sanitization for the local store, I
discovered that we only examine the store repo if a working directory
is available. In many scenarios, we may have a store repo but no
working directory.

In this commit, we resolve the path to the store using the same
mechanism that Mercurial's pooled storage uses. This guarantees
that we'll find a local store even if we don't have a working
directory pointing to it.

As the locks test change shows, we now correctly detect a store
lock when there isn't a working directory.

This change does result in a redudant "lookup" wire protocol
command being issued. That's why we had to increase the request
counter in test-server-failure.t.

Review commit: https://reviewboard.mozilla.org/r/169874/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/169874/

Comment 6

[Gregory Szorc [:gps]]

Attachment: robustcheckout: add tests for repos with weird requirements (bug 1391424);

I'm pretty sure some shared store repos in automation are empty. The
.hg directory exists but it has nothing in it. Mercurial treats this
as a super legacy repo with a layout that predates the requires file.
This is the only way I can explain log output reported in this bug:

   ensuring https://hg.mozilla.org/try@fc199105bfbe0d7ff774bf11c5150fae4e1b7c8e is available at .\src
   (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
   (sharing from existing pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
   applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/0cf94aae28707b28ff1fdd27448843af23136e0d.packed1.hg

This is arguably a bug in Mercurial. But until it is fixed, it is
something we must deal with.

We add tests around an empty .hg directory and a shared repo lacking
modern repository requirements. The next commit will improve the
behavior for these scenarios.

Review commit: https://reviewboard.mozilla.org/r/169876/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/169876/

Comment 7

[Gregory Szorc [:gps]]

Attachment: robustcheckout: validate requirements in store repo (bug 1391424);

For reasons I can't explain, we're seeing empty repos or repos without
.hg/requires files in automation. I suspect some process is attempting
to clone a repo, fails, and the .hg directory is somehow orphaned.

When Mercurial sees an existing .hg directory without a "requires"
file, it assumes it is an ancient repository format and it happily
starts writing new data to it.

One of the features not in this ancient repository format is the
ability to shorten paths in the store to avoid long path length limits
on common filesystems and operating systems. e.g. the 260 max path
length for various APIs on Windows.

The missing features also make some repository operations slower.

We should never encounter these ancient repository formats in the
wild. So, this commit teaches robustcheckout to verify that an
existing shared repo has modern requirements. If it doesn't, we
blow away that share and the working directory.

Review commit: https://reviewboard.mozilla.org/r/169878/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/169878/

Comment 8

[Gregory Szorc [:gps]]

Attachment: robustcheckout: print Mercurial version (bug 1391424);

I've been wanting this in the logs for a while to help with
debugging.

Review commit: https://reviewboard.mozilla.org/r/169884/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/169884/

Comment 9

[:glob ✱]

Comment on attachment 8898516 [details]
robustcheckout: add test for store lock and missing working directory (bug 1391424);

https://reviewboard.mozilla.org/r/169872/#review176276

Comment 10

[:glob ✱]

Comment on attachment 8898517 [details]
robustcheckout: more robustly detect local store (bug 1391424);

https://reviewboard.mozilla.org/r/169874/#review176290

Comment 11

[:glob ✱]

Comment on attachment 8898518 [details]
robustcheckout: add tests for repos with weird requirements (bug 1391424);

https://reviewboard.mozilla.org/r/169876/#review176320

Comment 12

[:glob ✱]

Comment on attachment 8898519 [details]
robustcheckout: validate requirements in store repo (bug 1391424);

https://reviewboard.mozilla.org/r/169878/#review176326

Comment 13

[:glob ✱]

Comment on attachment 8898521 [details]
robustcheckout: print Mercurial version (bug 1391424);

https://reviewboard.mozilla.org/r/169884/#review176336

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b08e2ff87c0e
robustcheckout: add test for store lock and missing working directory ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9727ff75d72a
robustcheckout: more robustly detect local store ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/002b0b8280fb
robustcheckout: add tests for repos with weird requirements ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/1674576dacfb
robustcheckout: validate requirements in store repo ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/134574b64ddf
robustcheckout: print Mercurial version ; r=glob

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/68cdc2d1184d
Vendor robustcheckout with capabilities detection; r=glob

Comment 16

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0b6969f89a2a
Backed out changeset 68cdc2d1184d for mozharness test bustage