1274456 - Cloning build/tools yields 403 errors from S3 bundles

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

There's an issue with the hg server pointing clone requests at bundles which aren't served.

On a linux slave (not in mock)
hg clone https://hg.mozilla.org/build/tools scripts
 in dir /builds/slave/tb-rel-c-beta-l64_rpk_8-000000/. (timeout 1320 secs)
...
applying clone bundle from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/build/tools/b084b9ccbb62a825726ad2489dfd441208f648e9.packed1.hg
warning: s3-us-west-2.amazonaws.com certificate with fingerprint 1a:0e:4a:64:90:c1:d0:2f:79:46:95:b5:17:dc:63:45:cf:19:37:bd not verified (check hostfingerprints or web.cacerts config setting)
HTTP error fetching bundle: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 191, in _runcatch
    return _dispatch(req)
...
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/commands.py", line 1563, in clone
    shareopts=opts.get('shareopts'))
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/hg.py", line 595, in clone
    streamclonerequested=stream)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/extensions.py", line 195, in closure
    return func(*(args + a), **kw)
  File "/usr/local/lib/hgext/bundleclone.py", line 590, in pull
    res = orig(repo, remote, *args, **kwargs)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/exchange.py", line 1185, in pull
    _maybeapplyclonebundle(pullop)
  File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/exchange.py", line 1791, in _maybeapplyclonebundle
    hint=_('if this error persists, consider contacting '
Abort: error applying bundle
abort: error applying bundle
(if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
program finished with exit code 255
elapsedTime=0.864460

For me, hg v3.8.1:
$ hg clone https://hg.mozilla.org/build/tools
destination directory: tools
applying clone bundle from https://hg.cdn.mozilla.net/build/tools/b084b9ccbb62a825726ad2489dfd441208f648e9.gzip.hg
HTTP error fetching bundle: HTTP Error 403: Forbidden
abort: error applying bundle
(if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

That revision is from May 11, and fubar says the objects are expiring from S3. The root cause is the bundle service died about a week ago. The errors are likely to spread to other trees rapidly.

Comment 2

[Kendall Libby [:fubar] (he/him)]

the hg bundle creation systemd service apparently threw an exception last week and hadn't been running. thus, we ran into the 7 day expiry on the buckets.

us-east-1 still had data, thankfully. disabled the policy and am syncing to the western regions. will run the bundle generator by hand after, so that things get updated.

we need 1) to actually get errors for things like this, and 2) monitor the age of bundles so that this doesn't happen a third? fourth? time.

Comment 3

[Gregory Szorc [:gps]]

This should be fixed. If not, ping me on IRC and I'll kick off a bundle generation for an individual repo.

fubar copied bundles to other S3 regions and deleted the expiration policy. The bundle generation service has been re-enabled and is running now. We'll fix up monitoring in bug 1274465.

Comment 4

[Kendall Libby [:fubar] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #3)
>
> fubar copied bundles to other S3 regions and deleted the expiration policy.

expiration policy re-established.