There's an issue with the hg server pointing clone requests at bundles which aren't served. On a linux slave (not in mock) hg clone https://hg.mozilla.org/build/tools scripts in dir /builds/slave/tb-rel-c-beta-l64_rpk_8-000000/. (timeout 1320 secs) ... applying clone bundle from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/build/tools/b084b9ccbb62a825726ad2489dfd441208f648e9.packed1.hg warning: s3-us-west-2.amazonaws.com certificate with fingerprint 1a:0e:4a:64:90:c1:d0:2f:79:46:95:b5:17:dc:63:45:cf:19:37:bd not verified (check hostfingerprints or web.cacerts config setting) HTTP error fetching bundle: HTTP Error 403: Forbidden Traceback (most recent call last): File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 191, in _runcatch return _dispatch(req) ... File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/commands.py", line 1563, in clone shareopts=opts.get('shareopts')) File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/hg.py", line 595, in clone streamclonerequested=stream) File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/extensions.py", line 195, in closure return func(*(args + a), **kw) File "/usr/local/lib/hgext/bundleclone.py", line 590, in pull res = orig(repo, remote, *args, **kwargs) File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/exchange.py", line 1185, in pull _maybeapplyclonebundle(pullop) File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/exchange.py", line 1791, in _maybeapplyclonebundle hint=_('if this error persists, consider contacting ' Abort: error applying bundle abort: error applying bundle (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false") program finished with exit code 255 elapsedTime=0.864460 For me, hg v3.8.1: $ hg clone https://hg.mozilla.org/build/tools destination directory: tools applying clone bundle from https://hg.cdn.mozilla.net/build/tools/b084b9ccbb62a825726ad2489dfd441208f648e9.gzip.hg HTTP error fetching bundle: HTTP Error 403: Forbidden abort: error applying bundle (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
That revision is from May 11, and fubar says the objects are expiring from S3. The root cause is the bundle service died about a week ago. The errors are likely to spread to other trees rapidly.
the hg bundle creation systemd service apparently threw an exception last week and hadn't been running. thus, we ran into the 7 day expiry on the buckets. us-east-1 still had data, thankfully. disabled the policy and am syncing to the western regions. will run the bundle generator by hand after, so that things get updated. we need 1) to actually get errors for things like this, and 2) monitor the age of bundles so that this doesn't happen a third? fourth? time.
This should be fixed. If not, ping me on IRC and I'll kick off a bundle generation for an individual repo. fubar copied bundles to other S3 regions and deleted the expiration policy. The bundle generation service has been re-enabled and is running now. We'll fix up monitoring in bug 1274465.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(In reply to Gregory Szorc [:gps] from comment #3) > > fubar copied bundles to other S3 regions and deleted the expiration policy. expiration policy re-established.
You need to log in before you can comment on or make changes to this bug.