Intermittent abort: error applying bundle

RESOLVED FIXED

Status

defect
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: intermittent-bug-filer, Assigned: gps)

Tracking

(Blocks 1 bug, {intermittent-failure})

Details

Attachments

(4 attachments)

Greg, looks like this is an issue with the robust checkout?
Flags: needinfo?(gps)
The log says a connection to https://s3-external-1.amazonaws.com is failing. The only thing you can blame robustcheckout for is not retrying in case of this error.

I don't think I've seen HTTP requests to S3 fail like this before. First time for everything.
Flags: needinfo?(gps)
Oh, indeed, we had some similar issues with mozharness.zip failing around the same time (bug 1266624).  Probably a common cause.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WORKSFORME
Still a thing, but with lower frequency: https://public-artifacts.taskcluster.net/SZRfPJKlSau5A5s1ovxwAQ/0/public/logs/live_backing.log
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Looks like it's time for some retry in robustcheckout
Component: General → Mercurial: robustcheckout
Product: Taskcluster → Developer Services
Example log output:

[vcs 2017-04-03T18:17:33.613036Z] executing ['hg', 'robustcheckout', '--sharebase', '/home/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', 'caee52f2863b8c46da0a28d593d9efff2c09465a', 'https://hg.mozilla.org/integration/autoland/', '/home/worker/workspace/build/src']
[vcs 2017-04-03T18:17:33.676151Z] ensuring https://hg.mozilla.org/integration/autoland/@caee52f2863b8c46da0a28d593d9efff2c09465a is available at /home/worker/workspace/build/src
[vcs 2017-04-03T18:17:33.676279Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[vcs 2017-04-03T18:17:34.381987Z] (sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
[vcs 2017-04-03T18:19:42.179885Z] applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/66146205625406fd7eabd6b03c8b4c7ec464c3a1.packed1-gd.hg
[vcs 2017-04-03T18:21:49.415396Z] error fetching bundle: Connection timed out
[vcs 2017-04-03T18:21:49.417022Z] abort: error applying bundle
[vcs 2017-04-03T18:21:49.417073Z] (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")


I think this is a matter of adding that "Connection timed out" message to our list of errors to retry after encountering.
I'll code up some patches.
Assignee: nobody → gps
Status: REOPENED → ASSIGNED
Comment on attachment 8856701 [details]
robustcheckout: add test demonstrating connection failure (bug 1317594);

https://reviewboard.mozilla.org/r/128636/#review131264
Attachment #8856701 - Flags: review?(glob) → review+
Comment on attachment 8856702 [details]
robustcheckout: factor network failure handling into own function (bug 1317594);

https://reviewboard.mozilla.org/r/128638/#review131280
Attachment #8856702 - Flags: review?(glob) → review+
Comment on attachment 8856703 [details]
robustcheckout: refactor handlepullabort to handle multiple types (bug 1317594);

https://reviewboard.mozilla.org/r/128640/#review131422
Attachment #8856703 - Flags: review?(glob) → review+
Comment on attachment 8856704 [details]
robustcheckout: retry after socket errors (bug 1317594);

https://reviewboard.mozilla.org/r/128642/#review131424
Attachment #8856704 - Flags: review?(glob) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8318beaacec6
robustcheckout: add test demonstrating connection failure ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6f25918d6b2b
robustcheckout: factor network failure handling into own function ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/03efe6e45246
robustcheckout: refactor handlepullabort to handle multiple types ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e0d30b04dac6
robustcheckout: retry after socket errors ; r=glob
Status: ASSIGNED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → FIXED
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/987931d9e607
Vendor latest version of robustcheckout extension; r=me
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/249a47720ddc
robustcheckout: restore compatibility with Mercurial 3.7
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f124872fb221
Vendor latest robustcheckout extension; r=me
You need to log in before you can comment on or make changes to this bug.