Intermittent abort: error applying bundle

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: intermittent-bug-filer, Assigned: gps)

Tracking

(Blocks: 1 bug, {intermittent-failure})

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(4 attachments)

Greg, looks like this is an issue with the robust checkout?
Flags: needinfo?(gps)
(Assignee)

Comment 2

2 years ago
The log says a connection to https://s3-external-1.amazonaws.com is failing. The only thing you can blame robustcheckout for is not retrying in case of this error.

I don't think I've seen HTTP requests to S3 fail like this before. First time for everything.
Flags: needinfo?(gps)
Oh, indeed, we had some similar issues with mozharness.zip failing around the same time (bug 1266624).  Probably a common cause.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
5 failures in 715 pushes (0.007 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* try: 1

Platform breakdown:
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2016-11-14&endday=2016-11-20&tree=all
Still a thing, but with lower frequency: https://public-artifacts.taskcluster.net/SZRfPJKlSau5A5s1ovxwAQ/0/public/logs/live_backing.log
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Looks like it's time for some retry in robustcheckout
Component: General → Mercurial: robustcheckout
Product: Taskcluster → Developer Services
6 failures in 526 pushes (0.011 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 5
* autoland: 1

Platform breakdown:
* linux64: 3
* android-4-0-armv7-api15: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2016-12-12&endday=2016-12-18&tree=all
9 failures in 690 pushes (0.013 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 5
* graphics: 2
* mozilla-inbound: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 7
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-01-16&endday=2017-01-22&tree=all
8 failures in 733 pushes (0.011 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* try: 1
* mozilla-central: 1
* mozilla-beta: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 7
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-01-30&endday=2017-02-05&tree=all
12 failures in 836 pushes (0.014 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 5
* mozilla-inbound: 4
* try: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 7
* android-4-0-armv7-api15: 2
* windows2012-64: 1
* linux64-qr: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-02-06&endday=2017-02-12&tree=all
11 failures in 833 pushes (0.013 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-inbound: 5
* autoland: 5
* try: 1

Platform breakdown:
* linux64: 5
* linux32: 3
* android-4-2-x86: 2
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-02-13&endday=2017-02-19&tree=all
17 failures in 783 pushes (0.022 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 7
* try: 3
* graphics: 3
* mozilla-central: 2
* mozilla-release: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 12
* linux32: 4
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-02-27&endday=2017-03-05&tree=all
16 failures in 777 pushes (0.021 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 9
* mozilla-inbound: 4
* mozilla-central: 1
* mozilla-beta: 1
* autoland: 1

Platform breakdown:
* linux64: 11
* linux32: 2
* windows7-32-vm: 1
* windows10-64-vm: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-03-13&endday=2017-03-19&tree=all
5 failures in 845 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 4
* autoland: 1

Platform breakdown:
* linux32: 4
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-03-27&endday=2017-04-02&tree=all
5 failures in 867 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 3
* try: 2

Platform breakdown:
* osx-10-7: 2
* android-4-2-x86: 2
* linux64-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-04-03&endday=2017-04-09&tree=all
(Assignee)

Comment 16

a year ago
Example log output:

[vcs 2017-04-03T18:17:33.613036Z] executing ['hg', 'robustcheckout', '--sharebase', '/home/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', 'caee52f2863b8c46da0a28d593d9efff2c09465a', 'https://hg.mozilla.org/integration/autoland/', '/home/worker/workspace/build/src']
[vcs 2017-04-03T18:17:33.676151Z] ensuring https://hg.mozilla.org/integration/autoland/@caee52f2863b8c46da0a28d593d9efff2c09465a is available at /home/worker/workspace/build/src
[vcs 2017-04-03T18:17:33.676279Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[vcs 2017-04-03T18:17:34.381987Z] (sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
[vcs 2017-04-03T18:19:42.179885Z] applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/66146205625406fd7eabd6b03c8b4c7ec464c3a1.packed1-gd.hg
[vcs 2017-04-03T18:21:49.415396Z] error fetching bundle: Connection timed out
[vcs 2017-04-03T18:21:49.417022Z] abort: error applying bundle
[vcs 2017-04-03T18:21:49.417073Z] (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")


I think this is a matter of adding that "Connection timed out" message to our list of errors to retry after encountering.
(Assignee)

Comment 17

a year ago
I'll code up some patches.
Assignee: nobody → gps
Status: REOPENED → ASSIGNED
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 22

a year ago
mozreview-review
Comment on attachment 8856701 [details]
robustcheckout: add test demonstrating connection failure (bug 1317594);

https://reviewboard.mozilla.org/r/128636/#review131264
Attachment #8856701 - Flags: review?(glob) → review+

Comment 23

a year ago
mozreview-review
Comment on attachment 8856702 [details]
robustcheckout: factor network failure handling into own function (bug 1317594);

https://reviewboard.mozilla.org/r/128638/#review131280
Attachment #8856702 - Flags: review?(glob) → review+

Comment 24

a year ago
mozreview-review
Comment on attachment 8856703 [details]
robustcheckout: refactor handlepullabort to handle multiple types (bug 1317594);

https://reviewboard.mozilla.org/r/128640/#review131422
Attachment #8856703 - Flags: review?(glob) → review+

Comment 25

a year ago
mozreview-review
Comment on attachment 8856704 [details]
robustcheckout: retry after socket errors (bug 1317594);

https://reviewboard.mozilla.org/r/128642/#review131424
Attachment #8856704 - Flags: review?(glob) → review+

Comment 26

a year ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8318beaacec6
robustcheckout: add test demonstrating connection failure ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6f25918d6b2b
robustcheckout: factor network failure handling into own function ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/03efe6e45246
robustcheckout: refactor handlepullabort to handle multiple types ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e0d30b04dac6
robustcheckout: retry after socket errors ; r=glob
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years agoa year ago
Resolution: --- → FIXED

Comment 27

a year ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/987931d9e607
Vendor latest version of robustcheckout extension; r=me

Comment 28

a year ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/249a47720ddc
robustcheckout: restore compatibility with Mercurial 3.7

Comment 29

a year ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f124872fb221
Vendor latest robustcheckout extension; r=me
5 failures in 894 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* pine: 2
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* linux64: 4
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-04-10&endday=2017-04-16&tree=all
6 failures in 891 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 3
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* linux32: 4
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-05-22&endday=2017-05-28&tree=all
You need to log in before you can comment on or make changes to this bug.