1317594 - Intermittent abort: error applying bundle

Status: RESOLVED FIXED

(Developer Services :: Mercurial: robustcheckout, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=6633280&repo=autoland

https://queue.taskcluster.net/v1/task/fovCB_XTRsOxhZPBOzRWvg/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Greg, looks like this is an issue with the robust checkout?

Comment 2

[Gregory Szorc [:gps]]

The log says a connection to https://s3-external-1.amazonaws.com is failing. The only thing you can blame robustcheckout for is not retrying in case of this error.

I don't think I've seen HTTP requests to S3 fail like this before. First time for everything.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, indeed, we had some similar issues with mozharness.zip failing around the same time (bug 1266624).  Probably a common cause.

Comment 4

[Intermittent Failures Robot]

5 failures in 715 pushes (0.007 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* try: 1

Platform breakdown:
* linux64: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2016-11-14&endday=2016-11-20&tree=all

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Still a thing, but with lower frequency: https://public-artifacts.taskcluster.net/SZRfPJKlSau5A5s1ovxwAQ/0/public/logs/live_backing.log

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like it's time for some retry in robustcheckout

Comment 7

[Intermittent Failures Robot]

6 failures in 526 pushes (0.011 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 5
* autoland: 1

Platform breakdown:
* linux64: 3
* android-4-0-armv7-api15: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2016-12-12&endday=2016-12-18&tree=all

Comment 8

[Intermittent Failures Robot]

9 failures in 690 pushes (0.013 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 5
* graphics: 2
* mozilla-inbound: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 7
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-01-16&endday=2017-01-22&tree=all

Comment 9

[Intermittent Failures Robot]

8 failures in 733 pushes (0.011 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* try: 1
* mozilla-central: 1
* mozilla-beta: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 7
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-01-30&endday=2017-02-05&tree=all

Comment 10

[Intermittent Failures Robot]

12 failures in 836 pushes (0.014 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 5
* mozilla-inbound: 4
* try: 1
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 7
* android-4-0-armv7-api15: 2
* windows2012-64: 1
* linux64-qr: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-02-06&endday=2017-02-12&tree=all

Comment 11

[Intermittent Failures Robot]

11 failures in 833 pushes (0.013 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* mozilla-inbound: 5
* autoland: 5
* try: 1

Platform breakdown:
* linux64: 5
* linux32: 3
* android-4-2-x86: 2
* windows2012-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-02-13&endday=2017-02-19&tree=all

Comment 12

[Intermittent Failures Robot]

17 failures in 783 pushes (0.022 failures/push) were associated with this bug in the last 7 days.  
Repository breakdown:
* autoland: 7
* try: 3
* graphics: 3
* mozilla-central: 2
* mozilla-release: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 12
* linux32: 4
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-02-27&endday=2017-03-05&tree=all

Comment 13

[Intermittent Failures Robot]

16 failures in 777 pushes (0.021 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 9
* mozilla-inbound: 4
* mozilla-central: 1
* mozilla-beta: 1
* autoland: 1

Platform breakdown:
* linux64: 11
* linux32: 2
* windows7-32-vm: 1
* windows10-64-vm: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-03-13&endday=2017-03-19&tree=all

Comment 14

[Intermittent Failures Robot]

5 failures in 845 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 4
* autoland: 1

Platform breakdown:
* linux32: 4
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-03-27&endday=2017-04-02&tree=all

Comment 15

[Intermittent Failures Robot]

5 failures in 867 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 3
* try: 2

Platform breakdown:
* osx-10-7: 2
* android-4-2-x86: 2
* linux64-stylo: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-04-03&endday=2017-04-09&tree=all

Comment 16

[Gregory Szorc [:gps]]

Example log output:

[vcs 2017-04-03T18:17:33.613036Z] executing ['hg', 'robustcheckout', '--sharebase', '/home/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', 'caee52f2863b8c46da0a28d593d9efff2c09465a', 'https://hg.mozilla.org/integration/autoland/', '/home/worker/workspace/build/src']
[vcs 2017-04-03T18:17:33.676151Z] ensuring https://hg.mozilla.org/integration/autoland/@caee52f2863b8c46da0a28d593d9efff2c09465a is available at /home/worker/workspace/build/src
[vcs 2017-04-03T18:17:33.676279Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[vcs 2017-04-03T18:17:34.381987Z] (sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
[vcs 2017-04-03T18:19:42.179885Z] applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/66146205625406fd7eabd6b03c8b4c7ec464c3a1.packed1-gd.hg
[vcs 2017-04-03T18:21:49.415396Z] error fetching bundle: Connection timed out
[vcs 2017-04-03T18:21:49.417022Z] abort: error applying bundle
[vcs 2017-04-03T18:21:49.417073Z] (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")


I think this is a matter of adding that "Connection timed out" message to our list of errors to retry after encountering.

Comment 17

[Gregory Szorc [:gps]]

I'll code up some patches.

Comment 18

[Gregory Szorc [:gps]]

Attachment: robustcheckout: add test demonstrating connection failure (bug 1317594);

This test demonstrates that connecting to a server that refuses the
connection doesn't retry. This is a bug. It will be fixed in a
subsequent commit.

Review commit: https://reviewboard.mozilla.org/r/128636/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/128636/

Comment 19

[Gregory Szorc [:gps]]

Attachment: robustcheckout: factor network failure handling into own function (bug 1317594);

We're going to introduce another consumer. Let's make the code easier
to call.

Review commit: https://reviewboard.mozilla.org/r/128638/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/128638/

Comment 20

[Gregory Szorc [:gps]]

Attachment: robustcheckout: refactor handlepullabort to handle multiple types (bug 1317594);

Not all network-related failures raise error.Abort. In preparation
for handling other error types, teach the function to check the
exception type and rename it so it isn't limited to error.Abort.

Review commit: https://reviewboard.mozilla.org/r/128640/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/128640/

Comment 21

[Gregory Szorc [:gps]]

Attachment: robustcheckout: retry after socket errors (bug 1317594);

Previously, we didn't handle urllib.URLError and Mercurial's default
exception handler converted this to an error.Abort for display to the
user.

This commit explicitly catches urllib.URLError (aliased as
pycompat.urlerr.urlerror in Mercurial for compatibility with Python
3) and detects when the underlying error is a socket.error. It then
invokes the retry code for this scenario.

This should detect a whole class of network-related failures,
including connection refused, routing errors, and connection timeout.
Some of these might make sense to not retry on. But I'm inclined to
cast a wide net to ensure we aren't chasing the long tail of one-off
conditions that can contribute an intermittent network event.

This patch also doesn't handle non-socket errors in URLError. I'm
not as confident that we should retry on these. Let's see what happens
and we can evaluate things later.

Review commit: https://reviewboard.mozilla.org/r/128642/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/128642/

Comment 22

[:glob ✱]

Comment on attachment 8856701 [details]
robustcheckout: add test demonstrating connection failure (bug 1317594);

https://reviewboard.mozilla.org/r/128636/#review131264

Comment 23

[:glob ✱]

Comment on attachment 8856702 [details]
robustcheckout: factor network failure handling into own function (bug 1317594);

https://reviewboard.mozilla.org/r/128638/#review131280

Comment 24

[:glob ✱]

Comment on attachment 8856703 [details]
robustcheckout: refactor handlepullabort to handle multiple types (bug 1317594);

https://reviewboard.mozilla.org/r/128640/#review131422

Comment 25

[:glob ✱]

Comment on attachment 8856704 [details]
robustcheckout: retry after socket errors (bug 1317594);

https://reviewboard.mozilla.org/r/128642/#review131424

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8318beaacec6
robustcheckout: add test demonstrating connection failure ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6f25918d6b2b
robustcheckout: factor network failure handling into own function ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/03efe6e45246
robustcheckout: refactor handlepullabort to handle multiple types ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e0d30b04dac6
robustcheckout: retry after socket errors ; r=glob

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/987931d9e607
Vendor latest version of robustcheckout extension; r=me

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/249a47720ddc
robustcheckout: restore compatibility with Mercurial 3.7

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f124872fb221
Vendor latest robustcheckout extension; r=me

Comment 30

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/1c79bcfea679

Comment 31

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/cfdbc0969cac

Comment 32

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/15f63401caef

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/cfdbc0969cac

Comment 34

[Iris Hsiao [:ihsiao]]

https://hg.mozilla.org/mozilla-central/rev/987931d9e607
https://hg.mozilla.org/mozilla-central/rev/f124872fb221

Comment 35

[Intermittent Failures Robot]

5 failures in 894 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* pine: 2
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* linux64: 4
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-04-10&endday=2017-04-16&tree=all

Comment 36

[Intermittent Failures Robot]

6 failures in 891 pushes (0.007 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 3
* mozilla-inbound: 2
* autoland: 1

Platform breakdown:
* linux32: 4
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1317594&startday=2017-05-22&endday=2017-05-28&tree=all