1354824 - Intermittent abort: error: DECRYPTION_FAILED_OR_BAD_RECORD_MAC cloning mozilla-unified

Status: RESOLVED FIXED

(Developer Services :: Mercurial: robustcheckout, defect)

Description

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/logviewer.html#?job_id=89799566&repo=autoland

[vcs 2017-04-08T14:00:30.296936Z] applying clone bundle from https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/21062cd63dd72565a4f9afc35bb4f6f78726096c.packed1-gd.hg
[vcs 2017-04-08T14:00:30.360908Z] 321676 files to transfer, 1.75 GB of data
[vcs 2017-04-08T14:00:32.363105Z] 
[vcs 2017-04-08T14:00:33.362222Z] clone [=>                                           ]  123982447/1877856720 29s
[vcs 2017-04-08T14:00:34.362291Z] clone [==>                                          ]  151350417/1877856720 35s
[vcs 2017-04-08T14:00:35.362434Z] clone [===>                                         ]  186061794/1877856720 37s
[vcs 2017-04-08T14:00:36.411613Z] clone [====>                                        ]  228161212/1877856720 37sabort: error: DECRYPTION_FAILED_OR_BAD_RECORD_MAC

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

See also https://www.mercurial-scm.org/pipermail/mercurial/2016-February/049385.html

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Example:
https://public-artifacts.taskcluster.net/NeVu198XSHGSoMwYXpuoWQ/0/public/logs/live_backing.log

Comment 3

[Gregory Szorc [:gps]]

This is likely an intermittent network issue. robustcheckout should retry after this failure.

Although, that post on mercurial-devel is interesting. I may look into that upstream.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This doe now sometimes occur in production, e.g. 1354824

Comment 5

[Intermittent Failures Robot]

6 failures in 1008 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 6

Platform breakdown:
* linux64: 5
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354824&startday=2017-07-24&endday=2017-07-30&tree=all

Comment 6

[Intermittent Failures Robot]

17 failures in 131 pushes (0.13 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 17

Platform breakdown:
* linux64: 10
* linux32: 6
* linux64-noopt: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354824&startday=2017-08-01&endday=2017-08-01&tree=all

Comment 7

[Gregory Szorc [:gps]]

So strange that this goes for months without issue and then we see a ton of failures over a few days. I guess something in CloudFront land got poked and isn't happy about it.

Comment 8

[Gregory Szorc [:gps]]

Attachment: robustcheckout: retry after SSLError (bug 1354824);

I /think/ Mercurial is failing to handle ssl.SSLError and this error is
making it all the way to the default exception handler. This manifests
as e.g. "abort: error: DECRYPTION_FAILED_OR_BAD_RECORD_MAC".

This commit handles uncaught ssl.SSLError and treats them as retriable
network failures.

Writing tests for this will be a PITA. The easiest solution is likely
monkeypatching. I didn't bother trying given how low-level the
monkeypatch would need to be.

Review commit: https://reviewboard.mozilla.org/r/163688/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163688/

Comment 9

[:glob ✱]

Comment on attachment 8892704 [details]
robustcheckout: retry after SSLError (bug 1354824);

https://reviewboard.mozilla.org/r/163688/#review169520

lgtm

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ebe0553439be
robustcheckout: retry after SSLError ; r=glob

Comment 11

[Intermittent Failures Robot]

23 failures in 888 pushes (0.026 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 17
* try: 6

Platform breakdown:
* linux64: 14
* linux32: 7
* linux64-noopt: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354824&startday=2017-07-31&endday=2017-08-06&tree=all

Comment 12

[Intermittent Failures Robot]

7 failures in 590 pushes (0.012 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 3
* mozilla-central: 2
* autoland: 2

Platform breakdown:
* android-4-0-armv7-api16: 3
* android-api-16-gradle: 1
* android-5-0-aarch64: 1
* android-4-2-x86-old-id: 1
* android-4-2-x86: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354824&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:gps, maybe it's time to close this bug?

Comment 14

[Gregory Szorc [:gps]]

I'm not sure why the leave-open is around. Maybe I wanted to track upgrading robustcheckout in other repos? If so, that's been done.