Detect and recover from active locks and transactions

RESOLVED FIXED

Status

Developer Services
Mercurial: robustcheckout
RESOLVED FIXED
a year ago
3 months ago

People

(Reporter: gps, Assigned: gps)

Tracking

(Depends on: 1 bug)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(4 attachments)

(Assignee)

Description

a year ago
See bug 1297151. Essentially a previous TC task was likely forcefully aborted with SIGKILL. Mercurial didn't release the lock or transaction. The next task to use the cache inherited a stale lock and timed out waiting on its release because the process creating it had been terminated.

We can work around this issue by teaching `hg robustcheckout` to forcefully release an active lock and/or transaction if instructed to do so. This would likely be the default behavior on TaskCluster, where we're guaranteed to only have a single task/process accessing a repo at the same time, since TC guarantees only a single task has access to a cache.
(Assignee)

Updated

6 months ago
Duplicate of this bug: 1367929
(Assignee)

Updated

3 months ago
Duplicate of this bug: 1388944
(Assignee)

Comment 3

3 months ago
This has cropped up a few times recently. I'm planning to install a newer Mercurial in CI for the Mercurial security fixes. We might as well get some robustcheckout fixes in there as well.
Assignee: nobody → gps
Status: NEW → ASSIGNED
(Assignee)

Comment 4

3 months ago
Example log, for posterity.

[vcs 2017-08-10T01:10:55.390297Z] executing ['hg', 'robustcheckout', '--sharebase', '/home/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '886618c4e475d310e9e5e5177ee1a9899e3f3dd1', 'https://hg.mozilla.org/try', '/home/worker/checkouts/gecko']
[vcs 2017-08-10T01:10:55.454389Z] ensuring https://hg.mozilla.org/try@886618c4e475d310e9e5e5177ee1a9899e3f3dd1 is available at /home/worker/checkouts/gecko
[vcs 2017-08-10T01:10:55.454549Z] (existing repository shared store: /home/worker/checkouts/hg-store/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29/.hg)
[vcs 2017-08-10T01:10:55.526199Z] (pulling to obtain 886618c4e475d310e9e5e5177ee1a9899e3f3dd1)
[vcs 2017-08-10T01:10:56.738195Z] waiting for lock on repository /home/worker/checkouts/gecko held by 'abd5439b1213:7'
[vcs 2017-08-10T01:20:57.467127Z] abort: repository /home/worker/checkouts/gecko: timed out waiting for lock held by abd5439b1213:7
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 13

3 months ago
mozreview-review
Comment on attachment 8896061 [details]
robustcheckout: support Mercurial 4.3;

https://reviewboard.mozilla.org/r/167336/#review173802
Attachment #8896061 - Flags: review?(glob) → review+

Comment 14

3 months ago
mozreview-review
Comment on attachment 8896062 [details]
robustcheckout: tweak warning message;

https://reviewboard.mozilla.org/r/167338/#review173804
Attachment #8896062 - Flags: review?(glob) → review+

Comment 15

3 months ago
mozreview-review
Comment on attachment 8896063 [details]
robustcheckout: add tests demonstrating lock interaction (bug 1297153);

https://reviewboard.mozilla.org/r/167340/#review173808
Attachment #8896063 - Flags: review?(glob) → review+

Comment 16

3 months ago
mozreview-review
Comment on attachment 8896064 [details]
robustcheckout: detect and recover from open locks (bug 1297153);

https://reviewboard.mozilla.org/r/167342/#review173810
Attachment #8896064 - Flags: review?(glob) → review-
(Assignee)

Comment 17

3 months ago
(In reply to Byron Jones ‹:glob› from comment #16)
> Comment on attachment 8896064 [details]
> robustcheckout: detect and recover from open locks (bug 1297153);
> 
> https://reviewboard.mozilla.org/r/167342/#review173810

r- with no issues or comment?
Flags: needinfo?(glob)
(Assignee)

Updated

3 months ago
Duplicate of this bug: 1367175
sorry about that; i swear i r+'d it.  guess i slipped.  will correct.
Flags: needinfo?(glob)

Comment 20

3 months ago
mozreview-review
Comment on attachment 8896064 [details]
robustcheckout: detect and recover from open locks (bug 1297153);

https://reviewboard.mozilla.org/r/167342/#review174306
Attachment #8896064 - Flags: review- → review+

Comment 21

3 months ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e6d65314e2e8
robustcheckout: add tests demonstrating lock interaction ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/19db5f4b5b10
robustcheckout: detect and recover from open locks ; r=glob
Status: ASSIGNED → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → FIXED
(Assignee)

Comment 22

3 months ago
This will go out to in-tree Docker images in bug 1389715.
Blocks: 1389715
You need to log in before you can comment on or make changes to this bug.