Abandoned transaction found when pushing to Try or MozReview

RESOLVED FIXED

Status

P1
normal
RESOLVED FIXED
a year ago
4 months ago

People

(Reporter: Mardak, Assigned: gps)

Tracking

Details

Attachments

(1 attachment)

This looks to have started failing from within a few hours ago (roughly around 7pm pacific August 3rd)

markh says:
I tried updating both version-control-tools and cinnabar, but that didn't help either
maybe it was the deploy - almost exactly 3 hours ago: gps> and hg.mo, mozreview, autoland, and vcs sync are now all on hg 4.2.2. finally.
am I'm fairly sure my push that worked was before that

heycam says:
can't push to mozreview from git or hg

It fails when pushing to an existing bug with mozreview or new bug without.

Here's the output I get:

$ git mozreview push
Traceback (most recent call last):
  File "~/.mozbuild/version-control-tools/git/cinnabar-debug-push.py", line 117, in <module>
    sys.exit(main(sys.argv[1:]))
  File "~/.mozbuild/version-control-tools/git/cinnabar-debug-push.py", line 85, in main
    pushed = push(repo, store, {commit: (None, False)}, heads, ())
  File "~/git-cinnabar/cinnabar/hg/__init__.py", line 674, in push
    reply = repo.unbundle(cg, repo_heads, '')
  File "/usr/local/lib/python2.7/site-packages/mercurial/wireproto.py", line 443, in unbundle
    stream = self._calltwowaystream('unbundle', cg, heads=heads)
  File "/usr/local/lib/python2.7/site-packages/mercurial/httppeer.py", line 359, in _calltwowaystream
    return self._callstream(cmd, data=fp_, headers=headers, **args)
  File "/usr/local/lib/python2.7/site-packages/mercurial/httppeer.py", line 230, in _callstream
    resp = self.urlopener.open(req)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 909, in http_error_401
    url, req, headers)
  File "/usr/local/lib/python2.7/site-packages/mercurial/url.py", line 405, in http_error_auth_reqed
    self, auth_header, host, req, headers)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 887, in http_error_auth_reqed
    return self.retry_http_basic_auth(host, req, realm)
  File "/usr/local/lib/python2.7/site-packages/mercurial/url.py", line 416, in retry_http_basic_auth
    return self.parent.open(req)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.13_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error
abort: error performing cinnabar push; please report this bug
What I got from hg is:
> pushing to ssh://reviewboard-hg.mozilla.org/autoreview
> searching for appropriate review repository
> redirecting push to ssh://reviewboard-hg.mozilla.org/gecko
> searching for changes
> remote: abort: abandoned transaction found!
> remote: (run 'hg recover' to clean up transaction)
> abort: stream ended unexpectedly (got 0 bytes, expected 4)
(Assignee)

Comment 2

a year ago
I ran `hg recover` on the server. I see pushes coming through now. Still unsure how it got in a bad state. Will likely have to wait until morning to debug further.
Flags: needinfo?(gps)
(Assignee)

Comment 3

a year ago
What I know are gandalf@aviary.pl and ehung@mozilla.com were the last people to push to the gecko repo before the first error (each around 03:20 UTC). rchien@mozilla.com was the first to encounter an HTTP 500 at 03:22:22.

cliu@mozilla.com established a series of SSH connections starting at 03:21:55 through 03:34:25. They appear to be push attempts. But they all appeared to fail. (The logs don't tell me why.)

I'm pretty confident saying that cliu@mozilla.com was the first person to encounter the problem. I just don't know if it was (inadvertently) caused by her client or whether she was the unlucky first victim.

liuche: I don't suppose you have your terminal output from your first mozreview push attempt, do you?
Flags: needinfo?(liuche)
Okay! Looking through my history, it looks like I started to push some commits to reviewboard, and then cancelled the push because I wanted to rebase. Mea culpa, it looks like.

First push (with ctrl-c):
> pushing to ssh://reviewboard-hg.mozilla.org/autoreview
> searching for appropriate review repository
> redirecting push to ssh://reviewboard-hg.mozilla.org/gecko
> running checkstyle… success!
> searching for changes
> ^Cinterrupted!
> remote: adding changesets
> remote: adding manifests
> remote: adding file changes
> remote: Killed by signal 2.
> remote: Killed by signal 2.

Then I switched to central, pull -u, switched back to my bookmark (didn't bother to rebase after all), and then tried to push to reviewboard again. And this is where my subsequent failed push happened:
> pushing to ssh://reviewboard-hg.mozilla.org/autoreview
> searching for appropriate review repository
> redirecting push to ssh://reviewboard-hg.mozilla.org/gecko
> running checkstyle… success!
> searching for changes
> remote: abort: abandoned transaction found!
> remote: (run ‘hg recover’ to clean up transaction)
> abort: stream ended unexpectedly (got 0 bytes, expected 4)

At this point I gave up and went home, but things worked the next morning so I forgot about it.

I promise to never ctrl-C in the middle of a push ever again.
Flags: needinfo?(liuche)
Ctrl-C on client side isn't suppose to affect server like this, I believe...
(Assignee)

Comment 6

a year ago
Correct: you should be able to abort on the client and the server should fail gracefully. After all, a network hiccup causing the client to lose a connection to the server is essentially the same as aborting the push via client-side process termination.
This happens to try today.
> Creating temporary commit for remote...
> pushing to ssh://hg.mozilla.org/try
> searching for changes
> remote: abort: abandoned transaction found!
> remote: (run 'hg recover' to clean up transaction)
> temporary commit removed, repository restored
> abort: stream ended unexpectedly (got 0 bytes, expected 4)
Blocks: 1389997
(Assignee)

Updated

a year ago
Component: General → Mercurial: hg.mozilla.org
Flags: needinfo?(gps)
Product: MozReview → Developer Services
Summary: Pushing gives urllib2.HTTPError: HTTP Error 500: Internal Server Error → Abandoned transaction found when pushing to Try or MozReview
(Assignee)

Comment 8

a year ago
This feels like a regression from upgrading servers from 4.1 to 4.2 (or another change made around that time). I have no clue what's causing it because I can't reproduce it locally via ^C. But I'm looking.
Assignee: nobody → gps
Blocks: 1359641
Status: NEW → ASSIGNED
Priority: -- → P1
(Assignee)

Comment 9

a year ago
https://bz.mercurial-scm.org/show_bug.cgi?id=5658 is what I think is happening.
(Assignee)

Comment 10

a year ago
I've submitted some patches upstream.

Our likely path forward is a custom build of 4.2 with the patches backported. That will hold us over until we upgrade servers to 4.3, which is likely a few weeks away.
Why not simply downgrade to 4.1? Are we already relying on the server being 4.2?
(Assignee)

Comment 12

a year ago
We can't run a stock 4.1 because the CVE patches aren't there. I'd rather run 4.2 with a minimal 4.3 backport than 4.1 with all the security patches backported.
None of the CVEs fixed in 4.3 matter on the server side. And I don't see any other CVE addressed in 4.2.
Comment hidden (mozreview-request)
(Assignee)

Comment 15

a year ago
We perform working directory checkouts on the server in some cases. I'm paranoid and don't want vulnerable code *anywhere* on the server.

Comment 16

a year ago
mozreview-review
Comment on attachment 8897647 [details]
ansible: install a custom Mercurial 4.2 with interrupt backports (bug 1387324);

https://reviewboard.mozilla.org/r/168912/#review174318
Attachment #8897647 - Flags: review?(glob) → review+
See Also: → bug 1391199
(Assignee)

Comment 17

a year ago
Upstream accepted my patches to 4.3. I was waiting for that to occur before deploying this. So I'll deploy this sometime today, after I get in the office.
Flags: needinfo?(gps)
(Assignee)

Comment 18

a year ago
Doing this now'ish.
Flags: needinfo?(gps)

Comment 19

a year ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/dfebe316dc8d
ansible: install a custom Mercurial 4.2 with interrupt backports ; r=glob
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
(Assignee)

Updated

a year ago
Duplicate of this bug: 1391194
(Assignee)

Updated

a year ago
Duplicate of this bug: 1391199
(Assignee)

Updated

4 months ago
Duplicate of this bug: 1388094
You need to log in before you can comment on or make changes to this bug.