645628 - Recover from |abort: abandoned transaction found - run hg recover!| on cn-sea-qm-win2k3-01

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Justin Wood (:Callek)]

We encountered an hg error on cn-sea-qm-win2k3-01 for SM2.1 builds.

I am attempting a clobberer clobber now, filing bug incase this doesn't fix it.

Comment 1

[Serge Gautherie (:sgautherie)]

Not fixed, or happening again:

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey2.1/1301656253.1301660870.4639.gz
WINNT 5.2 comm-2.0 nightly on 2011/04/01 04:10:53
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey2.1/1301669028.1301673062.22706.gz
WINNT 5.2 comm-2.0 leak test build on 2011/04/01 07:43:48
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey2.1/1301683406.1301683497.30593.gz
WINNT 5.2 comm-2.0 leak test build on 2011/04/01 11:43:26

Comment 2

[Serge Gautherie (:sgautherie)]

(In reply to comment #1)

Ah, the 2 first would be bug 647246,
which might be causing this bug in the 3rd.

Comment 3

[Justin Wood (:Callek)]

(In reply to comment #2)
> (In reply to comment #1)
> 
> Ah, the 2 first would be bug 647246,
> which might be causing this bug in the 3rd.

That bug is *not* the cause of this.

This is due to the time outs, above, which buildbot forcibly shuts down running process[es] when it aborts, and because hg was _active_ when we abort here, we lose.

The problem with the timeouts is that there is no stdout/stderr feedback during that whole time buildbot is waiting for that feedback; at least in any method that buildbot can detect. Bug 647246 was an attempt at working around the slow network to our Mercurial server on this machine, incase the way that extension implemented the progress feedback would feed into Buildbot and cause us to not timeout.

As mentioned in Bug 647246 the end-result right solution will be a manual clone on that machine, for the HG-Shared directory, when I setup hg shared with our buildbots. That will get us repo[s] that are in a already cloned state, that are good to go. Only needing hg pull/updates, which will be remarkably faster compared to the blank-slate clones.

Blank slate clones will be a necessary evil before hg-shared due to free-space-clobbers that we hit more often than we would like, as we do not have as large a builds drive as most Mozilla machines have, [most mozilla machines have 80GB, iirc our /largest/ is 30GB]

I am however leaving this as REOPEN for now, since I still do have to re-clean up after this machine, otherwise we'll continue to burn. But I'll wait until tomorrow, when I hope I can have an hg-shared updated patch ready to deploy anyway.

Comment 4

[Justin Wood (:Callek)]

Ok recovered.

Manually cloned on the slave, for hg-shared, all should be good [hopefully no more timeouts here]