1480203 - unifyrepo fails with "abort: 00changelog.i@<REVISION>: no node!"

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Connor Sheehan [:sheehan]]

Attachment: Full email output of [e-mail]

During the Mercurial 4.6 server upgrade, we had an error arise in unifyrepo that caused bookmarks to fail to update. Although we fixed the issue, the service had failed and we did not know about it. We added the mail-unit-output override systemd unit as an `OnFailure` dependency so we know when things go wrong in the future. Since then we have been receiving emails constantly saying the service is failing, ending with the following output:

> 2018-08-01T20:15:50.221622 [25455] aggregating 479354/494397 nodes from 214368 > original pushes
> 2018-08-01T20:15:50.221780 [25455] 1/479354 nodes will be pulled
> 2018-08-01T20:15:50.221935 [25455] consolidated into 1 pulls from 196679 unique > pushes
> 2018-08-01T20:15:50.222088 [25455] searching for changes
> 2018-08-01T20:15:50.222275 [25455] abort: 00changelog.i@2ac1a4b2dd84: no node!
> 2018-08-01T20:15:50.849025 [1] unifyrepo.service: main process exited, code=exited, status=255/n/a

It appears the service is restarting itself and moving past the issue, but we should probably investigate why this is happening and mitigate the issue. I've attached the full email output.

Comment 1

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/66e3ea5ac6ef
unifyrepo: disable mail-unit-output dependency

Comment 2

[Connor Sheehan [:sheehan]]

The above patch was not a fix - I've just turned off the noisy email generated as a result of this bug. Re-opening with a note that whoever fixes this bug should revert 66e3ea5ac6ef before closing.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a9a421985728
unifyrepo: re-enable OnFailure emails

Comment 4

[Connor Sheehan [:sheehan]]

We are still seeing failures here, so my change from bug 1484231 did not fix the failure. :(

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/81092d499ce9
unifyrepo: disable OnFailure emails

Comment 7

[Connor Sheehan [:sheehan]]

Attachment: unifyrepo: move stage repo setup immediately before repo unification (Bug 1480203) r?gps

Currently the unification process creates the staging
repo at the start of the command, does intermediate
processing to determine which changesets/pushlog
entries will need to be pulled in, then performs
the repo unification. During the intermediate
step, some new pushes may enter the source repos and
are collected as resources to add to the unified
repository - however these new pushes do not exist
in the staging repo and the process fails to complete
(most annoyingly, failing to update bookmarks).

This commit moves the staging repo pull step closer
to the end of the process. With this change, the process
should determine which actions it needs to take then
create the stage repo. This will ensure that no changeset
present in the set of required unification actions is
missing from the stage repo.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ca9cabc9909f
unifyrepo: move stage repo setup immediately before repo unification r=gps

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 9008902 [details]
unifyrepo: move stage repo setup immediately before repo unification (Bug 1480203) r?gps

Gregory Szorc [:gps] has approved the revision.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/499309eca5a6
unifyrepo: re-enable OnFailure emails

Comment 11

[Connor Sheehan [:sheehan]]

We deployed ca9cabc9909f earlier today and re-activated the `OnFailure` emails. It appears that all is well and the race condition has been squashed.