925383 - Allow repositories to enable obsolescence (support changeset evolution on the server)

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect, P4)

Description

[Gregory Szorc [:gps]]

The future of Mercurial involves Changeset Evolution (http://mercurial.selenic.com/wiki/ChangesetEvolution). The feature is amazing and will be a game changer. tl;dr is changes to history are annotated so clients can do things like automatically follow rebase operations.

Changeset Evolution is currently effectively in beta status. The core underpinnings of the feature (obsolescence markers) are available in Mercurial 2.5 and newer. However, the exchange of these markers (via push/pull) and the hg commands to manipulate and utilize these markers is currently being developed in an extension (http://mercurial.selenic.com/wiki/EvolveExtension). This means you need to install and activate the evolve extension *on the server* for the real power of changeset evolution to be realized.

In this bug, I'm requesting that Mozilla support enabling the evolve extension on *selected* repositories. Since the feature isn't yet fully mature and there are perf issues yet to be worked out, it definitely doesn't make sense to globally enable the extension. However, I believe individual user repositories and possibly twigs should have the option of opting in to the extension.

For an initial step, I propose we enable the evolve extension on a few test repositories. Next, we can integrate support for enabling evolve into the "user repositories" wizard (via |ssh hg.mozilla.org edit|). Eventually (once evolve has matured and is stable), we can talk about enabling it by default on repos that make sense. I think resolve criteria of this bug is when evolve can be enabled via the "edit" command.

Comment 1

[Gregory Szorc [:gps]]

The developer behind Changeset Evolution (marmoute on IRC) tells me the extension at https://bitbucket.org/marmoute/mutable-history/src/dcda64966616d3ba71e8d437fd482cf01dcff166/hgext/simple4server.py?at=stable is preferable to enabling the entire evolve extension for server-side usage. This works because obsolescence support is baked into the Mercurial core - it's just disabled by default (at least through Mercurial 2.7.2).

Comment 2

[Hal Wine [:hwine] use NI!]

I think we need a more detailed plan before we can go forward here, and may want to first deploy on the staging server. 

I'm also not clear what the definition of "success" is here. From the summit discussion, I understand changeset evolution to allow new workflows. It was unclear to me if there was general consensus on whether those workflows would be desired, and how they would interact with git workflows. I'm conservative with repositories -- if we don't have a demonstrable benefit to developers, I'm reluctant to experiment on the production system.

Before we'd "go live" with such a change, we'd want to ensure (demonstrate) that there is no impact to production in terms of performance or interoperability (e.g. with conversion to/from git).

I've opened bug 925423 for the enhancement to the 'ssh edit' command. As I read the rest of the plan to date, there aren't any actionable steps until that is in place on the production side. And it's a good idea for other reasons as well!

Comment 3

[Gregory Szorc [:gps]]

Speaking on behalf of many Mercurial developers, changeset evolution is amazing. It will dramatically increase a developer's desire to work with Mercurial. It is a game changer and differentiates Mercurial from Git in ways that I don't believe Git has plans of copying.

There is absolutely a demonstrable benefit to developers to have this enabled. If there weren't, I wouldn't have filed this bug!

I sympathize with concerns about impact to production systems.

I think supporting this on stage before production is fair. Test before you deploy, right?

AFAIK the main reason why obsolescence isn't enabled by default is because it doesn't scale up to massive repositories with dozens of developers all pushing obsolescence data. This will only be a concern for Mozilla if we enable obsolescence on a highly-active repo like try. And even then, this only becomes an issue if people start pushing obsolescence data to these repos. There is an open question of whether having obsolescence enabled on m-c, m-i, etc is even a good thing. The fact of the matter is obsolescence data opens up all kinds of collaboration possibilities not in use today, such as developers pushing "feature branches" to a central repo. There is all kinds of potential for increased collaboration and more streamlined workflows (especially when you start talking about try). This is why I say changeset evolution is a game changer.

I /think/ obsolescence data will play nice with hg-git because I /think/ this data isn't stored inside changesets, but rather as a separate part of the store. Unfortunately, Git won't see all this rich metadata because it doesn't support this concept. But that's Git's problem. Even if things don't play well, we can simply have the policy that repos with obsolescence enabled don't get synced to Git. Since I'm simply asking for user repos to support obsolescence in this bug and AFAIK user repos aren't synced, there should be no problem.

A core Mercurial contributor may wish to weigh in on the accuracy of my previous statements.

Comment 4

[Gregory Szorc [:gps]]

I've done some experiments and I think we can enable obsolescence for individual user repos via [experimental] evolve = all.

This will use the obsolescence exchange that is built into Mercurial core.

Pros:

* User repos can now store and exchange obsolescence markers

Cons / Risks:

* The obsolescence exchange built into core Mercurial is not very efficient, especially on the push side. Pushing to repos with obsolescence enabled will slow significantly as the number of markers increases.

* This feature is marked as experimental in Mercurial and the implementation may change in the future. We may have to perform a one-time upgrade of repositories with obsolescence enabled.

Non Risks:

* This is a per-repo setting that is baked into core. There shouldn't be a risk from an obsolescence-enabled repo "spilling over" into a non-obsolescence repo. (We can't use the mutable-history extensions on hgweb for this reason: they globally set obsolete._enabled.)

I'll submit a patch to pash shortly.

Comment 5

[Gregory Szorc [:gps]]

Attachment: pash: support enabling obsolescence on repositories ()

Reasons are discussed in the bug.

Comment 6

[Ben Kero [:bkero]]

+    if not config.has_section('experimental'):
+        config.add_section('experimental')
+
+    if enabled:
+        config.set('experimental', 'evolution', 'all')
+    else:
+        config.remove_option('experimental', 'evolution')

^ For this part, is config.remove_option going to throw an exception if that config section is not present?

Other than that concern, lgtm

Comment 7

[Justin Wood (:Callek)]

user repos by default have pushlog enabled,

I'd like some tests/defined-states with pushlog when evolution is enabled and obsolescence markers are involved.

(I don't particularly care for what the end result is with pushlog, and possibly a behavior similar to that of strip when obsolescence might make sense), but we should certainly account for it.

Comment 8

[Gregory Szorc [:gps]]

(In reply to Justin Wood (:Callek) from comment #7)
> user repos by default have pushlog enabled,
> 
> I'd like some tests/defined-states with pushlog when evolution is enabled
> and obsolescence markers are involved.
> 
> (I don't particularly care for what the end result is with pushlog, and
> possibly a behavior similar to that of strip when obsolescence might make
> sense), but we should certainly account for it.

Yes, a quick spot test is probably warranted. However, I really don't care too much about bugs between the interaction of pushlog and obsolescence. I wouldn't be surprised if the pushlog APIs broke for hidden changesets. It isn't something I'm inclined to support until obsolescence is formally supported by Mercurial core.

Comment 9

[Ben Kero [:bkero]]

Comment on attachment 8563002 [details] [diff] [review]
pash: support enabling obsolescence on repositories ()

Review of attachment 8563002 [details] [diff] [review]:
-----------------------------------------------------------------

see comment #6

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8563002 [details] [diff] [review]
pash: support enabling obsolescence on repositories ()

remove_option doesn't fail if the option doesn't exist. So I'm interpreting comment #6 as r+ since everything but remove_option was "LGTM."

Comment 11

[Gregory Szorc [:gps]]

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/25c1a9a295ac9a457db60109f2ec2931235b5515
changeset:  25c1a9a295ac9a457db60109f2ec2931235b5515
user:       Gregory Szorc <[e-mail]>
date:       Mon Aug 10 15:31:40 2015 -0700
description:
pash: support enabling obsolescence on repositories (bug 925383); r=bkero

Reasons are discussed in the bug.

The test was not explicitly reviewed.

Comment 12

[Gregory Szorc [:gps]]

This is deployed.