1268963 - Pushing to bookmarks adds @1, @2, etc.

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

dustin@dustin-moz-devel ~/p/m-c (bug1258497) $ hg push -f -B bug1258497 user
pushing to ssh://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 3100 changesets with 20603 changes to 12288 files (+1 heads)
remote: recorded push in pushlog
remote:
remote: View the pushlog for these changes here:
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/pushloghtml?changeset=e57ca0950a78
remote: recorded changegroup in replication log in 0.054s
updating bookmark bug1258497

..leads to..

https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/bookmarks
10 days ago 	bug1258497 	changeset | changelog | files
2 hours ago 	bug1258497@1 	changeset | changelog | files
10 seconds ago 	bug1258497@2 	changeset | changelog | files

I know there was a bug for this about a year ago, but that was supposed to be fixed, and yet..

Comment 1

[Gregory Szorc [:gps]]

I can confirm the repo on the master server doesn't have the @ bookmarks. This is a bug in replication.

This was supposedly fixed several months ago. It must have regressed :/

Comment 2

[Gregory Szorc [:gps]]

I'm curious what you are doing that https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hgserver/tests/test-bookmarks-replication.t isn't testing. Because that test passes...

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

dustin@dustin-moz-devel ~/p/m-c (bug1258497) $ hg push --debug -f -B bug1258497 user
Bugzilla username not defined; cannot define credentials
pushing to ssh://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/
running ssh hg.mozilla.org 'hg -R users/dmitchell_mozilla.com/mozilla-central/ serve --stdio'
sending hello command
sending between command
remote: 370
remote: capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream bundle2=HG20%0Ab2x%253Arebase%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 pushlog
remote: 1
query 1; heads
sending batch command
searching for changes
taking initial sample
query 2; still undecided: 367, sample size is: 200
sending known command
2 total queries
preparing listkeys for "phases"
sending listkeys command
received listkey for "phases": 3353 bytes
checking for updated bookmarks
preparing listkeys for "bookmarks"
sending listkeys command
received listkey for "bookmarks": 363 bytes
no changes found
preparing listkeys for "phases"
sending listkeys command
received listkey for "phases": 3353 bytes

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

From a push with changes:

dustin@dustin-moz-devel ~/p/m-c (bug1258497) $ hg push --debug -f -B bug1258497 user
Bugzilla username not defined; cannot define credentials
pushing to ssh://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/
running ssh hg.mozilla.org 'hg -R users/dmitchell_mozilla.com/mozilla-central/ serve --stdio'
sending hello command
sending between command
remote: 370
remote: capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream bundle2=HG20%0Ab2x%253Arebase%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 pushlog
remote: 1
query 1; heads
sending batch command
searching for changes
taking quick initial sample
query 2; still undecided: 295830, sample size is: 100
sending known command
sampling from both directions
query 3; still undecided: 969, sample size is: 200
sending known command
3 total queries
preparing listkeys for "phases"
sending listkeys command
received listkey for "phases": 3353 bytes
checking for updated bookmarks
preparing listkeys for "bookmarks"
sending listkeys command
received listkey for "bookmarks": 363 bytes
5 changesets found
list of changesets:
4c2ebe7c5672d2e6cf9fda7f908340ef5c6c50aa
c3b2cabae31c4476f237a2046380873416f89bb4
3d0afa6372bcfd38bae9f4aeac1830fb06fcdba2
7584f07982d0fbe12f342cd5e81c783c06f4307b
5903e2fccacd160baa801f2033ec968ba1db6443
sending unbundle command
bundle2-output-bundle: "HG20", 3 parts total
bundle2-output-part: "replycaps" 155 bytes payload
bundle2-output-part: "changegroup" (params: 1 mandatory) streamed payload
bundle2-output-part: "pushkey" (params: 4 mandatory) empty payload
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 5 changesets with 8 changes to 31 files (+1 heads)
remote: recorded push in pushlog
remote: 
remote: View your changes here:
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/rev/4c2ebe7c5672
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/rev/c3b2cabae31c
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/rev/3d0afa6372bc
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/rev/7584f07982d0
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/rev/5903e2fccacd
remote: recorded changegroup in replication log in 0.004s
bundle2-input-bundle: with-transaction
bundle2-input-part: "reply:changegroup" (advisory) (params: 0 advisory) supported
bundle2-input-part: "reply:pushkey" (params: 0 advisory) supported
bundle2-input-bundle: 1 parts total
updating bookmark bug1258497
preparing listkeys for "phases"
sending listkeys command
received listkey for "phases": 3353 bytes

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I have to admit, I'm starting to like this feature, as it means that I can push to my user repo frequently and even if some hunk or patch disappears from my repo (histedit and rebase are lossy operations..) I can find it in one of the backups.

Comment 6

[Gregory Szorc [:gps]]

histedit and rebase are not lossy: they store backup bundles under .hg/store-backups. I haven't used it, but you should try https://bitbucket.org/facebook/hg-experimental/src/be7a54df7b36f7e9ac93ae05b9ff7621e32c089c/backups.py?fileviewer=file-view-default for finding commits in your backup bundles.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

It's true -- I've used the bundles may times, but sometimes found it hard to figure out which bundle to use when the loss had occurred several operations ago.  The backups extension sounds useful in exactly that situation!

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I tried deleting and re-creating the user repo, to no avail:

https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/bookmarks

23 hours ago 	bug1252948@1 	changeset | changelog | files
23 hours ago 	bug1252948 	changeset | changelog | files

Comment 9

[Gregory Szorc [:gps]]

I need to find time to look at this. Bookmarks should "just work."

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a92571001fda
vcsreplicator: log pushkey messages at proper level 
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f1e5524cb68e
vcsreplicator: log processing of each message

Comment 11

[Gregory Szorc [:gps]]

I just deployed extra logging to make this easier to debug.

dustin: please update this bug next time you push something that creates bookmark badness. Try to include the UTC time of the push (or a link to the pushlog entry) to make it easier to find the log entries.

(I would needinfo you but you aren't accepting needinfo requests!)

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

..and yet, I responded. MAGIC! :)

(sandbox) dustin@lamport ~/p/m-c (bug1252948) $ hg push -r . user; date
pushing to ssh://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/
searching for changes
remote has heads on branch 'default' that are not known locally: e756ee86e3ea
remote: adding changesets                                                                                                                                                                                                                      
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 2 changes to 7 files (+1 heads)
remote: recorded push in pushlog
remote: 
remote: View your change here:
remote:   https://hg.mozilla.org/users/dmitchell_mozilla.com/mozilla-central/rev/b76e73b3be0d13ad7b7dd22bc773d38c0807bf2f
remote: recorded changegroup in replication log in 0.003s
updating bookmark bug1252948
Sat Dec  3 04:28:20 UTC 2016

this push appeared as bug1252948@6

Comment 13

[Gregory Szorc [:gps]]

Finally getting around to this needinfo. I think I tracked this down and should have a solution shortly.

Comment 14

[Gregory Szorc [:gps]]

Attachment: vcsreplicator: add test demonstrating bookmark divergence (bug 1268963);

We don't currently handle bookmark divergence properly in vcsreplicator.
We add a test for this.

Review commit: https://reviewboard.mozilla.org/r/142634/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/142634/

Comment 15

[Gregory Szorc [:gps]]

Attachment: vcsreplicator: use remote value for incoming bookmarks (bug 1268963);

Previously, `hg pull` relied on Mercurial's built-in behavior for
bookmark processing. The built-in behavior was to look at differences
between local and remote bookmarks and reconcile the differences.

Since the point of vcsreplicator is to run a mirror, the remote state
should be the final local state. So Mercurial's default behavior isn't
appropriate. This means we need to monkeypatch Mercurial.

We introduce a minimal consumer-only extension to be installed on
mirrors. The extension monkeypatches the bookmarks function that
is called when local bookmarks are to be updated from a remote.

The way bookmarks exchange works in Mercurial is that the full
set of bookmarks from the server is fetched and applied. In other
words, we never only receive bookmarks for just the changesets
that are being changed. This means it is sufficient to blow away
all local bookmarks and just let Mercurial's default behavior
kick in to re-set them. There may be some corner cases when
obsolescence is involved that this behavior breaks down. But
obsolescence is experimental. So I'd rather get bookmark replication
working for normal Mercurial and break obsolescence than spend a
lot of time writing tests that may not uncover a problem.

Writing this commit reminded me that I want to add an
`hg pull --mirror` feature to upstream Mercurial. I /think/ I have
buy-in on the idea. I just need to produce code.

Review commit: https://reviewboard.mozilla.org/r/142636/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/142636/

Comment 16

[:glob ✱]

Comment on attachment 8871122 [details]
vcsreplicator: add test demonstrating bookmark divergence (bug 1268963);

https://reviewboard.mozilla.org/r/142634/#review146320

Comment 17

[:glob ✱]

Comment on attachment 8871123 [details]
vcsreplicator: use remote value for incoming bookmarks (bug 1268963);

https://reviewboard.mozilla.org/r/142636/#review146324

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6039254c38d7
vcsreplicator: add test demonstrating bookmark divergence ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4da46067f88b
vcsreplicator: use remote value for incoming bookmarks ; r=glob

Comment 19

[Gregory Szorc [:gps]]

I deployed and forced a resync on repos having "@<N>" bookmarks. All the "@<N>" divergent bookmarks on hgweb no longer exist!

Comment 20

[Gregory Szorc [:gps]]

This should have been resolved 12 days ago, as this is no longer a problem.