1063208 - Rebuild repositories to have fncache and dotencode

Status: RESOLVED DUPLICATE of bug 1351859

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

There is a feature in Mercurial called "fncache" that basically maintains an in-memory cache of available data files so that Mercurial only needs to perform an in-memory lookup (as opposed to a filesystem lookup) to see if a path exists.

I was poking around the production servers and noticed that a hodgepodge of repositories (including mozilla-central) don't have the fncache.

The way you test for the fncache is |grep fncache .hg/requires|. If "fncache" isn't in that file, the repo was likely created with a very old Mercurial.

To "upgrade" a Mercurial repository, you clone it. The new repo will be created with the latest/greatest repository storage format.

We should identify repositories that are missing fncache and upgrade them by re-cloning. There are two phases to this:

1) Rebuild repositories on hgweb slaves
2) Rebuild repositories on master

#1 doesn't require downtime since you can take the slave offline and rebuild at will. #2 would require an outage window. Let's only focus on #1 now.

I'm not sure the performance implications of not having fncache on a server. But it definitely not helping not having fncache.

Comment 1

[Kendall Libby [:fubar] (he/him)]

Attachment: missing-fncache

Attaching list of repos missing fncache, from hgweb1.

Comment 2

[Kendall Libby [:fubar] (he/him)]

created /root/missing-fncache on each of the webheads with repos that need fixin'. all in the same ballpark in regards to number of repo:

     612 hgweb1/missing-fncache
     612 hgweb10/missing-fncache
     612 hgweb2/missing-fncache
     617 hgweb3/missing-fncache
     612 hgweb4/missing-fncache
     617 hgweb5/missing-fncache
     617 hgweb6/missing-fncache
     617 hgweb7/missing-fncache
     611 hgweb8/missing-fncache
     612 hgweb9/missing-fncache

Comment 3

[Gregory Szorc [:gps]]

I may work this into the TCW on Saturday.

Comment 4

[Gregory Szorc [:gps]]

I'm not going to do this today.

I think we should wait until bug 1089465 is in place, as it will make creating the new repo clones much easier.

Comment 5

[Gregory Szorc [:gps]]

dotencode is missing from 1686 repos :(

Comment 6

[Gregory Szorc [:gps]]

I've written an Ansible playbook to iterate through hgweb nodes and force a re-clone. A side-effect of re-cloning repositories is that they will get the latest/greatest format by default. The playbook integrates with the load balancers, so no client-side downtime is incurred during the switchover.

I've already executed the playbook against mozilla-central and I haven't seen any errors in the server logs nor have I heard any complaints that things broke. It appears the conversion "just works."

I'll continue to chug along and mass re-clone repositories during the week. This should be fully transparent from an end-user perspective.

Comment 7

[Gregory Szorc [:gps]]

I added a `hg debugrebuildfncache` command to Mercurial. This will ship in Mercurial 3.5. We can use that to easily create an fncache file. Although, it stops short of adding the requirement to the repo. Probably easier to just mass re-clone everything (which we need to do to get dotencode anyway).

Comment 8

[Gregory Szorc [:gps]]

As part of investigating bug 1232900, I discovered that many repos had invalid fncache files. mozilla-central, fx-team, mozilla-release, and nearly every b2g repo had incomplete fncache files. Why this is I'm not sure. I'm currently force rebuilding fncache on the important repos on the ssh server.

Comment 9

[Gregory Szorc [:gps]]

The new hgweb[11-14] machines were seeded via `hg clone` instead of rsync, so they utilize fncache and dotencode on all repos. We just have to convert the repos on the SSH/master server now. That could probably be scripted.

Comment 10

[Gregory Szorc [:gps]]

The upgrade to generaldelta in bug 1351859 will also add the fncache store requirement, since `hg debugupgraderepo` opts repos in to the latest/greatest storage format.