Closed Bug 1228086 Opened 6 years ago Closed 5 years ago

Upgrade hg.mozilla.org to Mercurial 3.6

Categories

(Developer Services :: Mercurial: hg.mozilla.org, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gps, Unassigned)

Details

Attachments

(23 files, 22 obsolete files)

40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
Our policy allows us to do this as early as December 1.

Things that need addressed:

* hgweb has a new option for highlight extension that disables overly aggressive content detection
* `hg bundle` will produce bundle2 bundles by default. These aren't compatible with old clients and will break cloning from bundles on old clients. Need to update bundle generation script and manifests to produce gzip-v1 bundles.
* Investigate HTTP header size limit config option
* Templates refresh

I should have another look through the changelog to look for additional things.
Addressing the HTTP header length in bug 1232492.
Will address reordered bundles in bug 1225673.
hgtemplates: remove gitweb_mozilla_pvt; r?smacleod

Nothing references these files AFAICT. They haven't been kept in sync in
ages either. Kill the dead weight.
Attachment #8698250 - Flags: review?(smacleod)
serverlog: use absolute_import; r?smacleod
Attachment #8698251 - Flags: review?(smacleod)
serverlog: refactor repo path logic into standalone function; r?smacleod

The serverlog extension needs some significant refactoring to work in
Mercurial 3.6. This patch starts the process by moving a repo path
computation method to a standalone function. This starts a mini series
that will result in the syslogmixin class being eliminated.
Attachment #8698252 - Flags: review?(smacleod)
serverlog: record syslog attributes on serverlog dict; r?smacleod

This removes the need to have a repo/ui reference at syslog write time.
Attachment #8698253 - Flags: review?(smacleod)
serverlog: extract syslog logging to own function; r?smacleod

This effectively cripples syslogmix, setting it up for removal.
Attachment #8698254 - Flags: review?(smacleod)
serverlog: stop calling _syslog from syslogmixin; r?smacleod

We now call logsyslog() directly. While I was here, I also corrected
some whitespace and switched some lookups to use the local "sl" alias
for self._serverlog.
Attachment #8698255 - Flags: review?(smacleod)
serverlog: remove syslogmixin; r?smacleod

It is no longer used.
Attachment #8698256 - Flags: review?(smacleod)
serverlog: remove localrepository wrapping; r?smacleod

This isn't working with at least Mercurial 3.5. The approach isn't
compatible with Mercurial 3.6 anyway. So nuke the code. If we want it,
we can always add it back in later.
Attachment #8698257 - Flags: review?(smacleod)
serverlog: tidy serverlog variable references; r?smacleod

We don't need to be using self._serverlog so much.
Attachment #8698258 - Flags: review?(smacleod)
ansible: mass upgrade to Mercurial 3.6.2 (bug 1228086); r?smacleod

Just the package updates. We'll need to synchronize templates and other
foo in subsequent patches.
Attachment #8698259 - Flags: review?(smacleod)
ansible: use built-in clonebundles extension; r?smacleod

We had vendored the clonebundles extension so we could run it with
Mercurial 3.5. Now that we're running 3.6 on the server, we can use the
built-in extension.
Attachment #8698260 - Flags: review?(smacleod)
serverlog: make compatible with Mercurial 3.6; r?smacleod

I refactored how repository instances are used by hgweb in Mercurial
3.6. In the new world, a repo instance is attached to the request passed
into _runwsgi rather than the hgweb instance itself. This required
refactoring which function we wrap so we can access the repo instance
properly.

This commit does drop compatibility with Mercurial < 3.6. This should be
fine: Mozilla is the only consumer of this extension.
Attachment #8698261 - Flags: review?(smacleod)
hgtemplates: synchronize templates with Mercurial 3.6.2; r?smacleod

Accepting upstream changes verbatim. We'll need to port the gitweb
changes to gitweb_mozilla in a subsequent patch.

This should be a rubber stamp review. Verifying we didn't remove any
Mozilla specific functionality would be a good idea though.
Attachment #8698262 - Flags: review?(smacleod)
hgtemplates: synchronize with gitweb (bug 1228086); r?smacleod

I manually went through the previous changes to the gitweb directory and
applied them to gitweb_mozilla.
Attachment #8698263 - Flags: review?(smacleod)
generate-hg-s3-bundles: use hg from virtualenv; r?smacleod

Without this, we pick up the system hg, which may not have the features
we need.
Attachment #8698264 - Flags: review?(smacleod)
generate-hg-s3-bundles: specify bundle version type; r?smacleod

`hg bundle` in Mercurial 3.6 will generate bundle2 bundle files by
default. bundle2 files aren't compatible with older clients and will
break bundleclone/clonebundles if they are served to old clients.

The bundle type string passed to `hg bundle` can control the type of
bundle generated. "v1" denotes the original bundle format. "v2" denotes
bundle2. Explicitly denote "v1" to force generation of the old bundle
format.
Attachment #8698265 - Flags: review?(smacleod)
generate-hg-s3-bundles: support for optionally generating bzip2 bundles; r?smacleod

These bundles take a long time to generate and aren't used that
frequently in the wild. We want to only generate them for certain
repositories. Start to add support for not generating bzip2 bundles for
every repo.
Attachment #8698266 - Flags: review?(smacleod)
generate-hg-s3-bundles: only generate bzip2 bundles for release repos; r?smacleod

bzip2 bundles aren't used that much in the wild. They are most relevant
for release repos. So only generate them for those repos.

As part of this, we change the repos manifest file format to introduce
the concept of flags. Each value after the repo name gets turned into a
named argument to the generate_bundles function.
Attachment #8698267 - Flags: review?(smacleod)
testing: don't install Mercurial 3.1 and 3.2; r?smacleod

We only support the last 4 Mercurial releases.
Attachment #8698268 - Flags: review?(smacleod)
hgserver: fix tests for Mercurial 3.6 changes; r?smacleod
Attachment #8698269 - Flags: review?(smacleod)
ansible/hg-web: don't try to syntax highlight for unknown file types; r?smacleod

The default behavior is almost always wrong. I added this feature
upstream and it is available in Mercurial 3.6.
Attachment #8698270 - Flags: review?(smacleod)
testing: use `hgmo clean`

"stop" stops containers gracefully. "clean" forces them to shut down.
This shaves ~10s off the run time of *each* test on my machine.
Comment on attachment 8698250 [details]
MozReview Request: hgtemplates: remove gitweb_mozilla_pvt; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27863/diff/1-2/
Comment on attachment 8698251 [details]
MozReview Request: serverlog: use absolute_import; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27865/diff/1-2/
Comment on attachment 8698252 [details]
MozReview Request: serverlog: refactor repo path logic into standalone function; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27867/diff/1-2/
Comment on attachment 8698253 [details]
MozReview Request: serverlog: record syslog attributes on serverlog dict; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27869/diff/1-2/
Comment on attachment 8698254 [details]
MozReview Request: serverlog: extract syslog logging to own function; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27871/diff/1-2/
Comment on attachment 8698255 [details]
MozReview Request: serverlog: stop calling _syslog from syslogmixin; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27873/diff/1-2/
Comment on attachment 8698256 [details]
MozReview Request: serverlog: remove syslogmixin; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27875/diff/1-2/
Comment on attachment 8698257 [details]
MozReview Request: serverlog: remove localrepository wrapping; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27877/diff/1-2/
Comment on attachment 8698258 [details]
MozReview Request: serverlog: tidy serverlog variable references; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27879/diff/1-2/
Comment on attachment 8698259 [details]
MozReview Request: ansible: mass upgrade to Mercurial 3.6.2 (bug 1228086); r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27881/diff/1-2/
Comment on attachment 8698260 [details]
MozReview Request: ansible: use built-in clonebundles extension; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27883/diff/1-2/
Comment on attachment 8698261 [details]
MozReview Request: serverlog: make compatible with Mercurial 3.6; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27885/diff/1-2/
Comment on attachment 8698262 [details]
MozReview Request: hgtemplates: synchronize templates with Mercurial 3.6.2; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27887/diff/1-2/
Comment on attachment 8698263 [details]
MozReview Request: hgtemplates: synchronize with gitweb (bug 1228086); r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27889/diff/1-2/
Comment on attachment 8698269 [details]
MozReview Request: hgserver: fix tests for Mercurial 3.6 changes; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27901/diff/1-2/
Comment on attachment 8698264 [details]
MozReview Request: generate-hg-s3-bundles: use hg from virtualenv; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27891/diff/1-2/
Comment on attachment 8698265 [details]
MozReview Request: generate-hg-s3-bundles: specify bundle version type; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27893/diff/1-2/
Comment on attachment 8698266 [details]
MozReview Request: generate-hg-s3-bundles: support for optionally generating bzip2 bundles; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27895/diff/1-2/
Comment on attachment 8698267 [details]
MozReview Request: generate-hg-s3-bundles: only generate bzip2 bundles for release repos; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27897/diff/1-2/
Comment on attachment 8698268 [details]
MozReview Request: testing: don't install Mercurial 3.1 and 3.2; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27899/diff/1-2/
Comment on attachment 8698270 [details]
MozReview Request: ansible/hg-web: don't try to syntax highlight for unknown file types; r?smacleod

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27903/diff/1-2/
hgwsgi: don't enable demandimport; r?smacleod

demandimport exists so `hg` client commands run faster. In server WSGI
environments, this performance benefit is not needed. Furthermore, in
server environments, demandimport opens up the door to franken-installs
where some .py files come from one release and other .py files another
release. It's best to leave disable demandimport disabled.
Attachment #8698285 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27863/diff/2-3/
Attachment #8698614 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27865/diff/2-3/
Attachment #8698615 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27873/diff/2-3/
Attachment #8698619 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27875/diff/2-3/
Attachment #8698620 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27877/diff/2-3/
Attachment #8698621 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27879/diff/2-3/
Attachment #8698622 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27883/diff/2-3/
Attachment #8698624 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27885/diff/2-3/
Attachment #8698627 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27901/diff/2-3/
Attachment #8698630 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27891/diff/2-3/
Attachment #8698631 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27899/diff/2-3/
Attachment #8698635 - Flags: review?(smacleod)
Review request updated; see interdiff: https://reviewboard.mozilla.org/r/27915/diff/1-2/
Attachment #8698637 - Flags: review?(smacleod)
Comment on attachment 8698250 [details]
MozReview Request: hgtemplates: remove gitweb_mozilla_pvt; r?smacleod

https://reviewboard.mozilla.org/r/27863/#review25283

rubber stamp? :P
Attachment #8698250 - Flags: review?(smacleod) → review+
Comment on attachment 8698251 [details]
MozReview Request: serverlog: use absolute_import; r?smacleod

https://reviewboard.mozilla.org/r/27865/#review25285
Attachment #8698251 - Flags: review?(smacleod) → review+
Comment on attachment 8698252 [details]
MozReview Request: serverlog: refactor repo path logic into standalone function; r?smacleod

https://reviewboard.mozilla.org/r/27867/#review25287
Attachment #8698252 - Flags: review?(smacleod) → review+
Attachment #8698253 - Flags: review?(smacleod) → review+
Comment on attachment 8698253 [details]
MozReview Request: serverlog: record syslog attributes on serverlog dict; r?smacleod

https://reviewboard.mozilla.org/r/27869/#review25387
Attachment #8698254 - Flags: review?(smacleod) → review+
Comment on attachment 8698254 [details]
MozReview Request: serverlog: extract syslog logging to own function; r?smacleod

https://reviewboard.mozilla.org/r/27871/#review25389
Comment on attachment 8698255 [details]
MozReview Request: serverlog: stop calling _syslog from syslogmixin; r?smacleod

https://reviewboard.mozilla.org/r/27873/#review25393
Attachment #8698255 - Flags: review?(smacleod) → review+
Comment on attachment 8698256 [details]
MozReview Request: serverlog: remove syslogmixin; r?smacleod

https://reviewboard.mozilla.org/r/27875/#review25395
Attachment #8698256 - Flags: review?(smacleod) → review+
Comment on attachment 8698257 [details]
MozReview Request: serverlog: remove localrepository wrapping; r?smacleod

https://reviewboard.mozilla.org/r/27877/#review25401
Attachment #8698257 - Flags: review?(smacleod) → review+
Comment on attachment 8698258 [details]
MozReview Request: serverlog: tidy serverlog variable references; r?smacleod

https://reviewboard.mozilla.org/r/27879/#review25407
Attachment #8698258 - Flags: review?(smacleod) → review+
Comment on attachment 8698259 [details]
MozReview Request: ansible: mass upgrade to Mercurial 3.6.2 (bug 1228086); r?smacleod

https://reviewboard.mozilla.org/r/27881/#review25411

::: create-test-environment:89
(Diff revision 3)
> -for old in 2.5.4 2.6 2.6.1 2.6.2 2.6.3 2.7 2.7.1 2.7.2 2.8 2.8.1 2.8.2 2.9 2.9.1 2.9.2 3.0 3.0.1 3.0.2 3.1 3.1.1 3.2 3.2.1 3.2.2 3.2.3 3.3 3.3.2 3.4 3.4.1 3.5.1 3.6; do
> +for old in 2.5.4 2.6 2.6.1 2.6.2 2.6.3 2.7 2.7.1 2.7.2 2.8 2.8.1 2.8.2 2.9 2.9.1 2.9.2 3.0 3.0.1 3.0.2 3.1 3.1.1 3.2 3.2.1 3.2.2 3.2.3 3.3 3.3.2 3.4 3.4.1 3.5.1 3.5.2 3.6; do

Why have you added 3.5.2 here?
Attachment #8698259 - Flags: review?(smacleod) → review+
Comment on attachment 8698260 [details]
MozReview Request: ansible: use built-in clonebundles extension; r?smacleod

https://reviewboard.mozilla.org/r/27883/#review25413
Attachment #8698260 - Flags: review?(smacleod) → review+
Attachment #8698261 - Flags: review?(smacleod) → review+
Comment on attachment 8698261 [details]
MozReview Request: serverlog: make compatible with Mercurial 3.6; r?smacleod

https://reviewboard.mozilla.org/r/27885/#review25421
Comment on attachment 8698262 [details]
MozReview Request: hgtemplates: synchronize templates with Mercurial 3.6.2; r?smacleod

https://reviewboard.mozilla.org/r/27887/#review25423
Attachment #8698262 - Flags: review?(smacleod) → review+
Comment on attachment 8698263 [details]
MozReview Request: hgtemplates: synchronize with gitweb (bug 1228086); r?smacleod

https://reviewboard.mozilla.org/r/27889/#review25429
Attachment #8698263 - Flags: review?(smacleod) → review+
Attachment #8698269 - Flags: review?(smacleod) → review+
Comment on attachment 8698269 [details]
MozReview Request: hgserver: fix tests for Mercurial 3.6 changes; r?smacleod

https://reviewboard.mozilla.org/r/27901/#review25431
Attachment #8698264 - Flags: review?(smacleod) → review+
Comment on attachment 8698264 [details]
MozReview Request: generate-hg-s3-bundles: use hg from virtualenv; r?smacleod

https://reviewboard.mozilla.org/r/27891/#review25435
Attachment #8698265 - Flags: review?(smacleod) → review+
Comment on attachment 8698265 [details]
MozReview Request: generate-hg-s3-bundles: specify bundle version type; r?smacleod

https://reviewboard.mozilla.org/r/27893/#review25437
Comment on attachment 8698266 [details]
MozReview Request: generate-hg-s3-bundles: support for optionally generating bzip2 bundles; r?smacleod

https://reviewboard.mozilla.org/r/27895/#review25439
Attachment #8698266 - Flags: review?(smacleod) → review+
Comment on attachment 8698267 [details]
MozReview Request: generate-hg-s3-bundles: only generate bzip2 bundles for release repos; r?smacleod

https://reviewboard.mozilla.org/r/27897/#review25441
Attachment #8698267 - Flags: review?(smacleod) → review+
Comment on attachment 8698268 [details]
MozReview Request: testing: don't install Mercurial 3.1 and 3.2; r?smacleod

https://reviewboard.mozilla.org/r/27899/#review25445
Attachment #8698268 - Flags: review?(smacleod) → review+
Comment on attachment 8698270 [details]
MozReview Request: ansible/hg-web: don't try to syntax highlight for unknown file types; r?smacleod

https://reviewboard.mozilla.org/r/27903/#review25447
Attachment #8698270 - Flags: review?(smacleod) → review+
Comment on attachment 8698285 [details]
MozReview Request: hgwsgi: don't enable demandimport; r?smacleod

https://reviewboard.mozilla.org/r/27915/#review25449
Attachment #8698285 - Flags: review?(smacleod) → review+
https://reviewboard.mozilla.org/r/27881/#review25411

> Why have you added 3.5.2 here?

Oops. Will fix. Good spot.
Comment on attachment 8698614 [details]
MozReview Request: hgtemplates: remove gitweb_mozilla_pvt; r?smacleod

MozReview got confused.
Attachment #8698614 - Flags: review?(smacleod)
Attachment #8698637 - Flags: review?(smacleod)
Attachment #8698627 - Attachment is obsolete: true
Attachment #8698627 - Flags: review?(smacleod)
Attachment #8698615 - Attachment is obsolete: true
Attachment #8698615 - Flags: review?(smacleod)
Attachment #8698636 - Attachment is obsolete: true
Attachment #8698636 - Flags: review?(smacleod)
Attachment #8698635 - Attachment is obsolete: true
Attachment #8698635 - Flags: review?(smacleod)
Attachment #8698634 - Attachment is obsolete: true
Attachment #8698634 - Flags: review?(smacleod)
Attachment #8698633 - Attachment is obsolete: true
Attachment #8698633 - Flags: review?(smacleod)
Attachment #8698632 - Attachment is obsolete: true
Attachment #8698632 - Flags: review?(smacleod)
Attachment #8698631 - Attachment is obsolete: true
Attachment #8698631 - Flags: review?(smacleod)
Attachment #8698630 - Attachment is obsolete: true
Attachment #8698630 - Flags: review?(smacleod)
Attachment #8698629 - Attachment is obsolete: true
Attachment #8698629 - Flags: review?(smacleod)
Attachment #8698628 - Attachment is obsolete: true
Attachment #8698628 - Flags: review?(smacleod)
Attachment #8698616 - Attachment is obsolete: true
Attachment #8698616 - Flags: review?(smacleod)
Attachment #8698617 - Attachment is obsolete: true
Attachment #8698617 - Flags: review?(smacleod)
Attachment #8698618 - Attachment is obsolete: true
Attachment #8698618 - Flags: review?(smacleod)
Attachment #8698619 - Attachment is obsolete: true
Attachment #8698619 - Flags: review?(smacleod)
Attachment #8698620 - Attachment is obsolete: true
Attachment #8698620 - Flags: review?(smacleod)
Attachment #8698621 - Attachment is obsolete: true
Attachment #8698621 - Flags: review?(smacleod)
Attachment #8698622 - Attachment is obsolete: true
Attachment #8698622 - Flags: review?(smacleod)
Attachment #8698623 - Attachment is obsolete: true
Attachment #8698623 - Flags: review?(smacleod)
Attachment #8698624 - Attachment is obsolete: true
Attachment #8698624 - Flags: review?(smacleod)
Attachment #8698637 - Attachment is obsolete: true
Attachment #8698614 - Attachment is obsolete: true
hg.mozilla.org and reviewboard-hg.mozilla.org are now running Mercurial 3.6.2.

There was an outage of the Mercurial wire protocol (as opposed to the HTML interface) for a few minutes. This created some failures in automation.

The outage was caused by the Mercurial 3.6 short circuit check in our vendored clonebundles.py. While this patch series removed that file, the deployment process upgraded the Mercurial client packages to 3.6 before removing and changing the reference to that file and the bomb exploded. I manually unwedged things and after a few tries, the automated deployment completed.

So much for my outage free production upgrade streak :/
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.