Update hg.mozilla.org to recent Mercurial (2.0.2)

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
7 years ago
2 years ago

People

(Reporter: timeless, Assigned: cshields)

Tracking

Dependency tree / graph
Bug Flags:
needs-treeclosure +

Details

(Whiteboard: [downtime 1/13 9am-12pm PST], URL)

Attachments

(1 obsolete attachment)

(Reporter)

Description

7 years ago
Currently we're on hg 1.5.4 (something 6 months old). Hg 1.7.3 was just released, we should at least get to 1.7.something so we're not on an old version.

To do this, someone needs to test our hooks+extensions+templates ala bug 511305 and ensure everything works fine (and passes all the unit tests).

Upgrading to 1.7+ gives support for pushable bookmarks
http://mercurial.selenic.com/wiki/BookmarksExtension

please? thanks :)
Just to be sure, don't deploy 1.7 (1.7.1 is an unscheduled bugfix release). But 1.7.3 probably makes the most sense anyway.
I had to update the hgpoller test harness to work with hg 1.7, but after doing that all the hook and pushlog tests pass for me with:
Mercurial Distributed SCM (version 1.7.3+2-5d1bb1174047)

Updated

7 years ago
Assignee: server-ops → aravind
Toss this back to us, when you guys are happy it all works well?  You have access to the staging server so feel free to hack on things there all you want.
Assignee: aravind → nobody
Component: Server Operations → Hg: Customizations
QA Contact: mrz → hg.customizations
While everyone's at it... there's Mercurial 1.7.5 for some weeks now, and possibly another version once March comes along...
If it matters at all, all the build slaves are going to be 1.6 or 1.7 shortly.

Updated

6 years ago
Depends on: 666870
Summary: Update hg.mozilla.org to Mercurial 1.7.3 → Update hg.mozilla.org to Mercurial 1.9.1
Blocks: 653625
Investigating a possible 1.9 upgrade, I've identified a problem caused by something in our templates.  Attempting to serve hgweb with our templates enabled results in the following exception. Using a standard template resolves the problem.  I haven't had a chance yet to investigate in depth.

mod_wsgi (pid=1919): Exception occurred processing WSGI script '/repo/hg/webroot_wsgi/hgwebdir.wsgi'.
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/mercurial/util.py", line 247, in increasingchunks
    for chunk in source:
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 214, in _flatten
    for j in _flatten(i):
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 207, in _flatten
    for i in thing:
  File "/usr/lib/python2.4/site-packages/mercurial/hgweb/hgweb_mod.py", line 245, in header
    yield tmpl('header', encoding=encoding.encoding, **map)
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 329, in __call__
    stream = proc.process(t, mapping)
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 263, in process
    return _flatten(func(self, mapping, data) for func, data in
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 256, in _load
    self._cache[t] = compiletemplate(self._loader(t), self)
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 93, in compiletemplate
    parseres, pos = p.parse(pd)
  File "/usr/lib/python2.4/site-packages/mercurial/parser.py", line 82, in parse
    res = self._parse()
  File "/usr/lib/python2.4/site-packages/mercurial/parser.py", line 42, in _parse
    token, value, pos = self._advance()
  File "/usr/lib/python2.4/site-packages/mercurial/parser.py", line 31, in _advance
    self.current = self._iter.next()
  File "/usr/lib/python2.4/site-packages/mercurial/templater.py", line 70, in tokenizer
    raise error.ParseError(_("syntax error"), pos)
ParseError: ('syntax error', 469)

Comment 7

6 years ago
Do you have the patch from bug 666870 applied? https://bugzilla.mozilla.org/attachment.cgi?id=541630&action=diff#a/gitweb_mozilla/header.tmpl_sec1 changes the header template, which this seems to stumble upon.
Cool, that patch seems to fix the problem on my test system. Thanks Axel.
Created attachment 555466 [details] [diff] [review]
Patch against http://hg.mozilla.org/hgcustom/hg_templates/

I've refreshed the patch from bug 666870 such that it applies against the current contents of http://hg.mozilla.org/hgcustom/hg_templates/ with their inline JS and CSS.
Mercurial 2.0 is about to be released probably at the start of November:

http://mercurial.selenic.com/wiki/WhatsNew#Mercurial_2.0_.282011-11-01.29
I think upgrading will greatly improve how long it takes to push to try. See http://atlee.ca/blog/2011/12/09/hg-performance/
Summary: Update hg.mozilla.org to Mercurial 1.9.1 → Update hg.mozilla.org to recent Mercurial
This is a server-ops task.

At the moment, this looks like something we could mix in during the move to scl3, which means Q1/Q2 of 2012.  Doing it earlier than that will preclude already-defererd maintenance and refactoring that we really need to take care of.
Assignee: nobody → server-ops
Component: Hg: Customizations → Server Operations
QA Contact: hg.customizations → cshields
Punting to cshields, this is for SCL3.
Assignee: server-ops → cshields
Whiteboard: SCL3
Generally what we've done to sanity check a new version of Mercurial is:
1) Run the unit tests in the hghooks repository ( http://hg.mozilla.org/hgcustom/hghooks , it's just runtests.py) with the new version, verify that they all pass.
2) Run the unit tests in the pushlog repository ( http://hg.mozilla.org/hgcustom/pushlog/ , runtests.py again) with the new version, verify that they all pass
3) Setup the new version with the hooks + extensions + templates on a staging server (Aravind created hgstage a while ago for this purpose) and manually verify that things look okay in the HTML interface.

The first two steps should verify that none of our custom hooks or extensions are broken. The third step is just a sanity check because it's hard to test that the HTML templates produce proper output.
Mozilla hg infrastructure suffers from a lack of internal documentation.  

So what sounds like a simple upgrade isn't.  We're working on back filling documentation but I'm real hesitant to push hard on this because a) it's working and b) no documentation and c) if it breaks and we can't back out correctly, we have big issues.

Incidentally, we just learned there's a staging hg server (yep, no documentation).  This might help things go quicker but it's sat dormant since Aravind left and I'm unsure of the state.
Comment on attachment 555466 [details] [diff] [review]
Patch against http://hg.mozilla.org/hgcustom/hg_templates/

I pushed this on a new hg_1_6+_updates branch:
http://hg.mozilla.org/hgcustom/hg_templates/rev/68902204d68b

I'll merge it to default when we're ready to roll out the new hg to production.
Attachment #555466 - Attachment is obsolete: true
Okay, I've run the hghooks tests and the pushlog tests with hg 2.0.1 on my local system, and they all pass (after applying that templates patch).

This was on:
Mercurial Distributed SCM (version 2.0.1+32-32a6e00e4cfe)
(Assignee)

Comment 18

6 years ago
going to draft up upgrade/backout docs and see if we can do this soon.  Will touch base with ted tomorrow on some additional requests he had for testing on irc when we got distracted by an outage.
per email w/cshields, this might be ready to happen next week - pending some last testing. cshields will let us know for certain once he's ready. Flagging for downtime, so buildduty are aware.
Flags: needs-treeclosure?
Corey: we need a downtime this week for bug 715026 and bug 715478 - would this be ready for a ride-along on that downtime?
Flags: needs-treeclosure? → needs-treeclosure+
Whiteboard: SCL3 → SCL3 [downtime 1/13 9am-12pm PST]
(Assignee)

Comment 21

6 years ago
We will be doing this with the tree closure on 1/13.

For those of you interested I've outlined the steps we'll be following here:

https://etherpad.mozilla.org/hg-2-0-1-upgrade

Backchannel will be #it for the hg portion of the downtime.

During the downtime, external ssh based connections will fail and http connections will show a hardhat page.  Anyone connected to a VPN at the time may still be able to pull, tree closure hooks should prevent anything from pushing, but in any case I'd appreciate no attempts either way until the maintenance is done.
Whiteboard: SCL3 [downtime 1/13 9am-12pm PST] → [downtime 1/13 9am-12pm PST]
FYI: The current Mercurial version is 2.0.2 released on 01-01-2012. 
http://mercurial.selenic.com/wiki/WhatsNew#Mercurial_2.0.2_.282012-01-01.29

Is it too late to use that version?
(Assignee)

Comment 23

6 years ago
(In reply to pretzer from comment #22)
> FYI: The current Mercurial version is 2.0.2 released on 01-01-2012. 
> http://mercurial.selenic.com/wiki/WhatsNew#Mercurial_2.0.2_.282012-01-01.29
> 
> Is it too late to use that version?

I'll try to get this built and tested before the window.
This update seems to be causing raw HTML files to be served as application/binary; before the update they were served as text/html.  Example url:

http://hg.mozilla.org/services/services-central/raw-file/2152f7c1273e/testing/tps/pages/page1.html

Is this a deliberate side-effect of the upgrade?
(Assignee)

Comment 25

6 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #24)
> This update seems to be causing raw HTML files to be served as
> application/binary; before the update they were served as text/html. 
> Example url:
> 
> http://hg.mozilla.org/services/services-central/raw-file/2152f7c1273e/
> testing/tps/pages/page1.html
> 
> Is this a deliberate side-effect of the upgrade?

Yes, as of 1.9.1 that is the behavior, and we are now at 2.0.2. See http://mercurial.selenic.com/wiki/UpgradeNotes

Well, this upgrade wasn't as clean as I would have liked but it is done for now.  We still have a lot of work to do on revamping and improving the hg infrastructure but I hope this will help.

There were a lot of hgweb changes (nothing real cosmetic - mostly shifts in backend files) so if something appears to look strange file a bug for it.  In addition if something does not appear at all where it use to, there may be a stuck pushlog entry - we had a bit of time where hooks were executing out of order and ended up clogging some trees.

Thanks for assistance goes out to ted, callek, catlee, lsblakk, philor, atoll, fox2mike
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Depends on: 718155

Updated

6 years ago
Depends on: 718172

Updated

6 years ago
Depends on: 718186

Updated

6 years ago
Depends on: 718221

Updated

6 years ago
Depends on: 718241

Updated

6 years ago
Depends on: 718251
Depends on: 718354
Depends on: 718492
(Assignee)

Updated

6 years ago
Depends on: 718562
Summary: Update hg.mozilla.org to recent Mercurial → Update hg.mozilla.org to recent Mercurial (2.0.2)
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.