671357 - Update wiki.mo to Mediawiki 1.17.0

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Deb Richardson [:dria] (plz NEEDINFO)]

Wiki.mo is a version old now (1.16.5), and there's some fixes in 1.17.0 that I could really use.

Any chance we could get wiki.mo updated soonish?  There's just a stupid JS bug they fixed that is making life complicated right now :)

Comment 1

[Jake Maul [:jakem]]

From the blog post about the new version:

"MediaWiki 1.17 is a very large release that contains many new features and bug fixes."

This is somewhat scary to me. In particular it looks like neither Semantic MediaWiki nor Semantic Forms has been seriously tested with 1.17.0 (SMW claims support up to 1.17alpha, but not release... SMF just says "1.14 or higher"). I don't think it would break much, but them calling it a "very large release" and not providing reassurance of a smooth upgrade process is unsettling.

There's been a good bit of push-back lately from wiki breakage. Because of that we're actually planning to set up a staging deployment of it. This would be a perfect trial run for it. QA is even planning on helping us out with some Selenium tests. The only issue is, I doubt those automated tests would be ready any time soon.

Since you're one of the bigger users of wikimo, would you be willing to test out the stage site a bit after the upgrade, to make sure it seems to be working right for you?

Comment 2

[Deb Richardson [:dria] (plz NEEDINFO)]

Absolutely -- happy to help however I can.  

Also, I've worked around the JS brokenness that I needed the update for, so this is less urgent now.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Until we can get Selenium coverage (thinking Selenium 2, which we haven't yet written in, but for which this would be a good working exercise), we can agree to manually test.

I've conjured up https://wiki.mozilla.org/QA/Execution/Web_Testing/Wiki_Test_Coverage to start planning a coverage checklist.  Jake/Dave/Matt/Deb, let's flesh it out and then tag-team it?

Comment 4

[Jake Maul [:jakem]]

I've had a look at that test coverage page, and I'm not sure what more to add. Looks good to me!

As far as what has broken in the past, I think Deb would be in a better position than me to say for sure. I know there's heat when something breaks, but I can't recall a specific breakage we could test for that isn't already covered by the things already listed there.

One thing worth mentioning is that the test shouldn't go to a specific URL to log in, it should try to click the login button/link/whatever... just to make sure it's properly visible. But, I'm sure you'd thought of that already. :)

Comment 5

[Deb Richardson [:dria] (plz NEEDINFO)]

I've actually not known of any past breakages other than some minor issues with recent theme changes, but those were just CSS fubars and easy to fix.

My key concern is making sure the Semantic Mediawiki + Semantic Mediawiki Forms add-ons don't break with the new system, but I'll test those in great detail on stage.

Comment 6

[Jake Maul [:jakem]]

wiki.allizom.org is now operational. I've sent out login credentials for it. Let me know if it appears to be fully functional, and I can update it to 1.17.0 for the real test. I just don't want to confuse the issue as to whether a particular problem is something I goofed up on the creation of the stage site, or an actual issue with the 1.17.0 update. :)

Comment 7

[Deb Richardson [:dria] (plz NEEDINFO)]

I'm not going to have time to work on this until early next week (Monday if at all possible, Tues at the latest).

Is that ok?  If not I can try to carve out some time on the weekend, but I'd rather not :)

Comment 8

[Jake Maul [:jakem]]

I'll try to do some preliminary testing. I already know login/logout works, but haven't done any editing or creating/deleting of pages.

I don't know anything about how the semantic extensions work. Not sure I can adequately test those... at least not by the time you'd be able to do a better job anyway.

In any case, I think you're the only one who's expressed interest in a newer version, and it's not advertised as a security update. So this isn't really holding up anyone but you. :)

Comment 9

[Jake Maul [:jakem]]

Deb has tested the new stage site (wiki.allizom.org) for initial functionality, and no obvious/major breakage occurred. I'm upgrading it to 1.17.0, and then we can test the upgrade.

Comment 10

[Jake Maul [:jakem]]

For the record, while doing the upgrade on wiki.allizom.org, I can confidently say that when we do the prod upgrade, there will likely be a few minutes of intermittent connectivity to the wiki, and there will definitely be a period of read-only activity... not sure how long yet that will be.

For my own notes, here's the rough process. This is based on the upgrade of wiki.allizom.org, using the instructions here as a guide:

http://www.mediawiki.org/wiki/Manual:Upgrading_MediaWiki

1) Check https://wiki.mozilla.org/Special:Version, note what's there
2) make a new dir, 'svn checkout' the new branch to it
     'svn switch' is not a reliable upgrade mechanism (via trial and error)
3) Save a copy of images/.svn that gets put in this new dir
4) copy over skins-mozilla, LocalSettings.php, and the images symlink
5) Restore the new images/.svn on top of the one in the existing symlink
6) Copy over old/extensions dir to new/extensions
     Do *not* overwrite things in .svn... keep the copy from upstream
7) edit LocalSettings.php, add this to the end:
     $wgReadOnly = 'Upgrading to MediaWiki 1.17.0';
8) compare permissions in the old and new dir, fix discrepancies
     the 'cache' dir is known to need updating, might be others
9) move the original dir out of the way, move the new one into its place
     site should now be up on the new file version, but in read-only mode
     logins should still work, but editing won't
10) cd maintenance; php update.php
     this takes a long time
11) Test

Check this page while the update is running:
https://wiki.mozilla.org/Special:Version
Make sure all the right stuff is still there, and the version has bumped.

Comment 11

[Jake Maul [:jakem]]

Step 9.5)
In mysql:
GRANT REPLICATION CLIENT ON *.* TO 'mozwiki'@'10.2.81.%'
Run the update.php script *from one of the pm-app-genericXX servers*, not from mradm02!

After it completes (over 30min most likely),
REVOKE REPLICATION CLIENT ON *.* FROM 'mozwiki'@'10.2.81.%'


The update.php calls code that has a check to make sure the slaves are semi-caught-up before doing too much, but this relies on these privileges to work. I think we can realistically leave this in place long-term... REPLICATION CLIENT is a read-only privilege, and IMO isn't all that sensitive.

step 10.5) comment out the $wgReadOnly setting from step 7

Comment 12

[Jake Maul [:jakem]]

The update completed on wiki.allizom.org, but the database is stuck in read-only mode. I'm trying to get help from the mediawiki support guys... no luck so far.

Comment 13

[Jake Maul [:jakem]]

wiki.allizom.org is operational with MediaWiki 1.17.0!

Please let me know if anything significant is broken. If it looks good we can push wiki.mozilla.org next week. :)

Comment 14

[Deb Richardson [:dria] (plz NEEDINFO)]

If I get a chance I'll check this over the weekend, otherwise it'll be first thing Monday. Thanks!

Comment 15

[Deb Richardson [:dria] (plz NEEDINFO)]

crap sorry, totally forgot to get back to this.  will test ASAP.

Comment 16

[Deb Richardson [:dria] (plz NEEDINFO)]

Just started testing on allizom and I notice that the custom CSS stuff doesn't seem to be working properly on it.  For example, this page:

   https://wiki.allizom.org/Features/Desktop

should look like this page:

   https://wiki.mozilla.org/Features/Desktop

It's hard for me to test further since most of the pages I'll be testing involve custom CSS like that.

The custom CSS stuff is defined on this page (for the Gmo skin):

   https://wiki.allizom.org/MediaWiki:Gmo.css

Comment 18

[Deb Richardson [:dria] (plz NEEDINFO)]

I know it's release day, but let me know when you might be able to sort out the CSS issues on allizom so I can schedule some time to keep testing.  Thanks :)

Comment 19

[Jake Maul [:jakem]]

Milos is helping out with this... he does a good bit of our skinning work with Mediawiki / Wordpress. I haven't been able to isolate an obvious cause as to why they're different.

Comment 20

[Deb Richardson [:dria] (plz NEEDINFO)]

Is it an exact mirror of the existing site?  Are the config files the same?  I'm away next week (and everyone'll be at allhands anyhow), but I'd really like to be have a reliable staging server for wiki.mo as soon as we can after that.  Any way we can get the priority on this raised?

Comment 21

[Jake Maul [:jakem]]

This has been a while, but my guess is that something in MediaWiki 1.17.0 is incompatible with our CSS.

One thing I see that's different between the two of them now is that wiki.allizom.org does not load some of the same files as wiki.mozilla.org. The following things are missing in the page load of wiki.allizom.org:

skins-mozilla/common/shared.css
skins-mozilla/common/jquery.min.js
skins-mozilla/common/ajax.js

The load order of CSS/JS files between the two are different as well. Prod loads commonPrint.css twice, whereas wiki.allizom.org loads it only once. I will attach a file that shows the load order differences between them.

I'm wondering if this is an issue with the SVN checkouts of the two different environments:

wiki.mozilla.org/skins-mozilla:
URL: http://svn.mozilla.org/projects/wikimo/tags/production
Last Changed Author: [e-mail]
Last Changed Rev: 94770
Last Changed Date: 2011-09-06 15:04:01 -0700 (Tue, 06 Sep 2011)

wiki.allizom.org/skins-mozilla:
URL: http://svn.mozilla.org/projects/wikimo/trunk/skins
Last Changed Author: [e-mail]
Last Changed Rev: 94329
Last Changed Date: 2011-08-25 17:12:00 -0700 (Thu, 25 Aug 2011)


Milos, would it be safe to change this wiki.allizom.org over to use tags/production instead of trunk?

Comment 22

[Milos Dinic [:Milos]]

Jake,

Yes, it wouldn't hurt. One thing though:

I just checked for differences in those two repos, and I got nothing. So, it's probably a prob with allizom mediawiki config or something else, even.

`diff -r -x "*.diff" trunk/skins/ prod/ > diff.diff`

Comment 23

[Jake Maul [:jakem]]

Attachment: skin differences between wiki.allizom.org and wiki.mozilla.org

Here is how the skins-mozilla directory loads differently on wiki.mozilla.org vs wiki.allizom.org. As you can see, some things are missing outright. For instance, common/shared.css is not loaded at all on the stage site... nor is common/jquery.min.js or common/ajax.js.

Comment 24

[Deb Richardson [:dria] (plz NEEDINFO)]

Any chance we can get this sorted out? I would really like to get things tested and updated so we can look at installing new extensions.

Comment 25

[Corey Shields [:cshields]]

How hard would it be to roll this update into the migration to phx1?

Jake, where did you setup allizom?  can/should we move it to genericrhel6 in phx1?  If this is going to be a bit of a mess + some downtime maybe we can minimize the downtime by doing the migration (a DNS flip)

Comment 26

[Jake Maul [:jakem]]

wiki.allizom.org is set up on mrapp-stage04. It is already running 1.17.0. The problem does not seem to be one of deployment or infra, but rather our theme appears to not work quite the same on 1.17.0 as it does on the older version.

Perhaps this bug should move to another product/component to give more weight to that... I will do that. Milos is the usual wiki theme editor for us, so I'll assign to him.

As for moving this to phx1, I have no problem with that... it should work just fine on genericrhel6-stage. But a migration to new hardware will not help with this upgrade.

Comment 27

[Milos Dinic [:Milos]]

Jake,

I'm at wiki.allizom.org now, and I don't see anything wrong with it. Mind pointing to exact problems so that I can fix them?

Also, I've been fixing some bugs on trunk a week ago, so that may have affected this bug, too.

Comment 28

[Deb Richardson [:dria] (plz NEEDINFO)]

Milos, see comment #16 and onward, there are links to example pages where the CSS isn't working and some further information.

Comment 29

[Milos Dinic [:Milos]]

Deb,

One thing I can't figure out is this Gmo.css file. Where did you get this? Is that a custom CSS file you created using wiki interface?

Comment 30

[Deb Richardson [:dria] (plz NEEDINFO)]

Yeah, it's a special Mediawiki feature -- I think there's one for every installed skin.  For example, we used to use this one when we were using the Cavendish skin:

https://wiki.mozilla.org/MediaWiki:Cavendish.css

~ d

Comment 31

[Jake Maul [:jakem]]

Upstream docs on this:

http://www.mediawiki.org/wiki/Manual:FAQ#How_do_I_edit_the_wiki.27s_CSS.3F

My guess is that in 1.17.0, something changed in Mediawiki such that our overrides in this file don't quite work anymore. Possibly just some elements got new names, or maybe it's something deeper.

Comment 32

[Jake Maul [:jakem]]

Taking this back. It's fixed!

Milos helped me figure this out. Apparently in the new version of MediaWiki, custom stylesheets like this are loaded differently. Instead of the old way, they are now loaded through a new ResourceLoader script, load.php.

Our Apache configs have a special RewriteRule in them designed to map any non-existant things over to MediaWiki, as if it was supposed to be a MediaWiki page. That way you can go to wiki.mozilla.org/some-random-page, and it will just work. If that wiki page doesn't exist, Mediawiki will throw a 404 and load the "would you like to make this page?" page.

This is what tipped us off finally... load.php existed on the system, but was still generating that 404.

This file (load.php) needed to be added as one of the non-redirected files in that RewriteRule. Once that was done, the 404 went away and the custom stylesheet started loading properly.

The links in comment 16 look good to me now. Deb, can you confirm if we're all clear to upgrade prod to 1.17.0? If so, we can do that on Monday. :)

Comment 33

[Milos Dinic [:Milos]]

When upgrading to prod, just keep in mind that I pushed some fixes to trunk, in r96425/r96427/r96442/r96446/r96449.

I don't really see anything that might need a security review, but feel free to ask for it if you see the need for it.

Comment 34

[Jake Maul [:jakem]]

Those look fine to me. Go ahead and push to tags/production whenever you're ready for them to be live. If wiki.allizom.org looks correct now, then we can push these fixes that before, during, or after the 1.17.0 upgrade... doesn't matter to me.

Comment 35

[Deb Richardson [:dria] (plz NEEDINFO)]

Sorry -- didn't realize I was blocking this.  Just verified that the CSS issues I pointed out earlier have been fixed on allizom.  Thanks!

Comment 36

[Dave Miller [:justdave]]

FYI. 1.16.x EOLed today.

Comment 37

[Dave Miller [:justdave]]

This bug should be opened to the public if there's not anything in it that needs to stay hidden.

Comment 38

[Milos Dinic [:Milos]]

Jake, Deb,

Are we good to go?

If so, let's just go for it.

Comment 39

[Jake Maul [:jakem]]

I'm good... I will work on this today. As far as I know we've run out of known issues to tackle in staging.

Comment 40

[Jake Maul [:jakem]]

This is completed (finally)!

Downtime should have been less than 30 seconds. Read-only time was less than 15 min.

New version is 1.17.1. Sadly, there is a 1.18 due out soon... although I gather it's a bit smaller update, and now that this one has kinda documented the procedure, I'm hoping it will be quite a bit easier to do that one.