Closed Bug 531977 Opened 15 years ago Closed 14 years ago

Update MDC to MindTouch 9.12

Categories

(mozilla.org Graveyard :: Server Operations: Projects, task)

Product:
mozilla.org Graveyard
Component:
Server Operations: Projects
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: sheppy, Assigned: oremj)

References

(
URL
)

Details

Attachments

(1 file)

stack trace from deki-api.log
10.83 KB, text/plain
Details 
We need to upgrade MDC to MindTouch 9.12.
Depends on: 531978
Assignee: server-ops → jeremy.orem+bugs
Looks like 9.12 isn't quite done yet. Let's reopen this when it's closer to release.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → INCOMPLETE
9.12 has been delayed until after the holidays; they decided to pick up a few more changes to improve things for us a little bit more.

In particular, they have backported from their planned next release a patch to let Lucene index rebuilds resume from where they left off after a failure, instead of having to start from scratch every time. This will be a big win for us in cases where we have to restart during a rebuild for whatever reason.

Internal testing is ongoing on these fixes they've backported and they expect to release when they return from the holiday break.
MindTouch 9.12 is final now. The package has not been updated, but SVN will get us the release version. We need to install this onto the test system and sic QA on it.
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Flags: needs-downtime+
Whiteboard: 01/26/2010 @ 7pm
Depends on: 541438
stage has been updated. Are we still on for tonight?
ready to announce this, how long of a window do you need for it?
30 min.
Has QA signed off yet? I thought they were cc'd on this but I just realized they weren't, so my guess is they haven't looked at it yet.
We need to delay this. QA had not in fact tested this; I dropped the ball on asking them to do it (thought I had but had not). I've asked them if they can do it in time for Thursday or Tuesday upcoming. Sorry about this!
If end of day Thursday works for you guys (Eric/IT), then QA can commit to that.
Whiteboard: 01/26/2010 @ 7pm → 02/02/2010 @ 7pm
Are we still going through with this tonight or delaying another 2 days?
(In reply to comment #10)
> Are we still going through with this tonight or delaying another 2 days?

From QA's side, I'm happy with the site, sans still-open bug 543537; Sheppy, it's your call.
Delay it (assuming you haven't already made that decision). I want to get this last thing ironed out. Thanks all!
Whiteboard: 02/02/2010 @ 7pm → 02/04/2010 @ 7pm
Is this happening tonight?
No, we need to sort out this RSS feed problem first, and so far nobody's sure why it's happening.
Whiteboard: 02/04/2010 @ 7pm → 02/09/2010 @ 7pm
What can I do to help them figure this out?
Is there anything informative at all in the PHP log?
[root@sm-devmostage01 devmo.dekiwiki.mozilla.org]# cat error_log_2010-02* | grep -v '/data/www/dekiwiki/robots.txt'  |grep -v 'link target not accessible'
[Sun Jan 31 21:14:16 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Sun Jan 31 21:18:18 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Sun Jan 31 21:18:44 2010] [error] [client 10.2.81.4] Invalid URI in request GET /servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Sun Jan 31 21:19:21 2010] [error] [client 10.2.81.4] Invalid URI in request GET /samples/servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Sun Jan 31 21:21:43 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/info.php' not found or unable to stat
[Sun Jan 31 21:21:43 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/phptest.php' not found or unable to stat
[Sun Jan 31 21:21:44 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/info.php3
[Sun Jan 31 21:21:45 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/dekiwiki/skins/common/logo/
[Sun Jan 31 21:21:48 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/mountain.cfg
[Sun Jan 31 21:22:20 2010] [error] [client 10.2.81.4] script '/data/www/css/phpinfo.php' not found or unable to stat
[Sun Jan 31 21:22:20 2010] [error] [client 10.2.81.4] script '/data/www/css/php.php' not found or unable to stat
[Sun Jan 31 21:22:21 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/phpinfo.php3
[Sun Jan 31 21:22:22 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/phptest.php3
[Sun Jan 31 21:22:23 2010] [error] [client 10.2.81.4] Invalid method in request get /css/ HTTP/1.1
[Sun Jan 31 21:23:14 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Sun Jan 31 21:24:20 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Sun Jan 31 21:25:19 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Sun Jan 31 21:26:09 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Sun Jan 31 21:26:34 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1
[Sun Jan 31 21:31:01 2010] [error] [client 10.2.81.4] Invalid method in request BADMTHD /css/ HTTP/1.0
[Sun Jan 31 21:31:52 2010] [error] [client 10.2.81.4] script '/data/www/css/userinfo.php' not found or unable to stat
[Tue Feb 02 06:15:53 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/admin-serv
[Tue Feb 02 06:26:22 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Tue Feb 02 06:26:45 2010] [error] [client 10.2.81.4] Invalid URI in request GET /servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Tue Feb 02 06:27:41 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/mountain.cfg
[Tue Feb 02 06:28:25 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/config.txt
[Tue Feb 02 06:29:49 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 33353 HTTP/1.1
[Tue Feb 02 06:31:07 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Tue Feb 02 06:32:12 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Tue Feb 02 06:32:18 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1
[Tue Feb 02 06:32:19 2010] [error] [client 10.2.81.4] script '/data/www/css/info.php' not found or unable to stat
[Tue Feb 02 06:32:19 2010] [error] [client 10.2.81.4] script '/data/www/css/phptest.php' not found or unable to stat
[Tue Feb 02 06:32:20 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/info.php3
[Tue Feb 02 06:32:21 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Tue Feb 02 06:33:39 2010] [error] [client 10.2.81.4] Invalid method in request BADMTHD /css/ HTTP/1.0
[Tue Feb 02 06:35:05 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/userinfo.php' not found or unable to stat

cat error_log
[Sun Jan 31 04:02:01 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 31 04:02:01 2010] [notice] Digest: done
[Sun Jan 31 04:02:01 2010] [notice] Apache configured -- resuming normal operations
[Sun Jan 31 18:17:44 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../../../ HTTP/1.1
[Sun Jan 31 18:18:32 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Sun Jan 31 18:33:03 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Sun Jan 31 18:33:09 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../"><script>alert(73541);</script> HTTP/1.1
[Sun Jan 31 18:33:53 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../makebook-2.2/bookfile.html HTTP/1.1
[Sun Jan 31 18:34:04 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Sun Jan 31 18:34:04 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 46614 HTTP/1.1
[Sun Jan 31 18:34:57 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Sun Jan 31 18:35:41 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Tue Feb 02 03:02:07 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../../../ HTTP/1.1
[Tue Feb 02 03:02:11 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Tue Feb 02 03:12:51 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../makebook-2.2/bookfile.html HTTP/1.1
[Tue Feb 02 03:13:44 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 51434 HTTP/1.1
[Tue Feb 02 03:13:44 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Tue Feb 02 03:14:28 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Tue Feb 02 03:15:37 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Tue Feb 02 03:16:27 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1


So no, not really anything informative.
MindTouch wants to confirm that we have PHP level logging enabled to be sure the errors they're wanting to look for aren't being suppressed; can they get a look at our Apache config?
LogLevel isn't set, so it's just default. This is in the Vhost:
	php_value error_reporting "E_ALL & ~E_NOTICE"

I'll bump up the error reporting in LocalSettings
I just added:
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL & ~E_NOTICE);

to the LocalSettings.php.

At first I did error_reporting(E_ALL) and saw a ton of notices, so error reporting is working.
MindTouch support has asked again to see the Apache config for the site.
There are only 3 entries in the logs and I think they were from when notices were on:
[Wed Feb 03 08:36:53 2010] [error] [client 10.2.81.4] PHP Notice:  Use of undefined constant NS_MEDIAWIKI_TALK - assumed 'NS_MEDIAWIKI_TALK' in /data/www/dekiwiki/includes/NamespaceAlternates.php on line 1538
[Wed Feb 03 08:36:53 2010] [error] [client 10.2.81.4] PHP Warning:  Cannot modify header information - headers already sent by (output started at /data/www/dekiwiki/includes/NamespaceAlternates.php:327) in /data/www/dekiwiki/includes/OutputPage.php on line 471
[Wed Feb 03 05:17:24 2010] [error] [client 10.2.81.4] PHP Warning:  Cannot modify header information - headers already sent by (output started at /data/www/dekiwiki/includes/NamespaceAlternates.php:327) in /data/www/dekiwiki/includes/OutputPage.php on line 583
Whiteboard: 02/09/2010 @ 7pm
Whiteboard: [blocked mindtouch]
Re-open when there's IT work to do.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 14 years ago
Resolution: --- → INCOMPLETE
MindTouch has released 9.12.2 today. Please update the staging server to this for testing. Thank you.
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
See http://blog.developer.mindtouch.com/2010/04/06/mindtouch-9122-noatak-released/ for notes; there's a database upgrade script that needs to be run as well as updating the code itself.
Staging server has been updated. Do you want to update production next Tuesday?
I think so, yeah. This should take care of our concerns.

Could you restart the staging server for me? I just changed a config setting that I don't think takes effect until after a restart.
BTW, RSS feeds seem to be much more responsive now with 9.12.2 installed on staging.
Let's definitely go live on the real site with 9.12.2 on Tuesday. Things look good.
Whiteboard: [blocked mindtouch] → 04/13/2010 @ 7pm
Whiteboard: 04/13/2010 @ 7pm → 04/13/2010 @ 4:30pm
Whiteboard: 04/13/2010 @ 4:30pm → 04/15/2010 @ 4:30pm
Can I ask what happened to delay the upgrade until Thursday?
Today will be fine.
Whiteboard: 04/15/2010 @ 4:30pm → 04/13/2010 @ 4:30pm
The upgrade is done, however, I see missing strings.  I'm on AIM if you have time to figure it out.
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
nm, fixed.
Verified FIXED on https://developer.mozilla.org/En; there was a bit of downtime, but all seems to be well now.
URL: https://developer.mozilla.org/En
Status: RESOLVED → VERIFIED
Seems pretty crashy.  The daemons have been dying a couple times an hour since the ugprade.
Sigh, yes; never saw this in staging.
OK, both boxes have had mono upgraded from 2.4 to 2.4.3, and also did the RHEL 5.4 -> 5.5 upgrade on them while I was at it.
Still seeing issues because of this push, having to kick MDC once in 20 mins sometimes to get it to work.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Caught this when I was watching the logs, not sure how relevant or helpful this is.
Component: Server Operations → Server Operations: Projects
Whiteboard: 04/13/2010 @ 4:30pm
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: