Last Comment Bug 531977 - Update MDC to MindTouch 9.12
: Update MDC to MindTouch 9.12
Status: RESOLVED FIXED
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations: Projects (show other bugs)
: other
: All All
: -- normal (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Mentors:
https://developer.mozilla.org/En
: 559305 (view as bug list)
Depends on: 531978 541438 543537
Blocks: 559341
  Show dependency treegraph
 
Reported: 2009-11-30 15:18 PST by Eric Shepherd [:sheppy]
Modified: 2015-03-12 08:24 PDT (History)
6 users (show)
mzeier: needs‑downtime+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
stack trace from deki-api.log (10.83 KB, text/plain)
2010-04-14 06:16 PDT, Shyam Mani [:fox2mike]
no flags Details

Description Eric Shepherd [:sheppy] 2009-11-30 15:18:47 PST
We need to upgrade MDC to MindTouch 9.12.
Comment 1 Jeremy Orem [:oremj] 2009-12-02 14:06:13 PST
Looks like 9.12 isn't quite done yet. Let's reopen this when it's closer to release.
Comment 2 Eric Shepherd [:sheppy] 2009-12-17 12:55:47 PST
9.12 has been delayed until after the holidays; they decided to pick up a few more changes to improve things for us a little bit more.

In particular, they have backported from their planned next release a patch to let Lucene index rebuilds resume from where they left off after a failure, instead of having to start from scratch every time. This will be a big win for us in cases where we have to restart during a rebuild for whatever reason.

Internal testing is ongoing on these fixes they've backported and they expect to release when they return from the holiday break.
Comment 3 Eric Shepherd [:sheppy] 2010-01-19 11:42:01 PST
MindTouch 9.12 is final now. The package has not been updated, but SVN will get us the release version. We need to install this onto the test system and sic QA on it.
Comment 4 Jeremy Orem [:oremj] 2010-01-26 11:09:37 PST
stage has been updated. Are we still on for tonight?
Comment 5 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-01-26 13:18:50 PST
ready to announce this, how long of a window do you need for it?
Comment 6 Jeremy Orem [:oremj] 2010-01-26 13:24:25 PST
30 min.
Comment 7 Eric Shepherd [:sheppy] 2010-01-26 13:47:42 PST
Has QA signed off yet? I thought they were cc'd on this but I just realized they weren't, so my guess is they haven't looked at it yet.
Comment 8 Eric Shepherd [:sheppy] 2010-01-26 14:40:20 PST
We need to delay this. QA had not in fact tested this; I dropped the ball on asking them to do it (thought I had but had not). I've asked them if they can do it in time for Thursday or Tuesday upcoming. Sorry about this!
Comment 9 Stephen Donner [:stephend] - PTO; back on 5/28 2010-01-26 15:27:35 PST
If end of day Thursday works for you guys (Eric/IT), then QA can commit to that.
Comment 10 Jeremy Orem [:oremj] 2010-02-02 14:10:34 PST
Are we still going through with this tonight or delaying another 2 days?
Comment 11 Stephen Donner [:stephend] - PTO; back on 5/28 2010-02-02 14:20:53 PST
(In reply to comment #10)
> Are we still going through with this tonight or delaying another 2 days?

From QA's side, I'm happy with the site, sans still-open bug 543537; Sheppy, it's your call.
Comment 12 Eric Shepherd [:sheppy] 2010-02-02 19:07:25 PST
Delay it (assuming you haven't already made that decision). I want to get this last thing ironed out. Thanks all!
Comment 13 Jeremy Orem [:oremj] 2010-02-04 11:42:50 PST
Is this happening tonight?
Comment 14 Eric Shepherd [:sheppy] 2010-02-04 11:43:42 PST
No, we need to sort out this RSS feed problem first, and so far nobody's sure why it's happening.
Comment 15 Jeremy Orem [:oremj] 2010-02-04 12:25:47 PST
What can I do to help them figure this out?
Comment 16 Eric Shepherd [:sheppy] 2010-02-04 12:27:59 PST
Is there anything informative at all in the PHP log?
Comment 17 Jeremy Orem [:oremj] 2010-02-04 13:40:51 PST
[root@sm-devmostage01 devmo.dekiwiki.mozilla.org]# cat error_log_2010-02* | grep -v '/data/www/dekiwiki/robots.txt'  |grep -v 'link target not accessible'
[Sun Jan 31 21:14:16 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Sun Jan 31 21:18:18 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Sun Jan 31 21:18:44 2010] [error] [client 10.2.81.4] Invalid URI in request GET /servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Sun Jan 31 21:19:21 2010] [error] [client 10.2.81.4] Invalid URI in request GET /samples/servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Sun Jan 31 21:21:43 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/info.php' not found or unable to stat
[Sun Jan 31 21:21:43 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/phptest.php' not found or unable to stat
[Sun Jan 31 21:21:44 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/info.php3
[Sun Jan 31 21:21:45 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/dekiwiki/skins/common/logo/
[Sun Jan 31 21:21:48 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/mountain.cfg
[Sun Jan 31 21:22:20 2010] [error] [client 10.2.81.4] script '/data/www/css/phpinfo.php' not found or unable to stat
[Sun Jan 31 21:22:20 2010] [error] [client 10.2.81.4] script '/data/www/css/php.php' not found or unable to stat
[Sun Jan 31 21:22:21 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/phpinfo.php3
[Sun Jan 31 21:22:22 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/phptest.php3
[Sun Jan 31 21:22:23 2010] [error] [client 10.2.81.4] Invalid method in request get /css/ HTTP/1.1
[Sun Jan 31 21:23:14 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Sun Jan 31 21:24:20 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Sun Jan 31 21:25:19 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Sun Jan 31 21:26:09 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Sun Jan 31 21:26:34 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1
[Sun Jan 31 21:31:01 2010] [error] [client 10.2.81.4] Invalid method in request BADMTHD /css/ HTTP/1.0
[Sun Jan 31 21:31:52 2010] [error] [client 10.2.81.4] script '/data/www/css/userinfo.php' not found or unable to stat
[Tue Feb 02 06:15:53 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/admin-serv
[Tue Feb 02 06:26:22 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Tue Feb 02 06:26:45 2010] [error] [client 10.2.81.4] Invalid URI in request GET /servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Tue Feb 02 06:27:41 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/mountain.cfg
[Tue Feb 02 06:28:25 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/config.txt
[Tue Feb 02 06:29:49 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 33353 HTTP/1.1
[Tue Feb 02 06:31:07 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Tue Feb 02 06:32:12 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Tue Feb 02 06:32:18 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1
[Tue Feb 02 06:32:19 2010] [error] [client 10.2.81.4] script '/data/www/css/info.php' not found or unable to stat
[Tue Feb 02 06:32:19 2010] [error] [client 10.2.81.4] script '/data/www/css/phptest.php' not found or unable to stat
[Tue Feb 02 06:32:20 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/info.php3
[Tue Feb 02 06:32:21 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Tue Feb 02 06:33:39 2010] [error] [client 10.2.81.4] Invalid method in request BADMTHD /css/ HTTP/1.0
[Tue Feb 02 06:35:05 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/userinfo.php' not found or unable to stat

cat error_log
[Sun Jan 31 04:02:01 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 31 04:02:01 2010] [notice] Digest: done
[Sun Jan 31 04:02:01 2010] [notice] Apache configured -- resuming normal operations
[Sun Jan 31 18:17:44 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../../../ HTTP/1.1
[Sun Jan 31 18:18:32 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Sun Jan 31 18:33:03 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Sun Jan 31 18:33:09 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../"><script>alert(73541);</script> HTTP/1.1
[Sun Jan 31 18:33:53 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../makebook-2.2/bookfile.html HTTP/1.1
[Sun Jan 31 18:34:04 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Sun Jan 31 18:34:04 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 46614 HTTP/1.1
[Sun Jan 31 18:34:57 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Sun Jan 31 18:35:41 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Tue Feb 02 03:02:07 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../../../ HTTP/1.1
[Tue Feb 02 03:02:11 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Tue Feb 02 03:12:51 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../makebook-2.2/bookfile.html HTTP/1.1
[Tue Feb 02 03:13:44 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 51434 HTTP/1.1
[Tue Feb 02 03:13:44 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Tue Feb 02 03:14:28 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Tue Feb 02 03:15:37 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Tue Feb 02 03:16:27 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1


So no, not really anything informative.
Comment 18 Eric Shepherd [:sheppy] 2010-02-04 15:07:51 PST
MindTouch wants to confirm that we have PHP level logging enabled to be sure the errors they're wanting to look for aren't being suppressed; can they get a look at our Apache config?
Comment 19 Jeremy Orem [:oremj] 2010-02-04 20:18:04 PST
LogLevel isn't set, so it's just default. This is in the Vhost:
	php_value error_reporting "E_ALL & ~E_NOTICE"

I'll bump up the error reporting in LocalSettings
Comment 20 Jeremy Orem [:oremj] 2010-02-04 20:23:18 PST
I just added:
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL & ~E_NOTICE);

to the LocalSettings.php.

At first I did error_reporting(E_ALL) and saw a ton of notices, so error reporting is working.
Comment 21 Eric Shepherd [:sheppy] 2010-02-05 12:54:21 PST
MindTouch support has asked again to see the Apache config for the site.
Comment 22 Jeremy Orem [:oremj] 2010-02-05 13:23:41 PST
There are only 3 entries in the logs and I think they were from when notices were on:
[Wed Feb 03 08:36:53 2010] [error] [client 10.2.81.4] PHP Notice:  Use of undefined constant NS_MEDIAWIKI_TALK - assumed 'NS_MEDIAWIKI_TALK' in /data/www/dekiwiki/includes/NamespaceAlternates.php on line 1538
[Wed Feb 03 08:36:53 2010] [error] [client 10.2.81.4] PHP Warning:  Cannot modify header information - headers already sent by (output started at /data/www/dekiwiki/includes/NamespaceAlternates.php:327) in /data/www/dekiwiki/includes/OutputPage.php on line 471
[Wed Feb 03 05:17:24 2010] [error] [client 10.2.81.4] PHP Warning:  Cannot modify header information - headers already sent by (output started at /data/www/dekiwiki/includes/NamespaceAlternates.php:327) in /data/www/dekiwiki/includes/OutputPage.php on line 583
Comment 23 matthew zeier [:mrz] 2010-02-24 10:23:50 PST
Re-open when there's IT work to do.
Comment 24 Eric Shepherd [:sheppy] 2010-04-06 12:05:54 PDT
MindTouch has released 9.12.2 today. Please update the staging server to this for testing. Thank you.
Comment 25 Eric Shepherd [:sheppy] 2010-04-06 12:42:42 PDT
See http://blog.developer.mindtouch.com/2010/04/06/mindtouch-9122-noatak-released/ for notes; there's a database upgrade script that needs to be run as well as updating the code itself.
Comment 26 Jeremy Orem [:oremj] 2010-04-06 13:06:50 PDT
Staging server has been updated. Do you want to update production next Tuesday?
Comment 27 Eric Shepherd [:sheppy] 2010-04-06 13:41:05 PDT
I think so, yeah. This should take care of our concerns.

Could you restart the staging server for me? I just changed a config setting that I don't think takes effect until after a restart.
Comment 28 Eric Shepherd [:sheppy] 2010-04-06 14:54:43 PDT
BTW, RSS feeds seem to be much more responsive now with 9.12.2 installed on staging.
Comment 29 Eric Shepherd [:sheppy] 2010-04-08 12:02:51 PDT
Let's definitely go live on the real site with 9.12.2 on Tuesday. Things look good.
Comment 30 Eric Shepherd [:sheppy] 2010-04-13 13:12:35 PDT
Can I ask what happened to delay the upgrade until Thursday?
Comment 31 Jeremy Orem [:oremj] 2010-04-13 13:35:00 PDT
Today will be fine.
Comment 32 Jeremy Orem [:oremj] 2010-04-13 16:55:17 PDT
The upgrade is done, however, I see missing strings.  I'm on AIM if you have time to figure it out.
Comment 33 Jeremy Orem [:oremj] 2010-04-13 17:07:30 PDT
nm, fixed.
Comment 34 Stephen Donner [:stephend] - PTO; back on 5/28 2010-04-13 18:49:57 PDT
Verified FIXED on https://developer.mozilla.org/En; there was a bit of downtime, but all seems to be well now.
Comment 35 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-04-13 20:43:59 PDT
Seems pretty crashy.  The daemons have been dying a couple times an hour since the ugprade.
Comment 36 Stephen Donner [:stephend] - PTO; back on 5/28 2010-04-13 20:49:34 PDT
Sigh, yes; never saw this in staging.
Comment 37 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-04-13 20:56:06 PDT
http://mindtouch.pastebin.mozilla.org/714818
Comment 38 Dave Miller [:justdave] (justdave@bugzilla.org) 2010-04-13 23:20:42 PDT
OK, both boxes have had mono upgraded from 2.4 to 2.4.3, and also did the RHEL 5.4 -> 5.5 upgrade on them while I was at it.
Comment 39 Shyam Mani [:fox2mike] 2010-04-14 05:26:01 PDT
Still seeing issues because of this push, having to kick MDC once in 20 mins sometimes to get it to work.
Comment 40 Shyam Mani [:fox2mike] 2010-04-14 06:16:51 PDT
Created attachment 438973 [details]
stack trace from deki-api.log

Caught this when I was watching the logs, not sure how relevant or helpful this is.
Comment 41 Shyam Mani [:fox2mike] 2010-04-14 09:05:44 PDT
*** Bug 559305 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.