Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Update MDC to MindTouch 9.12

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations: Projects
RESOLVED FIXED
8 years ago
2 years ago

People

(Reporter: sheppy, Assigned: oremj)

Tracking

Dependency tree / graph
Bug Flags:
needs-downtime +

Details

(URL)

Attachments

(1 attachment)

(Reporter)

Description

8 years ago
We need to upgrade MDC to MindTouch 9.12.
(Reporter)

Updated

8 years ago
Depends on: 531978
(Assignee)

Updated

8 years ago
Assignee: server-ops → jeremy.orem+bugs
(Assignee)

Comment 1

8 years ago
Looks like 9.12 isn't quite done yet. Let's reopen this when it's closer to release.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → INCOMPLETE
(Reporter)

Comment 2

8 years ago
9.12 has been delayed until after the holidays; they decided to pick up a few more changes to improve things for us a little bit more.

In particular, they have backported from their planned next release a patch to let Lucene index rebuilds resume from where they left off after a failure, instead of having to start from scratch every time. This will be a big win for us in cases where we have to restart during a rebuild for whatever reason.

Internal testing is ongoing on these fixes they've backported and they expect to release when they return from the holiday break.
(Reporter)

Comment 3

8 years ago
MindTouch 9.12 is final now. The package has not been updated, but SVN will get us the release version. We need to install this onto the test system and sic QA on it.
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Updated

8 years ago
Flags: needs-downtime+
Whiteboard: 01/26/2010 @ 7pm
(Reporter)

Updated

8 years ago
Depends on: 541438
(Assignee)

Comment 4

8 years ago
stage has been updated. Are we still on for tonight?
ready to announce this, how long of a window do you need for it?
(Assignee)

Comment 6

8 years ago
30 min.
(Reporter)

Comment 7

8 years ago
Has QA signed off yet? I thought they were cc'd on this but I just realized they weren't, so my guess is they haven't looked at it yet.
(Reporter)

Comment 8

8 years ago
We need to delay this. QA had not in fact tested this; I dropped the ball on asking them to do it (thought I had but had not). I've asked them if they can do it in time for Thursday or Tuesday upcoming. Sorry about this!
If end of day Thursday works for you guys (Eric/IT), then QA can commit to that.
(Assignee)

Updated

8 years ago
Whiteboard: 01/26/2010 @ 7pm → 02/02/2010 @ 7pm
(Assignee)

Comment 10

8 years ago
Are we still going through with this tonight or delaying another 2 days?
Depends on: 543537
(In reply to comment #10)
> Are we still going through with this tonight or delaying another 2 days?

From QA's side, I'm happy with the site, sans still-open bug 543537; Sheppy, it's your call.
(Reporter)

Comment 12

8 years ago
Delay it (assuming you haven't already made that decision). I want to get this last thing ironed out. Thanks all!

Updated

8 years ago
Whiteboard: 02/02/2010 @ 7pm → 02/04/2010 @ 7pm
(Assignee)

Comment 13

8 years ago
Is this happening tonight?
(Reporter)

Comment 14

8 years ago
No, we need to sort out this RSS feed problem first, and so far nobody's sure why it's happening.
Whiteboard: 02/04/2010 @ 7pm → 02/09/2010 @ 7pm
(Assignee)

Comment 15

8 years ago
What can I do to help them figure this out?
(Reporter)

Comment 16

8 years ago
Is there anything informative at all in the PHP log?
(Assignee)

Comment 17

8 years ago
[root@sm-devmostage01 devmo.dekiwiki.mozilla.org]# cat error_log_2010-02* | grep -v '/data/www/dekiwiki/robots.txt'  |grep -v 'link target not accessible'
[Sun Jan 31 21:14:16 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Sun Jan 31 21:18:18 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Sun Jan 31 21:18:44 2010] [error] [client 10.2.81.4] Invalid URI in request GET /servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Sun Jan 31 21:19:21 2010] [error] [client 10.2.81.4] Invalid URI in request GET /samples/servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Sun Jan 31 21:21:43 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/info.php' not found or unable to stat
[Sun Jan 31 21:21:43 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/phptest.php' not found or unable to stat
[Sun Jan 31 21:21:44 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/info.php3
[Sun Jan 31 21:21:45 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/dekiwiki/skins/common/logo/
[Sun Jan 31 21:21:48 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/mountain.cfg
[Sun Jan 31 21:22:20 2010] [error] [client 10.2.81.4] script '/data/www/css/phpinfo.php' not found or unable to stat
[Sun Jan 31 21:22:20 2010] [error] [client 10.2.81.4] script '/data/www/css/php.php' not found or unable to stat
[Sun Jan 31 21:22:21 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/phpinfo.php3
[Sun Jan 31 21:22:22 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/phptest.php3
[Sun Jan 31 21:22:23 2010] [error] [client 10.2.81.4] Invalid method in request get /css/ HTTP/1.1
[Sun Jan 31 21:23:14 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Sun Jan 31 21:24:20 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Sun Jan 31 21:25:19 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Sun Jan 31 21:26:09 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Sun Jan 31 21:26:34 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1
[Sun Jan 31 21:31:01 2010] [error] [client 10.2.81.4] Invalid method in request BADMTHD /css/ HTTP/1.0
[Sun Jan 31 21:31:52 2010] [error] [client 10.2.81.4] script '/data/www/css/userinfo.php' not found or unable to stat
[Tue Feb 02 06:15:53 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/admin-serv
[Tue Feb 02 06:26:22 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Tue Feb 02 06:26:45 2010] [error] [client 10.2.81.4] Invalid URI in request GET /servlet/com.livesoftware.jrun.plugins.ssi.SSIFilter/../../../../../../../etc/passwd HTTP/1.1
[Tue Feb 02 06:27:41 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/mountain.cfg
[Tue Feb 02 06:28:25 2010] [error] [client 10.2.81.4] File does not exist: /data/www/dekiwiki/config/config.txt
[Tue Feb 02 06:29:49 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 33353 HTTP/1.1
[Tue Feb 02 06:31:07 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Tue Feb 02 06:32:12 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Tue Feb 02 06:32:18 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1
[Tue Feb 02 06:32:19 2010] [error] [client 10.2.81.4] script '/data/www/css/info.php' not found or unable to stat
[Tue Feb 02 06:32:19 2010] [error] [client 10.2.81.4] script '/data/www/css/phptest.php' not found or unable to stat
[Tue Feb 02 06:32:20 2010] [error] [client 10.2.81.4] File does not exist: /data/www/css/info.php3
[Tue Feb 02 06:32:21 2010] [error] [client 10.2.81.4] Directory index forbidden by Options directive: /data/www/css/
[Tue Feb 02 06:33:39 2010] [error] [client 10.2.81.4] Invalid method in request BADMTHD /css/ HTTP/1.0
[Tue Feb 02 06:35:05 2010] [error] [client 10.2.81.4] script '/data/www/dekiwiki/config/userinfo.php' not found or unable to stat

cat error_log
[Sun Jan 31 04:02:01 2010] [notice] Digest: generating secret for digest authentication ...
[Sun Jan 31 04:02:01 2010] [notice] Digest: done
[Sun Jan 31 04:02:01 2010] [notice] Apache configured -- resuming normal operations
[Sun Jan 31 18:17:44 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../../../ HTTP/1.1
[Sun Jan 31 18:18:32 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Sun Jan 31 18:33:03 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Sun Jan 31 18:33:09 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../"><script>alert(73541);</script> HTTP/1.1
[Sun Jan 31 18:33:53 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../makebook-2.2/bookfile.html HTTP/1.1
[Sun Jan 31 18:34:04 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Sun Jan 31 18:34:04 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 46614 HTTP/1.1
[Sun Jan 31 18:34:57 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Sun Jan 31 18:35:41 2010] [error] [client 10.2.81.4] Invalid Content-Length
[Tue Feb 02 03:02:07 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../../../ HTTP/1.1
[Tue Feb 02 03:02:11 2010] [error] [client 10.2.81.4] Invalid URI in request   HEAD / HTTP/1.0
[Tue Feb 02 03:12:51 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../makebook-2.2/bookfile.html HTTP/1.1
[Tue Feb 02 03:13:44 2010] [error] [client 10.2.81.4] Invalid URI in request PRONECT 63.245.209.113 51434 HTTP/1.1
[Tue Feb 02 03:13:44 2010] [error] [client 10.2.81.4] request failed: error reading the headers
[Tue Feb 02 03:14:28 2010] [error] [client 10.2.81.4] Client sent malformed Host header
[Tue Feb 02 03:15:37 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../ HTTP/1.1
[Tue Feb 02 03:16:27 2010] [error] [client 10.2.81.4] Invalid URI in request GET /../../../../../../../../../ HTTP/1.1


So no, not really anything informative.
(Reporter)

Comment 18

8 years ago
MindTouch wants to confirm that we have PHP level logging enabled to be sure the errors they're wanting to look for aren't being suppressed; can they get a look at our Apache config?
(Assignee)

Comment 19

8 years ago
LogLevel isn't set, so it's just default. This is in the Vhost:
	php_value error_reporting "E_ALL & ~E_NOTICE"

I'll bump up the error reporting in LocalSettings
(Assignee)

Comment 20

8 years ago
I just added:
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL & ~E_NOTICE);

to the LocalSettings.php.

At first I did error_reporting(E_ALL) and saw a ton of notices, so error reporting is working.
(Reporter)

Comment 21

8 years ago
MindTouch support has asked again to see the Apache config for the site.
(Assignee)

Comment 22

8 years ago
There are only 3 entries in the logs and I think they were from when notices were on:
[Wed Feb 03 08:36:53 2010] [error] [client 10.2.81.4] PHP Notice:  Use of undefined constant NS_MEDIAWIKI_TALK - assumed 'NS_MEDIAWIKI_TALK' in /data/www/dekiwiki/includes/NamespaceAlternates.php on line 1538
[Wed Feb 03 08:36:53 2010] [error] [client 10.2.81.4] PHP Warning:  Cannot modify header information - headers already sent by (output started at /data/www/dekiwiki/includes/NamespaceAlternates.php:327) in /data/www/dekiwiki/includes/OutputPage.php on line 471
[Wed Feb 03 05:17:24 2010] [error] [client 10.2.81.4] PHP Warning:  Cannot modify header information - headers already sent by (output started at /data/www/dekiwiki/includes/NamespaceAlternates.php:327) in /data/www/dekiwiki/includes/OutputPage.php on line 583

Updated

8 years ago
Whiteboard: 02/09/2010 @ 7pm

Updated

8 years ago
Whiteboard: [blocked mindtouch]
Re-open when there's IT work to do.
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → INCOMPLETE
(Reporter)

Comment 24

7 years ago
MindTouch has released 9.12.2 today. Please update the staging server to this for testing. Thank you.
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
(Reporter)

Comment 25

7 years ago
See http://blog.developer.mindtouch.com/2010/04/06/mindtouch-9122-noatak-released/ for notes; there's a database upgrade script that needs to be run as well as updating the code itself.
(Assignee)

Comment 26

7 years ago
Staging server has been updated. Do you want to update production next Tuesday?
(Reporter)

Comment 27

7 years ago
I think so, yeah. This should take care of our concerns.

Could you restart the staging server for me? I just changed a config setting that I don't think takes effect until after a restart.
(Reporter)

Comment 28

7 years ago
BTW, RSS feeds seem to be much more responsive now with 9.12.2 installed on staging.
(Reporter)

Comment 29

7 years ago
Let's definitely go live on the real site with 9.12.2 on Tuesday. Things look good.

Updated

7 years ago
Whiteboard: [blocked mindtouch] → 04/13/2010 @ 7pm
Whiteboard: 04/13/2010 @ 7pm → 04/13/2010 @ 4:30pm

Updated

7 years ago
Whiteboard: 04/13/2010 @ 4:30pm → 04/15/2010 @ 4:30pm
(Reporter)

Comment 30

7 years ago
Can I ask what happened to delay the upgrade until Thursday?
(Assignee)

Comment 31

7 years ago
Today will be fine.
Whiteboard: 04/15/2010 @ 4:30pm → 04/13/2010 @ 4:30pm
(Assignee)

Comment 32

7 years ago
The upgrade is done, however, I see missing strings.  I'm on AIM if you have time to figure it out.
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago7 years ago
Resolution: --- → FIXED
(Assignee)

Comment 33

7 years ago
nm, fixed.
Verified FIXED on https://developer.mozilla.org/En; there was a bit of downtime, but all seems to be well now.
Status: RESOLVED → VERIFIED
Seems pretty crashy.  The daemons have been dying a couple times an hour since the ugprade.
Sigh, yes; never saw this in staging.
http://mindtouch.pastebin.mozilla.org/714818
OK, both boxes have had mono upgraded from 2.4 to 2.4.3, and also did the RHEL 5.4 -> 5.5 upgrade on them while I was at it.
Still seeing issues because of this push, having to kick MDC once in 20 mins sometimes to get it to work.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Created attachment 438973 [details]
stack trace from deki-api.log

Caught this when I was watching the logs, not sure how relevant or helpful this is.
Blocks: 559341

Updated

7 years ago
Duplicate of this bug: 559305

Updated

7 years ago
Component: Server Operations → Server Operations: Projects
Whiteboard: 04/13/2010 @ 4:30pm
(Assignee)

Updated

7 years ago
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.