495116 - developer.mozilla.org serving blank pages and connection resets

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

devmo is borked today, someone needs to look into this.

Comment 1

[Benjamin Smedberg]

I'm getting consistent connection resets when trying to load the "Move Page" dialog, for instance.

Comment 2

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

https://developer.mozilla.org/en/CSS/word-wrap is one URL that was doing it, if that helps for checking out the logs.

Comment 3

[Dave Miller [:justdave]]

Don't know if someone else did anything, but it all seems to be working fine for me?

Comment 4

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

It's been intermittent today -- what's in the logs?

Comment 5

[Dave Miller [:justdave]]

chizu's been poking at this...

Comment 6

[Dave Miller [:justdave]]

dmoore mentioned something about a similar problem last week and local file cache in deki needing to be cleaned...

Comment 7

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

What's the bug number for that previous problem?

Comment 8

[Dave Miller [:justdave]]

ok, it was bug 494409 and we were remembering wrong, the previous problem was AMO not MDC, but the symptoms were the same.

Comment 9

[Eric Shepherd [:sheppy]]

OK, someone please figure this out, it's killing me over here. :)

Comment 10

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

(In reply to comment #4)
> It's been intermittent today -- what's in the logs?

Anything?

Comment 11

[Jeremy Orem [:oremj]]

It appears apache has been segfaulting randomly on those boxes. I've pulled an apache update from RHN, which will hopefully solve the problem, and turned on core dumps for that cluster in case it doesn't.

Comment 12

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

That sounds like something that should page someone, maybe?

Comment 13

[Jeremy Orem [:oremj]]

Attachment: Backtrace and full backtrace of core dump

Typically our segfaults end up being with eaccelerator, so we can just blacklist the offending function call, but it appears that isn't the case here. This might might be a situation where we need to file a redhat bug.

Comment 14

[chizu]

The segfault issue seems resolved with the apache update. It will page on core dumps if this occurs again.

Comment 15

[Stephen Donner [:stephend] Not actively reading bugmail]

I've been using https://developer.mozilla.org/En and friends for a while since this push, without seeing the issue again.

Verified FIXED.