513773 - Out of disk space on Mozilla Service Week cluster/node?

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

STR:

Load https://mozillaservice.stage.mozilla.com/auth/change_locale/fr_FR?redirect_base=%2Fhome%2Findex&lang=en-us

Results:

Warning Message

An error was detected which prevented the loading of this page. If this problem persists, please contact the website administrator.

system/core/Kohana.php [528]:

file_put_contents() [function.file-put-contents]: Only 0 of 1251 bytes written, possibly out of free disk space

Comment 1

[matthew zeier [:mrz]]

Indeed.  That host isn't even being monitored.  Had added monitoring and am clearing disk space.

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; thanks, mrz.

Comment 3

[Austin King [:ozten]]

We are out of disk again.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #3)
> We are out of disk again.

Still?  Should be back up as of a few minutes ago.  WFM.

Comment 5

[Austin King [:ozten]]

It appears to be working... but at 12:20 I got the same error messages as yesterday when I created content on the your_stories page.

Leaving open, so we can figure out why the issue came back.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; back again!

Comment 7

[Austin King [:ozten]]

This is back again and blocking the L10n community!

Can we leave this bug open until we get a root cause analysis.

Comment 8

[matthew zeier [:mrz]]

I incorrectly added the disk space check. Fixed that.

Comment 9

[Shyam Mani [:fox2mike]]

Stage bugs are not blockers.

Comment 10

[Laura Thomson :laura]

Affecting SUMO stage as well (or at least it's broken and I suspect the same root cause).

Comment 11

[matthew zeier [:mrz]]

sumo's on the same box as mozillaservice and the disk space issue has been resolved.

[root@mrapp-stage02 domains]# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       64G   40G   22G  65% /

What errors are you seeing?

Comment 12

[Laura Thomson :laura]

If you try and load any pages you get redirected to http://kb  - which, while it looks like a broken redirect, last time we saw this behavior it was a disk space issue.

Comment 13

[Derek Moore [:dmoore]]

The SUMO stage redirect is being thrown from within the tikiwiki application itself. It looks like it might be a configuration problem or an issue with locale detection.

Comment 14

[Laura Thomson :laura]

The code hasn't changed for three days but it suddenly broke today, so it's unlikely to be a code problem.  Let me start a new bug for support-stage specifically.

Comment 15

[Derek Moore [:dmoore]]

Per comment #14, tracking this issue in a new bug.

Comment 16

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED -- staging seems fine here and elsewhere, now.