338650 - all Tp tests on office tinderboxen are broken ("Tp:nanms")

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Robert Kaiser]

Sorry for the bad choice of components, but I have no clue where this really belongs...

Since 05/20 04:35 or so (builds starting 4:30 are ok, 4:40 are failing), all Mac tinderboxen doing Tp tests are reporting "Tp:nanms", i.e. the tests are failing.

This affects pawn and maya (Camino) as well as planetoid (SeaMonkey). The only checkin visible on bonsai near to this time was my move of one line in mozilla/Makefile.in within a |ifdef MOZ_SUITE| block, which should not affect Camino in any way, though.

Comment 1

[Mark Mentovai]

We're seeing "nanms" on multiple platforms and all branches.  It's probably affecting all boxes using spider.office as their Tp server.  This last happened about a month ago (bug 334139), probably because spider.office ran out of disk space where it stores the results of tests.

(I think that stuff belongs in a cookie anyway.  Silly Tp...)

Moving to a possibly more useful component.

Comment 2

[Justin Fitzhugh]

Looks like something build/release will need to look at - the previous component was correct/ok.  Moving back to build/release.

Comment 3

[Robert Kaiser]

Interstingly, btek and luna still report Tp results correctly...

Comment 4

[Mark Mentovai]

btek and luna are in the colo, so they're probably using axolotl as their Tp server.

Comment 5

[Robert Kaiser]

Mark: oh, didn't know that...

Comment 6

[Robert Helmer [:rhelmer]]

I don't have access to axolotl (filed bug 338666 to get it) but I do have access to spider and I can confirm that it's full.

I am doing a "du | sort -rn" now to see what the biggest dirs are.

It doesn't look like there is a regular job to clear out old logs.. there is only 4GB allocated to /build and half of it is taken up by apache logs.

Comment 7

[Robert Helmer [:rhelmer]]

I am gzipping the 2GB worth of Apache access logs on spider right now, which seems to be helping. I added "compress" to /etc/logrotate.d/apache so the log rotator will do this in the future, let me know if this is a problem for anyone.