375221 - Move Firefox unit test tinderboxes to something IT-managed

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

We've gotten into an unfortunate situation where we're supposed to take failure on unit test tinderboxes seriously, except when we aren't supposed to: a real failure should close the tree until it's fixed or backed out, but because they aren't on IT's list of Tier 1 24/7 things, a burning hiccup evenings or weekends is supposed to be ignored until robcee notices. Or maybe it isn't. And thus the midsection of bug 375156.

Comment 1

[Rob Campbell [:rc] (:robcee)]

you should take failures on the unit test tinderboxes seriously. You should also cross-reference with the other unit test tinderboxes to see if the failure is on one or more platforms. As it stands, we know that the winxp box is slightly more fragile than the other two at the moment and should take that into consideration before shutting the tree down.

I am taking measures to get these machines under IT control.

Comment 2

[Adam Guthrie]

Any update on this, robcee?

Comment 3

[Rob Campbell [:rc] (:robcee)]

Nothing formalized although there is some support from IT in the form of Reed and Aravind having access to the machines and the ability to start/restart the slaves and master.

I'm not sure these are considered Tier1 supported machines, but they should be.

Comment 4

[Justin Fitzhugh]

In order for this to happen we'll need access and procedure documented and given to IT.  Also, we'll need to put monitors in place.  Finally, nothing in the MV office will be considered Tier 1 as we don't have the infrastructure to support that office 24/7 (i.e. remote hands, etc).  I am happy to sit down and go through what we need in order to cover these, but aravind and reed really aren't on the hook for these until we get the rest of this setup.

Comment 5

[Rob Campbell [:rc] (:robcee)]

hey Justin.

I have a rough user's document at: http://wiki.mozilla.org/BuildbotTestFarmUse

Take a look at that and let me know if that's sufficiently-detailed. If not, I'll flesh it out. If you use a particular format for these types of things (I expect a checklist format would be more useful for IT support), I can use that as a model.

Comment 6

[Mike Schroepfer]

Can we move this to Tier 1 support asap?

Comment 7

[Phil Ringnalda (:philor)]

Because I always forget (thanks to the summary), bug 384966 was the other unsupportable -> unsupportable-but-supported bug.

Comment 8

[Rob Campbell [:rc] (:robcee)]

I've updated and moved a proposed support document to:

http://wiki.mozilla.org/Buildbot/IT_Unittest_Support_Document

I'll be making a similar one for talos support by end of day.

Comment 9

[Rob Campbell [:rc] (:robcee)]

here's the other:

http://wiki.mozilla.org/Buildbot/IT_Unittest_Support_Document

waiting on justin for handoff of keys.

Comment 10

[Rob Campbell [:rc] (:robcee)]

gavin noticed I pasted in the same link (thanks!).

Here's the proper one:

http://wiki.mozilla.org/Buildbot/IT_Talos_Support_Document