850531 - Figure out what is different with talos-r3-w7-001, talos-r3-w7-002, talos-r3-w7-003 and talos-r3-w7-010

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=20579940&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=20560440&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=20583308&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=20585759&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=20585740&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=20585845&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=20585956&tree=Mozilla-Beta

I was retriggering like mad, ready to close the tree and file a blocker and all that noise, when I finally realized that nobody had landed a regression on mozilla-beta, it was just the newly moved-from-staging slaves being unable to actually pass tests.

Resolution? Lack of a dongle? Dunno, but please get them out of production, I'll take more of a backlog over incomprehensible outbreaks of unknown failures.

Comment 1

[Justin Wood (:Callek)]

I just disabled these slaves and marked slavealloc with this bug#, leaving next-step for armen.

Comment 2

[Armen [:armenzg]]

They needed Windows activation. Requested in bug 848944.

Comment 3

[Armen [:armenzg]]

(In reply to Armen Zambrano G. [:armenzg] from comment #2)
> They needed Windows activation. Requested in bug 848944.

This got done.

I'm re-triggering all orange jobs on staging and see which ones clear up.

Comment 4

[Armen [:armenzg]]

This is ridiculous!

After activating the 4 machines, opt reftest fails and the only difference (see attachment) is a slight difference when drawing the letter "i".
http://cl.ly/NbeQ

For not accelerated reftests I can notice a slight shadow difference:
http://cl.ly/NbLA

Comment 5

[Armen [:armenzg]]

Attachment: reftest difference

Comment 6

[Armen [:armenzg]]

Any of you have any ideas on what I could look into?

The dxdiag doesn't show anything relevant on why would this four machines would be different than production machines.

Comment 7

[cmtalbert]

Drivers are the only things I could think of, or different actual hardware inside the boxes themselves, both of which seem unlikely. If it happens every single time the reftest is run, then perhaps having a gfx dev look into it would be the next step.

Comment 8

[Armen [:armenzg]]

All jobs came out green this time :? (except xperf)
Maybe I chose a bad changeset (my masters got obliterated so I can't go and find which revision I used last time).

I will re-trigger them again and see in the morning.

For the record, this time I'm using this:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1364229324
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1364229324

Comment 9

[Armen [:armenzg]]

These machines have at times shown intermittent behavior but I don't want to have to do all sorts of experiments to discover if it was just a coincidence or there was actually something wrong.

I would like to re-image them as xp slaves and see if they give any trouble or not.

If as XP machines they give us trouble we will decommission them and not spend more time on it.

We should not worry about the word decommissioning since by the end of the month we will have re-purposed the Fedora rev3 machines as XP and Win7 machines.

By the end of the quarter we will hopefully have moved our load to iX machines (at least for win7).

Comment 10

[Armen [:armenzg]]

These machines were turned into xp machines.