782627 - Only ~100 tegras are running tests; tegra pending test job count is at 700 and rising

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ed Morley [:emorley]]

+++ This bug was initially created as a clone of Bug #777273 +++

http://mobile-dashboard.pub.build.mozilla.org says roughly half the tegras are not coming out to play.

Comment 1

[Hal Wine [:hwine] use NI!]

This _might_ be a reporting anomaly. We're in process of rolling out a batch of additional tegras (bug 767456), and they may be in the "systems to check" database, but not fully enabled yet. I'll ask Callek to confirm.

Comment 2

[Ed Morley [:emorley]]

Unfortunately I don't believe this is just a reporting anomaly, the tegra pending count is really high (and now higher than when the bug was filed):
http://build.mozilla.org/builds/pending/pending.html

There are also many tegra jobs that have been waiting for up to 15 hours, eg:
https://tbpl.mozilla.org/?tree=Try&jobname=Android&noignore=1&rev=47971dbd8c3f
https://tbpl.mozilla.org/?tree=Try&jobname=Android&noignore=1&rev=731276a76ec5
https://tbpl.mozilla.org/?tree=Try&jobname=Android&noignore=1&rev=5c590a30e04a

Comment 3

[Justin Wood (:Callek)]

rail learned that bm22 was stuck/hung... my theory is that it was hung all weekend.

That cuts out aprox 1/3 of our active tegra pool. Which is why the backup is so visible.

100 active tegras per dashboard is *slightly* low, but not really unusual, and should come up when we triage/get on file the bad ones for IT to handle... So not too much left to do here from our end, since the issue fell on the "broken master" rather than "broken tegras infra"

And relatedly due to Bug 782665 we have dropped our pending count remarkably down by now.

Comment 4

[Ed Morley [:emorley]]

Ah great - cheers! :-)

Comment 5

[Ed Morley [:emorley]]

82 tegras online according to http://mobile-dashboard.pub.build.mozilla.org/ and...

Pending test(s) @ Aug 15 05:25:03
tegra (65)
   30 mozilla-inbound
   30 fx-team
    5 ionmonkey


 Pending test(s) @ Aug 15 05:25:03
tegra (744)
  744 try

Comment 6

[Hal Wine [:hwine] use NI!]

128 tegras online now per dashboard, test count should drop or issue is not tegra related