All users were logged out of Bugzilla on October 13th, 2018

Only ~100 tegras are running tests; tegra pending test job count is at 700 and rising

RESOLVED FIXED

Status

--
blocker
RESOLVED FIXED
6 years ago
5 months ago

People

(Reporter: emorley, Assigned: rail)

Tracking

Details

(Whiteboard: [buildduty][capacity])

(Reporter)

Description

6 years ago
+++ This bug was initially created as a clone of Bug #777273 +++

http://mobile-dashboard.pub.build.mozilla.org says roughly half the tegras are not coming out to play.
Whiteboard: [buildduty][capacity]
This _might_ be a reporting anomaly. We're in process of rolling out a batch of additional tegras (bug 767456), and they may be in the "systems to check" database, but not fully enabled yet. I'll ask Callek to confirm.
(Reporter)

Comment 2

6 years ago
Unfortunately I don't believe this is just a reporting anomaly, the tegra pending count is really high (and now higher than when the bug was filed):
http://build.mozilla.org/builds/pending/pending.html

There are also many tegra jobs that have been waiting for up to 15 hours, eg:
https://tbpl.mozilla.org/?tree=Try&jobname=Android&noignore=1&rev=47971dbd8c3f
https://tbpl.mozilla.org/?tree=Try&jobname=Android&noignore=1&rev=731276a76ec5
https://tbpl.mozilla.org/?tree=Try&jobname=Android&noignore=1&rev=5c590a30e04a
rail learned that bm22 was stuck/hung... my theory is that it was hung all weekend.

That cuts out aprox 1/3 of our active tegra pool. Which is why the backup is so visible.

100 active tegras per dashboard is *slightly* low, but not really unusual, and should come up when we triage/get on file the bad ones for IT to handle... So not too much left to do here from our end, since the issue fell on the "broken master" rather than "broken tegras infra"

And relatedly due to Bug 782665 we have dropped our pending count remarkably down by now.
Assignee: nobody → rail
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 4

6 years ago
Ah great - cheers! :-)
(Reporter)

Comment 5

6 years ago
82 tegras online according to http://mobile-dashboard.pub.build.mozilla.org/ and...

Pending test(s) @ Aug 15 05:25:03
tegra (65)
   30 mozilla-inbound
   30 fx-team
    5 ionmonkey


 Pending test(s) @ Aug 15 05:25:03
tegra (744)
  744 try
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
128 tegras online now per dashboard, test count should drop or issue is not tegra related
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering

Updated

5 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.