768176 - Tegras that are not coming back to production_config.py should not be in the list

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Armen [:armenzg]]

I would like to remove these tegras from production_config.py if they are never coming back to production.

tegra-065   55 days, 3:04:56    bug 732886 - remove from production
tegra-222   26 days, 13:04:11   bug 741684 - remove from production 
tegra-224   65 days, 3:40:26    bug 747641 -- Todo move to staging 
tegra-031   168 days, 4:42:00   bug 718697 - moved to staging 
tegra-033   192 days, 16:35:43  bug 707584 - swap with warroom tegras
tegra-034   197 days, 15:12:58  bug 707584 - swap with warroom tegras
tegra-043   224 days, 1:00:18   bug 707584 - swap with warroom tegras
tegra-044   222 days, 14:36:32  bug 707584 - swap with warroom tegras 
tegra-069   197 days, 15:15:06  bug 707584 - swap with warroom tegras  
tegra-153   192 days, 16:59:36  bug 707584 - swap with warroom tegras
tegra-156   193 days, 6:37:51   bug 707584 - swap with warroom tegras 
tegra-175   208 days, 17:00:54  bug 707584 - swap with warroom tegras 
tegra-176   192 days, 22:08:35  bug 707584 - swap with warroom tegras 
tegra-230   180 days, 11:35:48  bug 715762 - swapped for tegra-287 
tegra-198   55 days, 4:19:47    secteam fuzzing loanerbug 749637 
tegra-202   61 days, 2:57:07    secteam fuzzing loaner bug 749637
tegra-203   55 days, 2:07:43    secteam fuzzing loaner bug 749637 
tegra-205   55 days, 4:54:37    secteam fuzzing loaner bug 749637  
tegra-275   58 days, 4:31:04    secteam fuzzing loaner bug 749637   
tegra-204   61 days, 20:31:05   bug 749637 - sec team fuzzing 

Please speak if any of them should stay.

Comment 1

[Armen [:armenzg]]

Attachment: remove tegras from production_config.py

Comment 2

[Armen [:armenzg]]

Comment on attachment 636769 [details] [diff] [review]
remove tegras from production_config.py

bear, I assume I can request from IT to remove these tegras from nagios and remove them from slavealloc.

I will wait on your confirmation.

Landed on "default":
http://hg.mozilla.org/build/buildbot-configs/rev/6e196f39631a

Comment 3

[Mike Taylor [:bear]]

please do file a relops bug to remove these from nagios

Comment 4

[Armen [:armenzg]]

tegra-110 to be moved to staging - bug 734321

bear, do you know what is the state of tegra-287?
I have these data:
* tegra-230   180 days, 11:35:48  bug 715762 - swapped for tegra-287 
* "_comment": "Bug 749637: Assigned to Sec-Team" [1]

Is tegra-261 waiting for a foopy as well?

[1] https://hg.mozilla.org/build/tools/file/default/buildfarm/mobile/tegras.json#l1274

Comment 5

[Mike Taylor [:bear]]

yes.

this is determined by looking for tegras with "foopy": "None" showing in tegras.json

Comment 6

[Armen [:armenzg]]

Attachment: Comment adjustments

If we load up tegras.json:
http://mxr.mozilla.org/build/source/tools/buildfarm/mobile/tegras.json
We can see some tegras with no comment, no foopy assigned yet a PDU assigned.
Those tegras are waiting for more foopies to show up so we can attach them there.

Loaned:
* tegra-202 (secteam fuzzing loaner bug 749637)
* tegra-205 (secteam fuzzing loaner bug 749637)
* tegra-222 (bug 741684 - ctalbert has it)

No foopy assigned:
* tegra-219
* tegra-226
* tegra-261 
* tegra-286 -- None (also not on slavealloc - 10.250.51.126)

Need to be added to slavealloc:
* tegra-178 -- foopy17 -- not on slavealloc (10.250.50.88)
* tegra-288 -- foopy05 -- not on slavealloc (10.250.51.128)

Comment 7

[Armen [:armenzg]]

Comment on attachment 637689 [details] [diff] [review]
Comment adjustments

http://hg.mozilla.org/build/tools/rev/e3a541bb6f62

Comment 8

[Armen [:armenzg]]

Attachment: removing some more

removing:
tegra-110 - It is staging - bug 734321#c4
tegra-230 - it does not exist on tegras.json
tegra-287 - assigned to sec team

Comment 9

[Armen [:armenzg]]

I really need help with this.

Which of these tegras should be in production taking jobs? Those with no foopy should stay and have a note.
Whatever is not I would like to assume that:
* they're staging
* they're loaned
* they're decommissioned

tegra-032   12 days, 23:55:10
tegra-037   15 days, 23:22:37 
tegra-041   15 days, 21:26:02 
tegra-045   11 days, 5:17:46 
tegra-046   10 days, 23:22:41
tegra-048   14 days, 3:39:17 
tegra-050   14 days, 14:10:04 
tegra-057   14 days, 17:23:06 
tegra-059   15 days, 8:55:24     bug 740456 
tegra-073   13 days, 18:19:01
tegra-082   7 days, 19:15:23     bug 730726 - recovery 
tegra-084   14 days, 21:09:57    bug 734458 
tegra-103   10 days, 13:09:00
tegra-110   112 days, 9:49:54    bug 768176 - bug 734321 - remove from production  
tegra-136   8 days, 3:34:31 
tegra-169   23 days, 9:43:50  
tegra-219   65 days, 15:31:10    bug 758043 - waiting on foopy 
tegra-230   184 days, 7:35:50    bug 715762 - swapped for tegra-287 
tegra-261   58 days, 22:50:35 
tegra-280   20 days, 22:48:41
tegra-287   59 days, 0:58:02     bug 768176 - remove; assigned to Sec Team

Comment 10

[Mike Taylor [:bear]]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #9)
> I really need help with this.
> 
> Which of these tegras should be in production taking jobs? Those with no
> foopy should stay and have a note.
> Whatever is not I would like to assume that:
> * they're staging
> * they're loaned
> * they're decommissioned
> 
> tegra-032   12 days, 23:55:10
> tegra-037   15 days, 23:22:37 
> tegra-041   15 days, 21:26:02 
> tegra-045   11 days, 5:17:46 
> tegra-046   10 days, 23:22:41
> tegra-048   14 days, 3:39:17 
> tegra-050   14 days, 14:10:04 
> tegra-057   14 days, 17:23:06 
> tegra-073   13 days, 18:19:01
> tegra-103   10 days, 13:09:00
> tegra-280   20 days, 22:48:41
> tegra-261   58 days, 22:50:35 

the above should be swapped out with the new arrow order.  We can then put them in staging to see if they can recover and/or assign to devs

> tegra-136   8 days, 3:34:31 
> tegra-169   23 days, 9:43:50  

the above are active now

> tegra-059   15 days, 8:55:24     bug 740456 
> tegra-082   7 days, 19:15:23     bug 730726 - recovery 
> tegra-084   14 days, 21:09:57    bug 734458 
> tegra-110   112 days, 9:49:54    bug 768176 - bug 734321 - remove from
> production  
> tegra-219   65 days, 15:31:10    bug 758043 - waiting on foopy 
> tegra-230   184 days, 7:35:50    bug 715762 - swapped for tegra-287 
> tegra-287   59 days, 0:58:02     bug 768176 - remove; assigned to Sec Team

Comment 11

[Mike Taylor [:bear]]

swiping from armen

Comment 12

[Justin Wood (:Callek)]

(In reply to Mike Taylor [:bear] from comment #10)
> (In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #9)
> > I really need help with this.
> > 
> > Which of these tegras should be in production taking jobs? Those with no
> > foopy should stay and have a note.
> > Whatever is not I would like to assume that:
> > * they're staging
> > * they're loaned
> > * they're decommissioned
> > 
> > tegra-032   12 days, 23:55:10
> > tegra-037   15 days, 23:22:37 
...
> 
> the above should be swapped out with the new arrow order.  We can then put
> them in staging to see if they can recover and/or assign to devs
> 

Before we decide to fully pull them, we should identify the issue in the dashboard, == for example -032 and -037 have |error.flg [Remote Device Error: Unable to properly remove /mnt/sdcard/tests] |

Which is a matter of doing the manual reformat, and if it persists a replace of the SDCard.

Comment 13

[Mike Taylor [:bear]]

how close are we to having the linux foopy up?

we could populate it with these to see how they do after formatting.

remind me again who has to do the formatting - can we do it or does relops have to?

Comment 14

[Chris Cooper [:coop] (he/him)]

Sorry, I'm late to this party.

I've added a new environment to slavealloc named 'decomm' and I've started to slot known-decommissioned slaves into that bucket. I've also started doing bugzilla archaeology to try to find out why they were decommissioned -- in some cases they were loaned -- and trying to add that info to the notes field in slavealloc. For completeness' sake (and so I stop trying to find them), I've added the missing tegras (e.g. tegra-030) back into slavealloc as decommissioned.

Callek suggested tonight that perhaps tegras.json could be replaced by slavealloc since they contain functionally the same info. If we're willing to accept/add foopys as locked masters, I could see this working. I'm all for eliminating another source of truth.

I will go through comment #10 and add that info the slavealloc as well.

Comment 15

[Chris Cooper [:coop] (he/him)]

Waiting on the list of tegras that can't be resurrected in the tegra-recovery bug (bug 778812).

Comment 16

[Chris Cooper [:coop] (he/him)]

The list is accurate as of now, if somewhat ugly.