Closed Bug 952249 Opened 6 years ago Closed 6 years ago

Cleanup Releng data stores after tegra decomm

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86_64
Windows 7
task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Callek, Assigned: Callek)

References

Details

Attachments

(1 file)

Attached patch decomSplinter Review
We have a few places that should be cleaned up now that we decomm'd a handful of tegras.

devices.json is one of those places, lets start with that.
Attachment #8350255 - Flags: review?(jhopkins)
Comment on attachment 8350255 [details] [diff] [review]
decom

Cross-checked against bugzilla, all devices are indeed decomm.
Attachment #8350255 - Flags: review?(jhopkins) → review+
raw SQL for slavealloc:

UPDATE slaves SET envid=5, enabled=0, notes="decommed in Bug 918168" where name LIKE '%tegra-056%' OR name LIKE '%tegra-064%' OR name LIKE '%tegra-076%' OR name LIKE '%tegra-090%' OR name LIKE '%tegra-112%' OR name LIKE '%tegra-116%' OR name LIKE '%tegra-174%' OR name LIKE '%tegra-221%' OR name LIKE '%tegra-290%' OR name LIKE '%tegra-295%' OR name LIKE '%tegra-296%' OR name LIKE '%tegra-298%' OR name LIKE '%tegra-300%' OR name LIKE '%tegra-302%' OR name LIKE '%tegra-305%' OR name LIKE '%tegra-306%' OR name LIKE '%tegra-308%' OR name LIKE '%tegra-310%' OR name LIKE '%tegra-314%' OR name LIKE '%tegra-319%' OR name LIKE '%tegra-322%' OR name LIKE '%tegra-323%' OR name LIKE '%tegra-324%' OR name LIKE '%tegra-325%' OR name LIKE '%tegra-326%' OR name LIKE '%tegra-327%' OR name LIKE '%tegra-328%' OR name LIKE '%tegra-330%' OR name LIKE '%tegra-332%' OR name LIKE '%tegra-333%' OR name LIKE '%tegra-336%' OR name LIKE '%tegra-337%' OR name LIKE '%tegra-340%' OR name LIKE '%tegra-341%' OR name LIKE '%tegra-342%' OR name LIKE '%tegra-346%' OR name LIKE '%tegra-347%' OR name LIKE '%tegra-350%' OR name LIKE '%tegra-358%' OR name LIKE '%tegra-359%' OR name LIKE '%tegra-360%' OR name LIKE '%tegra-361%' OR name LIKE '%tegra-365%';

Query OK, 43 rows affected (0.16 sec)
Rows matched: 43  Changed: 43  Warnings: 0

(after r+ from jhopkins on IRC)
with another r+ from jhopkins I hotpatched manage_foopies.py (and foopy_fabric.py) to run as root and have an action of:

+@per_device
+def remove_device_dir(device):
+    if not device:
+       raise Exception
+    with show('running'):
+       with cd('/builds'):
+          run('rm -rf ./%s' % device);
+    print OK, "Removed device (%s) dir on foopy" % device;

Then I ran

(fabric)[jwood@cruncher.srv.releng.scl3 fabric]$ python manage_foopies.py -f devices.json
 -j15 `for i in 056 064 076 090 112 116 174 221 290 295 296 298 300 302 305 306 308 310 3
14 319 322 323 324 325 326 327 328 330 332 333 336 337 340 341 342 346 347 350 358 359 36
0 361 365; do echo -n "-D tegra-$i "; done` remove_device_dir
[foopy111.build.mozilla.org] run: rm -rf ./tegra-064
[foopy112.build.mozilla.org] run: rm -rf ./tegra-090
[foopy113.build.mozilla.org] run: rm -rf ./tegra-056
[foopy114.build.mozilla.org] run: rm -rf ./tegra-076
[foopy117.build.mozilla.org] run: rm -rf ./tegra-112
[foopy121.build.mozilla.org] run: rm -rf ./tegra-174
[foopy123.build.mozilla.org] run: rm -rf ./tegra-221
[foopy125.build.mozilla.org] run: rm -rf ./tegra-290
[foopy29.build.mozilla.org] run: rm -rf ./tegra-342
[foopy30.build.mozilla.org] run: rm -rf ./tegra-359
[foopy31.build.mozilla.org] run: rm -rf ./tegra-341
[foopy32.build.mozilla.org] run: rm -rf ./tegra-360
[foopy28.build.mozilla.org] run: rm -rf ./tegra-314
[OK]   Removed device (tegra-342) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-340
[OK]   Removed device (tegra-174) dir on foopy
[OK]   Removed device (tegra-112) dir on foopy
[foopy117.build.mozilla.org] run: rm -rf ./tegra-116
[OK]   Removed device (tegra-314) dir on foopy
[foopy28.build.mozilla.org] run: rm -rf ./tegra-319
[OK]   Removed device (tegra-221) dir on foopy
[OK]   Removed device (tegra-056) dir on foopy
[OK]   Removed device (tegra-076) dir on foopy
[OK]   Removed device (tegra-360) dir on foopy
[foopy32.build.mozilla.org] run: rm -rf ./tegra-350
[OK]   Removed device (tegra-064) dir on foopy
[OK]   Removed device (tegra-341) dir on foopy
[foopy31.build.mozilla.org] run: rm -rf ./tegra-333
[OK]   Removed device (tegra-090) dir on foopy
[OK]   Removed device (tegra-290) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-298
[OK]   Removed device (tegra-359) dir on foopy
[foopy30.build.mozilla.org] run: rm -rf ./tegra-361
[OK]   Removed device (tegra-116) dir on foopy
[OK]   Removed device (tegra-333) dir on foopy
[foopy31.build.mozilla.org] run: rm -rf ./tegra-327
[OK]   Removed device (tegra-340) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-346
[OK]   Removed device (tegra-350) dir on foopy
[foopy32.build.mozilla.org] run: rm -rf ./tegra-358
[OK]   Removed device (tegra-319) dir on foopy
[foopy28.build.mozilla.org] run: rm -rf ./tegra-310
[OK]   Removed device (tegra-298) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-295
[OK]   Removed device (tegra-361) dir on foopy
[foopy30.build.mozilla.org] run: rm -rf ./tegra-347
[OK]   Removed device (tegra-327) dir on foopy
[foopy31.build.mozilla.org] run: rm -rf ./tegra-337
[OK]   Removed device (tegra-346) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-325
[OK]   Removed device (tegra-295) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-296
[OK]   Removed device (tegra-337) dir on foopy
[foopy31.build.mozilla.org] run: rm -rf ./tegra-336
[OK]   Removed device (tegra-347) dir on foopy
[foopy30.build.mozilla.org] run: rm -rf ./tegra-308
[OK]   Removed device (tegra-325) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-324
[OK]   Removed device (tegra-358) dir on foopy
[OK]   Removed device (tegra-336) dir on foopy
[foopy31.build.mozilla.org] run: rm -rf ./tegra-322
[OK]   Removed device (tegra-296) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-306
[OK]   Removed device (tegra-308) dir on foopy
[foopy30.build.mozilla.org] run: rm -rf ./tegra-365
[OK]   Removed device (tegra-322) dir on foopy
[foopy31.build.mozilla.org] run: rm -rf ./tegra-328
[OK]   Removed device (tegra-324) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-332
[OK]   Removed device (tegra-365) dir on foopy
[OK]   Removed device (tegra-306) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-305
[OK]   Removed device (tegra-328) dir on foopy
[OK]   Removed device (tegra-332) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-326
[OK]   Removed device (tegra-305) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-302
[OK]   Removed device (tegra-302) dir on foopy
[foopy125.build.mozilla.org] run: rm -rf ./tegra-300
[OK]   Removed device (tegra-326) dir on foopy
[foopy29.build.mozilla.org] run: rm -rf ./tegra-330
[OK]   Removed device (tegra-300) dir on foopy
[OK]   Removed device (tegra-330) dir on foopy
[OK]   Removed device (tegra-310) dir on foopy
[foopy28.build.mozilla.org] run: rm -rf ./tegra-323
[OK]   Removed device (tegra-323) dir on foopy
Comment on attachment 8350255 [details] [diff] [review]
decom

https://hg.mozilla.org/build/tools/rev/92acb03e1020

Temporarily leaving bug open to think about if there are any leftovers for this work.
Attachment #8350255 - Flags: checked-in+
(In reply to Justin Wood (:Callek) from comment #4)
> Temporarily leaving bug open to think about if there are any leftovers for
> this work.

Dustin suggests that we do inventory management of these decom devices ~now, rather than wait [for move to happen]. Which makes sense.

That said, removing them from DNS violates no-change-window for EOY, so I'll defer that until we return
Ok, this should be done now:

(pep8)[jwood@cruncher.srv.releng.scl3 inv-tool]$ for t in "CNAME" "A" "PTR"; do echo invt
ool search -q "`(echo -n "tegra-056 (type=:$t)"; for i in 064 076 090 112 116 174 221 290
 295 296 298 300 302 305 306 308 310 314 319 322 323 324 325 326 327 328 330 332 333 3363
37 340 341 342 346 347 350 358 359 360 361 365; do echo -n " OR tegra-$i (type=:$t)"; don
e;)`" > ~/tegra-decomm.txt; invtool search -q "`(echo -n "tegra-056 (type=:$t)"; for i in
 064 076 090 112 116 174 221 290 295 296 298 300 302 305 306 308 310 314 319 322 323 324
325 326 327 328 330 332 333 336337 340 341 342 346 347 350 358 359 360 361 365; do echo -
n " OR tegra-$i (type=:$t)"; done;)`" | tee -a ~/tegra-decomm.txt | cut -d" " -f1 | xargs
 -n1 invtool $t delete --pk | tee -a ~/tegra-decomm.txt; done

And no errors!

Finally did:

for i in 064 076 090 112 116 174 221 290  295 296 298 300 302 305 306 308 310 314 319 322 323 324 325 326 327 328 330 332 333 3363 37 340 341 342 346 347 350 358 359 360 361 365; do invtool SYS update --hostname tegra-$i.build.mtv1.mozilla.com --system-status-pk 6; done

Where the 6 pointer is decomm (grabbed by quering another decom host). and all those are successful. I think this is now done.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.