862955 - tooltool repo nearing disk capacity

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

The tooltool repo has 26GB free at the moment.  With each b2g snapshot using  ~650-850MB, we should plan what we're going to do to avoid an out-of-disk space situation.
- move to a new server?
- increase filer capacity?
- implement a b2g snapshot retention policy?

Comment 1

[Armen [:armenzg]]

Maybe we archive all of the snapshot that are not the latest for one of our branches since it is *very* unlikely that we will trigger b2g jobs for very old changesets.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, regular uploads are not the intended use of tooltool, nor are 850MB downloads.  Yikes.  Depending on how often these are uploaded and how easy it is to purge old versions, ftp may be a better choice for these files.

Simone, I see now why you wanted to gather usage statistics on tooltool downloads!

Comment 3

[Armen [:armenzg]]

We don't do snapshots anymore.

Fine if we close?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Are the older snapshots deleted?

Comment 5

[Armen [:armenzg]]

dustin, how could I check which snapshots have *not* been accessed in the last 6 months?

(CCing myself - I fixed the setting)

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

The best we have right now is checking the Apache access logs on the webheads.  And those only go back to June (3 months).  I'll get that for you now (or at least, a list of the hashes that have been accessed)

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Since mid-June, these are the hits, with counts:

> 1 cf0802c788b26d6f29ada54b986956417b0619a5850c9974ca14e89fc0f2bf5c5c38af7a50cf2d68faa809b1024190eda0154751c4c377657a6802ba50a30cc2
> 1 3d0648af43bdaa4eea6ba43a2f1f3cd6ac29b7ef1a54c55612c74599f386d945c0b609ba4abf833903319fa3144a40d626cdfbe3211c93ddd4d8d6563faad253
> 1 66ae7e1bd47c5ee958f070d380b550e849ebb7d8180eb7f07dec5079fc7db667743674a806c708ba825ab3124131a07ae4d53a4061a52363f062a261ccfcf4ba
> 1 254cdd70474f506269184cd1847afb73f8f7170d9a048dd3ba0e4c9eff688b946a6159bf41e6003811f45b77b53ba907802fdec1a8b338606650cb3b4bd8e24c
> 1 ae22b181014c03a5e400ebe2333834ff33ca35c5da5af2a410042c205895c0cfec87b4546ab14d361f34a4a052fbb0bb4e58ed8bee0567677ffe565b271d191e
> 2 c5cb7b45abce0d3c0c29bd7a57d5cd3c5667fddcbff364a81a949cee0c68733b4cc9da3ffc427c31a718e7df51e9ca7f6aee82773e40b044e102f573eba7fd90
> 2 e06156ebb2c4d66be14d0969ea965f5b6de5683cd6492e924af1e343919c9d787544b9747fc3f3fbbca67c7584e18baf576af9cf5d38d5580f9a0959222978b6
> 4 68327ed36d8a53615fd66a300d5c022517f2f3ea0199b9d95f80683bbf1d0a154266700c8bde235467cc707c4c37341cd3f5e9816bb2c4b66efc96f9e08c92df
> 4 c5bc3424d9a52bd12dc6d849e8bd111514a7edfb84180bf8dd67f77e925fea95e9f41afade187694e1724b3d9b817e7cf2ba62be000a9f11db1e7ff689bb85fd
> 4 a6b8046bd9485f9387dcb1c14b8d442822f02b1caa61b653e8b6cfd96906deadfb4b29809f2cd2b71f919b321d97dd2ebec6020c15f6d485f1641c0f710a762f
> 4 79cf8ecf3c8faa18c4f485eed5f611fdaea25585f2ab3203eb912185c644179708b283c6222661006c145fa67d18e39c28fdf76ebe971ef5e0fbaf348f16a1ac
> 4 209d6875524e6f81c158fe7d8c543683747cb186a53ddb729c6437273404f69468d4c91bfd74a26448ad2a726da1a9b48e6243602cbe60b5d0e0f97cdb29e2cc
> 5 fe1a86f1b21030ef89bef83051b22dca22f0836c311a6e6565cb1115ed24dae654ad649d05c8f0a547dc8dfa2e94c4b694651a8da249c23532bf0944eddcb18d
> 11 b042722f6ff19d7921adacebe45a67d9e48056d29938ac79335c39adf175d44e68bc2ef484185911856b21b9d732a991e264e136f553529f91f9b7eb06437434
> 12 3e106d37151d8a54a4dec280d5274426c09b44a6e77e4eb369fe32b3fb9544d2523e39a0a947138c6944b5dd51000026b080cb5b76ba74d031b140eae6a91e62
> 13 33c21b42836fe26ae097449a756724f961a6ca032f08e43e89c225dbb9ba6e15e3f4f8cb42cce2b40129cc23d864bf43f87c8e03f50ecae4837d96d0f1e6829c
> 20 9ba4f195052e0d5eaae734375aed52f52b7b3337fa8caf01a61830529266e282be19697095c88135379c984d7242ec36b384cf81155176bb6527e6621f32eed3
> 24 8208645d24ac87975a091ff66a90c20589ff8945936ed9b16ca81976c59bf1166ed9f79709698d435480774fba8ed9f9f178dc189305c86162acac8fda19830e
> 96 8201020a5214337cf19d43bb3fde8c4624d7a7a71f04a4522a34783194694fe5e66f8087acf6fe7080d7bf787b8311dd9ed8facfbac39198d0121f817b8eef32
> 103 bf721decb42e4828a6efa795099838b4983855a62579aa63795e5287c9c26441149eb41329a81056d0a432e8e9e2043f177d9896f6654b2f656a417f6c8dc9d1
> 105 27aced8feb0e757d61df37839e62410ff30a059cfa8f04897d29ab74b787c765313acf904b1f9cf311c3e682883514df7da54197665251ef9b8bdad6bd0f62c5
> 105 09373556ddb4325897b9e008184228f9088b4c8c22bacf4fa2d39793ecfd264316ad69c2bc8082229ad7fdb80f89154c7b995a60f9b18beb1847e7111e7e69b2
> 106 2483df1a949df53d02ca33a87731cedd8f7cd07114d723bde1addf63fd71154c23b6f11f64f390b9849121725fb53a402db8df2f96a3673ec52416f45260f79d
> 118 cbda63d37b8db104a0fddcb5d8ffe1da173922805863905cb82e819c80cf6032caffa49903cb74e8c703da35df876cc305a7d7b989a52c045e99845e20718f7e
> 122 e1d76c4c9d12edfba0d7c8db4cd37669d90e6824ee3d1f949dddc0f168815b115f9809a335d3ee0025432df995f898d9dc7545ee89e499944ed84e2407fc54b8
> 211 98069e80018e0b6b155ab688332f4b8daada4bf0c28cc43399f99c33ffdaa0de475df05df2aff1cea5272dea4f82cc6775b4fa658c02352be7560b7f171d70f4
> 229 3db7c58c2002e96e6c0c34468f3349ca7de17c7473e07edec054ee3083ae39e6ead0e2a8f9177c1ce5891539073280e1afda1a6518f78e999dddc5d3f02a1e18
> 229 57c69c5f55722f197ca6a5bd7d366d3562df474da8d98a6df753830c8234c269c1a174e518168b801e39290db081ef192a8fa24fcc5133039b1ad489dfe45f48
> 278 adc4d2ab02faf8e141a655502966ff5de862315b8acd543c8cdb859d1356f78fa0a988ede0d3c66a5b1125a413b811951386a202e168a195216bec73289f80f3
> 297 4ff723d8a372e2af95e62efb8739517943f362b515b4cee78006137a8dbc97f3b42edbd2e17429eaa7fa16b7f24594ed058b7032bcb7aa2ecfe9788a212dda4d
> 325 16e74278e4e9b0d710df77d68af1677c91823dccfc611ab00ee617298a63787f9f9892bd1a41eccb8d45fb18d61bfda0dbd1de88f1861c14b4b44da3b94a4eca
> 350 7a9a71799a114791868a2e631c4593f2885bb38f2620b83f100acf17581b3631f11e7c87aee30a81fc754074213347c68bef4bfa8a6906d09050165d04bc37fb
> 350 6f65553e882316582b944e46c659915a1b907c4a326104cb31d81356330dddacba757e3eafbd282063da0e670c3c5d6b9a0905ab88da84b47848d810c37571cb
> 409 af1d883664bf4f02637060855d76026cceffd28a796ab3d65e16f1b0b6d1a8d5bb634388152ba63ecbfa2fbb1bd59317c0cfd508e163eaf950923b6b999369bc
> 429 a75243cae46f29842c8c4de367df48a0ce4205621601beb50412d8e4b41b173285edae0dab7e51d303999ed6105ceb65d8bdf4721ae4a59d359da48933026b29
> 533 0c21a793d87773b3ed82d141bea33869a6b647e5dddab3ecfee1738751e5c8d3d0b9ca45d7453ef88d335949c73f935c37de3521978b318ca51d8369be894b92
> 542 225ee9bb331a08ec496acc47ef8a1e9d9a21c290bfe22271cdc6ee2d10befecc372faf2a718e3e0ad5c36938a9b94f5d6b6b2d999880bc476c576856f1c99812
> 714 0bbba7483a483803fa0277ddeeb3f8f9592b57cb83d57794c366fb8a3541b47f0ab901c071b7ceb6828bceec7170e5d5c05c5b1dcc926c70f3da46a7e224f078
> 962 e14ccefd965372a57c540647b2b99e21a4aa82f81a8b9a9e18dac7cba4c3436181bef0dfab8c51bcb5c343f504a693fdcfbe7d609f10291b5dd65ab059979d29
> 1015 4754612c52330f4d25a250eb52adbf245950e3411a46f45836950b96fdff99e37e3563667d9a0069a7097f3900af3710d9e94010a26bf2e7e1a27c97f37ef447
> 1029 2c5c26a44402f974c2a4ccffb07ea1ac2d01d84dc3e4281fef6e047a62606811a16534d034477dfd9be055a07d931b17ca4e883c8edcd1f8d3a8c91b150e2288
> 1049 d4a0da54e75c27cd2f535e66b586f119ef08b3bde4a9eee03662d296b3434189c542c0a7e7a75954030c04396a9823e22e1f884f5d87c0f4017944cd50ff38de
> 1197 50eb6fa636403f444bab10aee2370f1ac624a1e0105566639c7c266e784dbc6e4cd3901bbd11f53c4c63b2a6d2d07a603b3d9c333f5049bdc7a816b7d225631b
> 1656 cea07d65a39a244961f183711b14d054c90690b69a79d89a3783f9a634f9ace7f6e70033e963a4f58ca8482b3aec8d4c5d3227cc7a0bc61e6afeccf2acc1a789
> 2126 e26e3501b4119724f94d5d3edcdfefde00faceddab3114c9c3b118fec11118d6f2293ea3c25426ec5ba7c7e0fe14d8349dbb8fa940e2f04ea83e26694d933696
> 2501 c9f93f7abf2fc5706ecc07787c30970980931d230047df7c3d180aad99ab144cc733188d13f1538b293161e1ee59b1c0c9ff09f4ebcdb78f2958eafa06ae4108
> 2598 9f2150350e6fb2e8fe8744f47c02799865de16f8539844374188c2b421d8a044c213af57c8fe9dc7ad2038150687b50eba251dc782fa606674a86d8acc9e3dad
> 2612 aa60b13458fabcb60c671e80db4dfc0cda3090fc5df27215cec3fb01f9cf2ac4403c93be99d1816a1f2d903febd4b027f7193a5fcec1aacf18ebb419ee9f32d7
> 3046 6cd04e8ec44c6fef159349c22bd0476891e4a2d46479f9586283eaf3305e42f79c720d40dfec0e78d8899c1651189b12e285de60862ffd0612b0dac7a0c336c6
> 3864 be67a012963a5c162834f9fcb989bcebd2d047dcb4e17ee23031b694dcf7cdfd6d7a6545d7a1f5e7293b6d24415403972f4ea1ab8c6c78fefcabfaf3f6875214
> 4079 236362c71c433971c36b46d34e8560342435718364bc390df8de6a33249fb1fbf4fc3d0143f1e22bca262a7af7dc1b277a920bfde3ee8197eb07db2e7cef3e1f
> 4121 fcf629c815b5cbed7858d7697815f355275dcc6b060ae5455b4b31fde0d78ebc176927564a5353ceacdb9f9c9bfc1357f1341bf6ba844c25153a89664e661510
> 4705 86662ebc0ef650490559005948c4f0cb015dad72c7cac43732c2bf2995247081e30c139cf8008d19670a0009fc302c4eee2676981ee3f9ff4a15c01af22b783b
> 6103 ccd13527c0ba3979f4030eae713e6529e916a701b9b16a371256d92618e914fb0fd2ba70547efca0e93d02010c89eeb7c222d40a9b6f26fab80911386cdfecc7
> 6887 36681be904b20a52dbebf38b86466026430d59adb0e72428ae7557a442d037eb378d278aab181b04a753821ff0a99b6228380d59f86ddd5fbf291284fe54932b
> 6906 b833dae269b02fec9a0549f467a78717a7b2bf96512caafa3736efe72610b50c5d2073b68afcdb2fea0779e2007e5ec9efc25b14d94f06e194e4ac66d49c676e
> 8144 cba263cef46b57585334f4b71fbbd15ce740fa4b7260571a9f7a76f8f0d6b492b93b01523cb01ee54697cc9b1de1ccc8e89ad64da95a0ea31e0f119fe744c09f
> 8618 ea03de74df73b05e939c314cd15c54aac7b5488a407b7cc4f5f263f3049a1f69642c567dd35c43d0bc3f0d599d0385a26ab2dd947a6b18f9044e4918b382eea7
> 9285 e156e2a39abd5bf272ee30748a6825f22ddd27565b097c66662a2a6f2e9892bc5b4bf30a3552dffbe867dbfc39e7ee086e0b2cd7935f6ea216c0cf936178a88f
> 9437 fa70d0a5f340ebec8a3aa9df65157ad174e7880ee14e3d3c9e07532343c02acf176e54bb7de69b74a0218697d64c5b4b865c64ba3211cc36a01ba5d78b844151
> 12013 8ff42509ecebfd7e20f8fac9987ed2b2c04942641eead674ee66f74014c5153f1c20080cd3ccb243af76ca7432df3c3f5b5ae08a478fd2817e62661a4edb437c
> 12463 2005a41fe97a5e00997063705f39d42b6a43b1cf7ba306cbc7b1513de34cdcd050fc6326efa2107f19ba0cc67914745dbf13154fa748010a93cf072481ef4aaa
> 12813 7a7e9edf3aaa2cd5f7d923a82a65ae0bb4d13082736ebe52f06a7fc133c5a0dcb304d516a7859c542dcab598afb8bd6a2d2233ce481497f5c8f76709074920ad
> 14660 56fc5dc8c5e8cf46605667db153477e32bdf724bfa989663bbbed7e52ee47a4cb8cf7f8bf49b82b7e15fc89efe59a0f1656ca4b7a4ffcc8de31f9b2229549dab
> 33292 d4297e762649b174070a33d039fd062edd9f29a751650f0508327a6cf366b3a35fe24e7cd0f7b728d74f7d15399f9c1adc5b178e5803a3a66bfce7a8dcd62daa
> 39449 0748e900821820f1a42e2f1f3fa4d9002ef257c351b9e6b78e7de0ddd0202eace351f440372fbb1ae0b7e69e8361b036f6bd3362df99e67fc585082a311fc0df
> 58861 199236aefecc1657cdc1b791ec38c8184557ab9249aff9c63a74abf73edc1dc0ea36b19b558f34ca3b14f8a511b10bcf37408b19701929522b4dc22dbaddcbe9

Comment 8

[Armen [:armenzg]]

I will move these files into a temp dir.
If nothing breaks in a week (nothing should) we will delete them.

Thanks dustin.

Comment 9

[Armen [:armenzg]]

Attachment: files_to_be_deleted

Planning to run this as root:
cd /mnt/netapp/relengweb/tooltool/pvt/build/sha512
mkdir to_be_deleted_soon
for file in `cat files_to_be_deleted`; do mv $file to_be_deleted_soon; done

Callek, edmorley: would you be fine if I did this tomorrow morning?

These are files in tooltool that have not been hit in the last three months.

Comment 10

[Ed Morley [:emorley]]

I have no idea as to what might (or might not) be affected by this, 302 someone else.. :-)

Comment 11

[Justin Wood (:Callek)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #9)
> Created attachment 807344 [details]
> files_to_be_deleted
> 
> Planning to run this as root:
> cd /mnt/netapp/relengweb/tooltool/pvt/build/sha512
> mkdir to_be_deleted_soon
> for file in `cat files_to_be_deleted`; do mv $file to_be_deleted_soon; done
> 
> Callek, edmorley: would you be fine if I did this tomorrow morning?
> 
> These are files in tooltool that have not been hit in the last three months.

WFM, lets plan to leave the old dir in place at least until after-summit though, so we have time to shake things out.

Also make sure before we delete it that we have run at least a full set of jobs on all b2g18 branches and esr10/17/24!

Comment 12

[Armen [:armenzg]]

edmorley: more than technical expertise I was just wanting to let you know and if you had any objections.

Callek: good idea on triggering jobs on old branches

Comment 13

[Ed Morley [:emorley]]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #12)
> edmorley: more than technical expertise I was just wanting to let you know

Thank you :-)

Comment 14

[Armen [:armenzg]]

I completely forgot about this.
I will try tomorrow morning.

Comment 15

[Armen [:armenzg]]

I moved the files.
Let's see if anything breaks!

Comment 16

[Armen [:armenzg]]

I removed the files.
[[e-mail]3 ~]# cd /mnt/netapp/relengweb/tooltool/pvt/build/sha512/
[[e-mail]3 ~]# rm -rf to_be_deleted_soon/