hg.mozilla.org repository pruning

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
5 years ago
2 years ago

People

(Reporter: bkero, Unassigned)

Tracking

({spring-cleaning})

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/782] )

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
In an effort to save space towards our move to hg using local disk I'd propose to prune some old repositories. In particular are these repositories, their size, and how long it's been since their last change.

Many of the groups will need to be contacted to see if they still care about the code, and releng will need to be consulted to see if they still have procedures that depend on these repositories.

The proposal was to move the unwanted items onto an 'hg-archive' volume in case someone realizes they need something later.

actionmonkey - 2008/07 - 520MB
actionmonkey-tamarin - 2008/07 - 82MB
cvs-trunk-mirror (maybe important for history?) - 07/2009 - 500MB
mobile-browser - 03/2011 - 13MB
penelope - 2010/10 - 3.2MB
tamarin-central - 2010/03 - 63MB
tamarin-tracing - 2008-10 - 83MB
tracemonkey - 2011/07 - 1.1GB
xforms - 2012/02 - 2.5MB

incubator/offscreen - 402MB
incubator/symbian - 322MB
incubator/taskfox -  394MB

l10n/l20n - 2009/07 - never used

MANY things in labs/

projects/2007-configure-rewrite - 2007/08 - 400MB
projects/mccoy - 2009/06 - 1.2MB
projects/firefox-lorentz - 2010/04 - 791MB
projects/addonsmgr - 2010/05 - 453MB
projects/kraken - 2010/11 - 5.4MB
projects/private-browsing - 2011/05 - 1.2GB
projects/places - 2011/07 - 1.2GB
projects/electrolysis - 2011/09 - 485MB
projects/devtools - 2012/02 - 1.4GB
projects/devtools2 - 2011/10 - 910MB
projects/nanojit-central - 2012/01 - 4.7MB
projects/addon-sdk-release - 2012/03 - 19MB
projects/addon-sdk-beta - 2012/03 - 20MB
projects/accessibility - 2012/04 - 492MB
projects/jaegermonkey - 2012/05 - 1.2GB

integration/mozilla-inbound-dead - 2013/05 - 1.1GB (pushlog broke once, just a dup)

releases/comm-miramar - 2011/09 - 153MB
releases/mobile-1.1 - 2010/06 - 12MB
releases/mobile-2.0 - 2011/04 - 9.7MB
releases/mozilla-2.1 - 2011/04 - 1.2GB
releases/mozilla-2.0 - 709MB
releases/comm-1.9.1 - 97MB
releases/comm-2.0 - 153MB
releases/mobile-5.0 - 2.9MB
releases/mozilla-miramar - 805MB
releases/mozilla-1.9.1 - 544MB
releases/mobile-6.0  - 3.9MB
releases/mozilla-mobile-6.0 - 709MB
releases/comm-miramar - 153MB
releases/mozilla-mobile-5.0 - 1.2GB
releases/comm-1.9.2 - 103MB

weave-l10n/ - the whole thing - newest change was 2012/11, before that 2011/07 - 6.3MB
For clarity when you say "prune" do you mean "rm -rf $1" or something else?

I can see many of those being marked/setup as read-only and not actively handled/managed beyond hgweb/clone-over-ssh, but I can't easily see most of these being removed.
(Reporter)

Comment 2

5 years ago
I mean like I state in the third paragraph, wherein these repositories are moved onto a backup archive somewhere never to be seen again -- unless requested.

This is an effort to save disk space since currently the hg web mirrors are running at 95% disk full with 16GB available.
(Reporter)

Updated

5 years ago
Blocks: 948159
(Reporter)

Comment 3

5 years ago
Another thing to be considered would be to archive user repositories of users who are not with us anymore. Do we have a policy on how long we retain/make available for those?
Keywords: spring-cleaning
Hal is there anything you can do to help coordinate ownership of these repos so we can move this forward?  Thanks!
Flags: needinfo?(hwine)
See Also: → bug 650518
See Also: → bug 877975

Comment 5

5 years ago
This is actually tricky for the l10n dashboard automation.

That doesn't expect repositories to go away, so we'll need to figure out what to do with that.

For at least some of those repos, it might be just fine to throw the l10n dashboard data into a backup and remove it. Not sure if that's for all of that, and I'm also not sure how to actually have a conversation about that. Doesn't feel like I'm the only person that should make the call, but I also don't know if anyone else cares.

I'll attach a list of repos the l10n dashboard knows about in a second. There's more food for thought in there, too.

Comment 6

5 years ago
Created attachment 8366865 [details]
repos known to the l10n dashboard, don't remove without coordination with l10n
Based on offline chats with cshields, and in similar light as comment 5, there are not any repos under release/* gaia/* integration/* that can go completely offline.

Whether certain repos in projects/* can go offline is up to the owners of those repos (or their descendants).

However, not all of these repositories need to be given the same high service level as our active ones. There are repositories that could be made read-only, and not served in an HA fashion. Ideally, the URL would remain the same (symlinks to the rescue?), or the "old" url(s) should serve a page on how to find the archive location.

Let me know if that is an option we can pursue.
Flags: needinfo?(hwine)
(Reporter)

Comment 8

5 years ago
Got permission from jorendorff to move to the archive: actionmonkey actionmonkey-tamarin tamarin-central tamarin-tracing tracemonkey, so they were moved to /repo/hg/retired_repos/, then removed from hgweb.config, then deleted from the hgweb hosts.
(Reporter)

Comment 9

5 years ago
Talked to robcee about removing devtools and devtools2 projects, so done
(Reporter)

Comment 10

5 years ago
Talked to luke about removing projects/jaegermonkey and projects/nanojit. No problems there.
Mark can you comment on :

releases/comm-miramar
releases/comm-1.9.1 
releases/comm-1.9.2
releases/comm-2.0

I believe penelope is a dead project but ain't sure whom to check with (Gerv looked at it not too long ago so he might be a good candidate).

Comment 12

5 years ago
Those are referenced by the l10n infrastructure still, and need code/database changes on my side to go away.
(In reply to Ludovic Hirlimann [:Usul] from comment #11)
> releases/comm-miramar
> releases/comm-1.9.1 
> releases/comm-1.9.2
> releases/comm-2.0

As Alex said, these are still referenced by the L10n infra. Also per comment 7 these shouldn't really go away, as they are effectively release archives.

mxr.mozilla.org also references some of these repo, and it is likely socorro might as well.

They could certainly move to a lower service level space, but I'd kinda expect all the same urls to work etc (referencing old repos can be useful now and again).
hgweb5.dmz.scl3.mozilla.com alerting for low disk.

Comment 15

4 years ago
In bug 1003132, I modified the l10n automation to stop polling all repositories it knows about. I flagged repos that I don't care about, and if we were to be archived, it won't break the l10n automation anymore.

I didn't investigate if any of these have other stakeholders, fwiw.

I stopped polling the following repository trees:

releases/gaia-l10n/v1_0_1
releases/gaia-l10n/v1_1
releases/l10n-miramar
releases/l10n-mozilla-1.9.1
releases/l10n-mozilla-1.9.2
releases/l10n-mozilla-2.0
testpilot-l10n
weave-l10n

I also stopped polling the following individual repos:

mobile-browser
projects/firefox-lorentz
releases/comm-1.9.1
releases/comm-1.9.2
releases/comm-2.0
releases/comm-miramar
releases/mobile-1.1
releases/mobile-2.0
releases/mozilla-1.9.1
releases/mozilla-1.9.2
releases/mozilla-2.0
releases/mozilla-2.1
releases/mozilla-miramar

Updated

4 years ago
Assignee: server-ops-devservices → bkero
(Reporter)

Updated

4 years ago
Duplicate of this bug: 948159
hgweb7 hit 95% today while resetting cedar. need to do something Real Soon Now.
Fri 04:53:28 PDT [5185] hgweb7.dmz.scl3.mozilla.com:Disk - All is CRITICAL: DISK CRITICAL - free space: / 15396 MB (5% inode=56%)
(Reporter)

Comment 19

4 years ago
I've removed the repositories found in comment 15.
(In reply to Ben Kero [:bkero] from comment #19)
> I've removed the repositories found in comment 15.

If you're looking for some more targets, I've had a quick glance and I some of the following look promising (but do double-check with owners)...

Repos that were temporary backups, so can be removed now:
https://hg.mozilla.org/projects/holly-new/ (ask #releng, this is a rentable repo backup)

Repos that were imported/merged into another, so may not be required any more:
https://hg.mozilla.org/mobile-browser/ (ask :mfinkle ; can't remember if the history was preserved during import)

Repos that are clones of mozilla-central & but are no longer hooked up to releng automation and no recent commits:
https://hg.mozilla.org/projects/accessibility/ (ask :MarcoZ or :tbsaunde)
https://hg.mozilla.org/projects/ionmonkey/ (speak to the js team)
https://hg.mozilla.org/projects/places/ (ask :mak)
https://hg.mozilla.org/projects/private-browsing/ (ask :ehsan or :jdm)
https://hg.mozilla.org/projects/profiling/ (ask :ehsan)
https://hg.mozilla.org/projects/2007-configure-rewrite/ (ask :ted)

Empty repos (would be nice to remove to clean up the listings on hg.m.o):
https://hg.mozilla.org/tbbuild/
https://hg.mozilla.org/automation/fennecpm/
https://hg.mozilla.org/automation/firebug/

Repos possibly moved to github:
https://hg.mozilla.org/labs (I don't know these well enough to say for sure for each - but I think you may be able to remove a bunch of these after speaking to owners - last commits quite a while ago and https://github.com/mozilla?query=weave)

(https://github.com/mozilla?query=addon-sdk ; ask :erikvold)
https://hg.mozilla.org/projects/addon-sdk-beta/
https://hg.mozilla.org/projects/addon-sdk-jetperf-tests/
https://hg.mozilla.org/projects/addon-sdk-release/

Also perhaps the deprecated/* repos under here?
https://hg.mozilla.org/services

Updated

4 years ago
See Also: → bug 1026620
More warnings:
Thu 06:43:23 PDT [5115] hgweb7.dmz.scl3.mozilla.com:Disk - All is CRITICAL: DISK CRITICAL - free space: / 14605 MB (5% inode=56%): (http://m.mozilla.org/Disk+-+All)
hgweb5.dmz.scl3.mozilla.com reach 90% today which is a warning.
Guys,

hgweb3.dmz.scl3.mozilla.com has just reached critical 95%.
nagios-scl3	 Thu 02:13:05 PDT [5021] hgweb3.dmz.scl3.mozilla.com:Disk - All is WARNING: DISK WARNING - free space: / 16144 MB

Updated

4 years ago
Depends on: 1054754

Updated

4 years ago
Component: Server Operations: Developer Services → Mercurial: hg.mozilla.org
Product: mozilla.org → Developer Services

Updated

4 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/225]

Updated

4 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/225] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/782] [kanban:engops:https://kanbanize.com/ctrl_board/6/225]
(Assignee)

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/782] [kanban:engops:https://kanbanize.com/ctrl_board/6/225] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/782]
(Reporter)

Comment 25

3 years ago
Back to the queue
Assignee: bkero → nobody
QA Contact: nmaul → hwine
(Reporter)

Updated

3 years ago
Depends on: 975519
QA Contact: hwine → klibby

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.