1136149 - Clean up data, strip a bunch of legacy data for old branches

Status: RESOLVED FIXED

(Webtools Graveyard :: Elmo, defect)

Description

[Axel Hecht [:Pike]]

We have a bunch of data for old branches, which we don't really need. Haven't looked at them in ages, no objections from l10n drivers, too.

Here's my take on my local dump:

trees = list(Tree.objects.exclude(run__build__isnull=False).distinct().exclude(code__startswith='gaia'))
> [<Tree: tb30x>, <Tree: fx35x>, <Tree: fx40x>, <Tree: sea20x>, <Tree: fx36x>, <Tree: tb33x>, <Tree: sea21x>, <Tree: fennec40x>, <Tree: fennec10x>, <Tree: lorentz>, <Tree: tb31x>, <Tree: fennec11x>, <Tree: calendar10x>, <Tree: calendar10b3pre>, <Tree: weave>]
set(t.l10n for t in trees)
> set([<Forest: releases/l10n-miramar>, <Forest: releases/l10n-mozilla-1.9.1>, <Forest: releases/l10n-mozilla-1.9.2>, <Forest: weave-l10n>, <Forest: releases/l10n-mozilla-2.0>])

for tree in trees:
  tree.delete()

AppVersion.objects.exclude(trees__isnull=False)
> [<AppVersion: Firefox 3.5>, <AppVersion: Firefox 3.6>, <AppVersion: Fennec 1.0>, <AppVersion: Thunderbird 3.0>, <AppVersion: Thunderbird 3.1>, <AppVersion: SeaMonkey 2.0>, <AppVersion: Fennec 1.1>, <AppVersion: Firefox Lorentz>, <AppVersion: Weave Addon>, <AppVersion: Lightning 1.0>, <AppVersion: Firefox 4.0>, <AppVersion: SeaMonkey 2.1>, <AppVersion: Fennec 4.0>, <AppVersion: Thunderbird 3.3>]

AppVersion.objects.exclude(trees__isnull=False).delete()

Forest.objects.exclude(tree__isnull=False)
> [<Forest: releases/l10n-mozilla-1.9.1>, <Forest: releases/l10n-mozilla-1.9.2>, <Forest: weave-l10n>, <Forest: testpilot-l10n>, <Forest: releases/l10n-mozilla-2.0>, <Forest: releases/l10n-miramar>]

Forest.objects.exclude(tree__isnull=False).delete()

Repository.objects.filter(forest__isnull=True).exclude(push__changesets__run__build__isnull=False)
> [<Repository: releases/mozilla-1.9.1>, <Repository: dom-inspector>, <Repository: releases/mozilla-1.9.2>, <Repository: releases/comm-1.9.1>, <Repository: projects/firefox-lorentz>, <Repository: releases/mobile-1.1>, <Repository: releases/comm-1.9.2>, <Repository: releases/mozilla-2.0>, <Repository: releases/mozilla-2.1>, <Repository: releases/mobile-2.0>]

Repository.objects.filter(forest__isnull=True).exclude(push__changesets__run__build__isnull=False).delete()

Application.objects.exclude(appversion__isnull=False)
> [<Application: Weave>]
Application.objects.exclude(appversion__isnull=False).delete()

This gives me the following cleanups, in table name, prior value, removed rows:

l10nstats_run 467896 128484
l10nstats_run_revisions 1126970 338888
l10nstats_run_unchangedmodules 2724948 715256
life_forest 17 6
life_push 196476 33040
life_push_changesets 1016302 57770
life_repository 1183 425
life_repository_changesets 1418324 277591
life_tree 43 15
shipping_action 27793 7913
shipping_application 7 1
shipping_appversion 182 14
shipping_appversiontreethrough 497 14
shipping_milestone 862 142
shipping_milestone_signoffs 47510 7469
shipping_signoff 15155 4226
shipping_snapshot 3247 3127

The snapshot count is sad, because that's a symptom that snapshots had been broken for ages.

Need to investigate if we need the following archived repos and forests:

Repository.objects.filter(archived=True)
> [<Repository: mobile-browser>, <Repository: releases/comm-2.0>, <Repository: releases/comm-miramar>, <Repository: releases/mozilla-miramar>]

Forest.objects.filter(archived=True)
> [<Forest: releases/gaia-l10n/v1_0_1>, <Forest: releases/gaia-l10n/v1_1>]

(Forests, yeah, I think so, repos not so much)

Comment 1

[Axel Hecht [:Pike]]

...

I suspect that the repos we have archived are listed with builds because they share revisions with other repos, I find builds like sea_central and fennec_central.

Let's get rid of them as they are old branches (we know).

Repository.objects.filter(archived=True).delete()

l10nstats_run 467896 128484
l10nstats_run_revisions 1126970 338888
l10nstats_run_unchangedmodules 2724948 715256
life_forest 17 6
life_push 196476 35771
life_push_changesets 1016302 68792
life_repository 1183 429
life_repository_changesets 1418324 364389
life_tree 43 15
shipping_action 27793 7913
shipping_application 7 1
shipping_appversion 182 14
shipping_appversiontreethrough 497 14
shipping_milestone 862 142
shipping_milestone_signoffs 47510 7469
shipping_signoff 15155 4226
shipping_snapshot 3247 3127

Comment 2

[Axel Hecht [:Pike]]

FWIW, I did spot check if the deleted runs have corresponding data in ES, and it seems that we deleted their build data before converting to ES, and thus, they don't.

Comment 3

[Axel Hecht [:Pike]]

I've done this on https://l10n-dev-sj.mozilla.org/ now, doing some tests there next.

Comment 4

[Axel Hecht [:Pike]]

Done on prod, marking FIXED.