1051836 - Prune hive_results after import

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Administration, defect, P1)

Description

[Jason Thomas [:jason]]

Please create a cleanup process to remove data after import. The path is t NETAPP_STORAGE/tmp/hive_results

[[e-mail]1 hive_results]# du -hs *
4.3G	2014-07-15
4.3G	2014-07-16
4.2G	2014-07-17
4.1G	2014-07-18
3.6G	2014-07-19
1.4G	2014-07-20
4.2G	2014-07-21
4.3G	2014-07-22
4.6G	2014-07-23
4.3G	2014-07-24
4.0G	2014-07-25
3.5G	2014-07-26
3.4G	2014-07-27
3.7G	2014-07-28
3.7G	2014-07-29
4.6G	2014-07-30
4.5G	2014-07-31
4.1G	2014-08-01
3.7G	2014-08-02
3.6G	2014-08-03
4.5G	2014-08-04
4.4G	2014-08-05
4.4G	2014-08-06
4.3G	2014-08-07
4.1G	2014-08-08
3.7G	2014-08-09
3.6G	2014-08-10

Comment 1

[David Larlet [:davidbgk]]

There are a few strategic questions here:

* How do we envision the switch from previous stats to new ones? I see two options here, switching at d+1 with a highly visible difference OR logging results for a month and switching with results from d-30, the gap will only be visible for annual views in that case.

* What will be our reactivity on filling gaps when the script fails (because files are missing for instance)? Is that a week, a month? Deleting hive results without having filled gaps would result in a more painful process (re-running hive script + re-running cron tasks).

* How much space left do we have on that disk? :)

Comment 2

[Wil Clouser [:clouserw]]

I'm not sure I understand the first question.

As far as a strategy:  Why don't we just process anything that exists on disk and once it is done processing we delete it.  If we want to re-process data we re-dump it from Hive.  If data is missing, we re-dump it from Hive.  Basically, if it exists on disk, we'll eventually pull it in to our database and delete it.  The "normal" would be to have an empty directory.

Comment 3

[David Larlet [:davidbgk]]

Sorry for not being clear. We're currently filling _tmp tables in production, maybe we can reuse that data to fill real _count tables for let's say the month of August once we make the switch to our new scripts (in September?). That way the differences are less visible/brutal for addons' owners. Thoughts?

Otherwise we just have to verify that the command works for a given missing day and delete all hive_results data.

I'll create a command for that anyway.

Comment 4

[Jason Thomas [:jason]]

We have about 70GB left on this volume. We should have more space once bug 1051854 is completed.

FWIW hive data push -> mktadm host was a temporary solution as per anurag and ops was not expecting this much hive data to be stored long term.

Comment 5

[Wil Clouser [:clouserw]]

(In reply to David Larlet [:davidbgk] from comment #3)
> Sorry for not being clear. We're currently filling _tmp tables in
> production, maybe we can reuse that data to fill real _count tables for
> let's say the month of August once we make the switch to our new scripts (in
> September?). That way the differences are less visible/brutal for addons'
> owners. Thoughts?

Depending on why the numbers aren't lining up correctly, sure.

Comment 6

[David Larlet [:davidbgk]]

Jason, I think we can safely remove 2014-07-* data given that it was mostly for testing purpose and we'll not use that data anymore. Do you confirm Wil?

Comment 7

[David Larlet [:davidbgk]]

PR: https://github.com/mozilla/olympia/pull/225

Comment 8

[Wil Clouser [:clouserw]]

(In reply to David Larlet [:davidbgk] from comment #6)
> Jason, I think we can safely remove 2014-07-* data given that it was mostly
> for testing purpose and we'll not use that data anymore. Do you confirm Wil?

sure

Comment 9

[Jason Thomas [:jason]]

Removed 2014-07-* data.

Before:
147G	hive_results/

After:
81G	hive_results/

Comment 10

[David Larlet [:davidbgk]]

https://github.com/mozilla/olympia/commit/fd074305fa73293b8dae6b4d9b1add4bdbe15459

With that change, *new* imports will clean up files. We'll have to clean up older ones once the command in https://bugzilla.mozilla.org/show_bug.cgi?id=1014752#c75 has been run.

Comment 11

[Iulian Timis]

Please add STR here or mark it with [qa-] if no QA is needed.

Comment 12

[Mathieu Agopian [:magopian]]

Marked as [qa-]

Comment 13

[David Larlet [:davidbgk]]

Given the differences in results we can remove hive data for 2014-08-* too as we'll not refill as previously described in https://bugzilla.mozilla.org/show_bug.cgi?id=1014752#c75. Jason can you do that?

Comment 14

[Jason Thomas [:jason]]

2014-08-* removed.