1363550 - squash filesystem crashstorage classes

Status: RESOLVED WONTFIX

(Socorro :: General, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

In Socorro, there are 7 filesystem crash storage classes, two of which are just renamed versions of other classes.

They're predominantly used by the collector and crashmover and FetchTransformSave apps of which the processor is one.

Between the Mozilla Socorro configuration and the codebase, the following classes are used:

* FSTemporaryStorage: collector and crashmover
* FSPermanentStorage (alias of FSLegacyRadixTreeStorage): FetchTransformSave apps
* FSLegacyRadixTreeStorage: middleware
* FSDatedPermanentStorage (alias of FSLegacyDatedRadixTreeStorage): Processor

We no longer use the collector and crashmover, so those uses will get removed in bug #1361832.

We no longer use the middleware, so that use will get removed in bug #1353371.

When I extracted the collector/crashmover code for the socorro-collector project, I squashed this family of classes down to two: FSRadixTreeStorage and FSTemporaryStorage.

I want to pull in the socorro-collector squashings and then change the Socorro users of filesystem crashstorage to use FSRadixTreeStorage and FSTemporaryStorage.

Why? There's about 2,000 lines of filesystem crashstorage code and tests. There are some 60 tests. I claim that the bulk of the work for this bug is already done, so this should be like a half day of work and save us a bunch of time as we dockerize, deprecate, rewrite, etc.

This bug covers doing that.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to work on this week. It'll make dockerizing easier.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

My notes.

All the instances of "external.fs.crashstorage" in the Socorro codebase except what's in unittest, config, and docs:

./socorro/app/fetch_transform_save_app.py:
  104 :             'socorro.external.fs.crashstorage.FSPermanentStorage',
  106 :             'socorro.external.fs.crashstorage.FSPermanentStorage',
./socorro/collector/collector_app.py:
  100 :         default='socorro.external.fs.crashstorage.FSLegacyDatedRadixTreeStorage',
./socorro/collector/crashmover_app.py:
   53 :                 'socorro.external.fs.crashstorage.TarFileCrashStore',
./socorro/collector/wsgi_generic_collector.py:
  147 :         default='socorro.external.fs.crashstorage.FSLegacyDatedRadixTreeStorage'
./socorro/external/fs/fs_new_crash_source.py:
   19 :         default='socorro.external.fs.crashstorage.FSLegacyDatedRadixTreeStorage'
./socorro/middleware/middleware_app.py:
  140 :         default='socorro.external.fs.crashstorage.FSLegacyRadixTreeStorage',
./socorro/processor/processor_app.py:
   22 : from socorro.external.fs.crashstorage import FSDatedPermanentStorage


All the instances of "external.fs.crashstorage" in the Socorro configuration:

Prod:
    7 : socorro/collector/storage.crashstorage_class	socorro.external.fs.crashstorage.FSTemporaryStorage
   65 : socorro/crashmover/source.crashstorage_class	socorro.external.fs.crashstorage.FSTemporaryStorage

Stage:
    8 : socorro/collector/storage.crashstorage_class	socorro.external.fs.crashstorage.FSTemporaryStorage
   53 : socorro/crashmover/source.crashstorage_class	socorro.external.fs.crashstorage.FSTemporaryStorage

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3767

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I nixed that PR for now. It'd need review time and it's probably not worth that investment right now. If we decide to do it later, we can resurrect the PR.

Given that, I think I'm going to close this out as WONTFIX.