919026 - switch /report/index to using postgres

Status: RESOLVED FIXED

(Socorro Graveyard :: Middleware, defect)

Description

[Robert Helmer [:rhelmer]]

We have raw crashes in postgres now, and will have the processed dump soon as well (bug 907305).

Let's switch /report/index off of hbase and onto postgres only - the only places it will still be needed are:

1) priority processing (e.g. the crash was throttled)
2) downloading minidump (logged-in users only)

This means that the basic /report/index page will work even when hbase is down.

Comment 1

[Peter Bengtsson [:peterbe]]

report/index/ loads both ProcessedCrash and RawCrash. But the lesser traffic to HBase the better. In other words, we can make this bug about just switching out the backed being used for RawCrash.

Comment 2

[Brandon Savage [:brandon]]

Is there a way to completely eliminate Hbase from the equation? Since we store the raw crash in the database, and we'll soon store the processed crash in the database, theoretically we could eliminate Hbase entirely.

That could/would make it easier to eliminate Hbase as a dependency for Socorro altogether.

Comment 3

[Brandon Savage [:brandon]]

By "eliminate Hbase as a dependency" I mean it would be possible to easily stand up a Socorro that excludes Hbase (not that we would eliminate it entirely from our setup)

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Brandon Savage [:brandon] from comment #3)
> By "eliminate Hbase as a dependency" I mean it would be possible to easily
> stand up a Socorro that excludes Hbase (not that we would eliminate it
> entirely from our setup)

Yes, this is already the case - just use filesystem for raw storage. In fact this is the checked-in default in the configs right now.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> (In reply to Brandon Savage [:brandon] from comment #3)
> > By "eliminate Hbase as a dependency" I mean it would be possible to easily
> > stand up a Socorro that excludes Hbase (not that we would eliminate it
> > entirely from our setup)
> 
> Yes, this is already the case - just use filesystem for raw storage. In fact
> this is the checked-in default in the configs right now.

This is what you get if you follow http://socorro.readthedocs.org/en/latest/installation.html too.

Comment 6

[Robert Helmer [:rhelmer]]

Fixing this bug would make it possible to use Postgres for all raw JSON access I think (but not raw minidumps), and I think we should do that for the Mozilla implementation. We'll continue to store raw/processed JSON and the minidumps in HBase though (the JSON for analytic purposes and the latter for archival purposes)

It means that everything except for processing incoming crashes and downloading minidumps will work on crash-stats.mozilla.com when HBase is down or otherwise unreachable.

Comment 7

[Peter Bengtsson [:peterbe]]

Nope. We're using S3 now instead.