660087 - Socorro - TCBS, TCBU replacements - [tracker]

Status: RESOLVED FIXED

(Socorro :: Backend, task, P2)

Description

[K Lars Lohn [:lars] [:klohn]]

even though we're planning on using elastic search to eventually replace Top Crashes by Signature (TCBS) and Top Crashes by URL (TCBU) with Elastic Search (ES), we cannot abandon Socorro users that may not want climb aboard the ES bandwagon.  It is my intent to replace these Python cron jobs as quickly as I can. 

This bug is to track the project.

On Thursday, May 26th, jberkus, mpressman and I discussed what it would take to make this happen.  I stressed that I'd rather not make database schema changes.  This will ensure that the project will not require UI changes.  We agree that we should drop the current data in the TCBS and TCBU tables and regenerate from scratch.

One of the bigger issues in this endeavor is to rationalize the 'osdims' table.  We're currently tracking  more than two thousand OS versions, yet we only display three.  This has contributed to the TCBS and TCBU tables being grossly oversized. In turn, this means that querying the table is nearly always an additional level of aggregation. This is an unnecessary expense.

To reduce the size of the 'osdims' table, we need a way to take the 'os_name' and 'os_version' from the 'reports' table and map them onto a limited subset of the same information.  In other words, 'Windows NT'/'3.561813 sp3 Dell' should map to something like 'Windows Vista'.  We lose resolution in this mapping, but since we never display anything of greater resolution, that shouldn't be a problem.  It is important to make sure that the loss in detail will not adversely affect anything.  If anyone has objections, please speak up.

I've asked mpressman to write the queries that will be employed for this effort.  I will file additional bugs to delegate tasks.  I will also post additional thought and issues to this bug over the next week.

Comment 1

[Robert Helmer [:rhelmer]]

Do we need this for 2.2? We're not replacing any reports with ES for 2.2, only providing it as an option (configurable) for search.

Comment 2

[Robert Helmer [:rhelmer]]

Pretty sure I took this for 2.2 but it turned out not to be needed. Handing this back to lars.

Comment 3

[K Lars Lohn [:lars] [:klohn]]

long ago resolved