Closed Bug 706612 Opened 14 years ago Closed 14 years ago

Socorro Processor - our little Fennec problem

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lars, Assigned: rhelmer)

References

Details

refactor the processor to change the name of the product: if the original product name is 'Fennec' and a lookup of the 'appid' indicates that this crash is eligible for a name change, then change the product name to 'Fennec Android' code this change in such a way that we can add more products to this renaming scheme in the future. I'm assuming that there will exist a database table that associates product names and appids along with an indication as to what the new product name should be.
This bug will include putting the app ID in the processed crash output, right? Marking dependency on the breakpad side as we'll at least need to know what the key is actually called there.
Depends on: 706560
Blocks: 706801
Blocks: 706803
Blocks: 706805
Blocks: 706807
Target Milestone: 2.4 → 2.3.5
Assignee: lars → rhelmer
Depends on: 706900
OK I think this is ready for review: https://github.com/mozilla/socorro/pull/219 lonnen and peterbe already took a look at an earlier version, and I believe this addresses the concerns on that one. Two things: 1) we might want to make the exception handling more specific; just to be clear I think everything should stop if we're unable to load the productIdMap, since we'd start putting Fennec and Fennec Native crashes together otherwise, which is why it's so generic right now 2) the unit test is so huge because I copied the existing one test which runs through full successful job processing, and didn't put much effort into paring it down. What I want is to test the full process chain, and make sure that the addition of productID causes no problems and the productID->name mapping is preserved (and that if not present, things work as before).
How often will the ProductIdMap table change? As it is written right now, the only way to pickup changes is to restart the processor since the table is cached at startup. I see a couple ways to address this: 1) an update thread that wakes up every N minutes and refreshes the cache. depending on the frequency of changes, N could be in days. 2) if threads are too frightening, an access counter could be added to the cache, after X accesses to the cache it could be reloaded from the database. Since this cache is operating in a multithreaded environment, if there is to be an update during run time, we'll have to implement a locking mechanism on the cache object. I favor the updating thread idea, but that's just the way I roll.
The table will change when we add a new product. I would anticipate this would be yearly or less, although we are expecting one more in 2012.
(In reply to K Lars Lohn [:lars] [:klohn] from comment #3) > How often will the ProductIdMap table change? As it is written right now, > the only way to pickup changes is to restart the processor since the table > is cached at startup. Yes this is on purpose; we had this discussion in IRC. > I see a couple ways to address this: > > 1) an update thread that wakes up every N minutes and refreshes the cache. > depending on the frequency of changes, N could be in days. > > 2) if threads are too frightening, an access counter could be added to the > cache, after X accesses to the cache it could be reloaded from the database. I am worried about not picking up the changes at the same time, that is different processors having a different map loaded. I have a feeling that in practice we'll end up asking for a processor restart when changes to this table are made. Further, I don't expect changes to be made to this table very often, if ever. > > Since this cache is operating in a multithreaded environment, if there is to > be an update during run time, we'll have to implement a locking mechanism on > the cache object. > > I favor the updating thread idea, but that's just the way I roll.
(In reply to Laura Thomson :laura from comment #4) > The table will change when we add a new product. I would anticipate this > would be yearly or less, although we are expecting one more in 2012. AFAICT we don't really need to make entries to this table unless we want to override the product name. In some ways I think it would be clearer to just do it this way.
Reasons to list all products in this table: 1. If only some products are listed, it will not be at all clear to any unintiated user why, whereas having a "rewrite" boolean column makes it clear what we are doing, or not doing, with each product. 2. If we don't include all products, sooner or later someone will do an INNER JOIN against this table and then spend a lot of time troubleshooting why only FennecAndroid shows up in the resulting dataset. 3. Maintenance on this table is liable to be completely nil. ProductIDs for most products have not changed in over 4 years.
OK re: comment 7 I am still not convinced it's worth the effort, but I have no strong objections either. One change this code needs before being merged is to honor the rewrite column. I had been thinking we could pare this down to just product_name,productid and have the presence of the row in the table mean that the rewrite should happen, but given comment 7 I'll just make loadProductIdMap() ignore any rows where rewrite is false.
OK rebased, https://github.com/mozilla/socorro/pull/219 is ready for merge (or further comments).
Commit pushed to https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/53b346cd3050e4e63532cb1818ac5d3c288c1987 Merge pull request #219 from rhelmer/bug706612-productid-product-mapping bug 706612 - change name based on productid
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Commits pushed to https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/c864ed1925d3fcd086eb6715e83d3c377775dc6f bug 706612 - columns names have changed, just select the ones we need https://github.com/mozilla/socorro/commit/a3ebe1d6cd5093a2e863704b3ea04f64b99e9c59 Merge pull request #222 from rhelmer/bug706612-productid-product-mapping bug 706612 - columns names have changed, just select the ones we need
Commits pushed to https://github.com/mozilla/socorro https://github.com/mozilla/socorro/commit/119f1482028c8a333c7afef5498e4c42f8125482 bug 706612 - unbust unit test expecting certain SQL string https://github.com/mozilla/socorro/commit/7a5de378a4b1014ace3c3399777cdde82ea0ac30 Merge pull request #225 from rhelmer/bug706612-productid-product-mapping bug 706612 - unbust unit test expecting certain SQL string
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.