There have been a number of minor data discrepancies that have turned up while troubleshooting issues from the 1.8 rollout. We fixed a few of the issues during the rollout, and have had to dig into data issues after the rollout. All of the data issues seemed to be related to the data collected between the 1.8 rollout and the rollback to 1.7. Some related bugs filed during / after the rollout: https://bugzilla.mozilla.org/show_bug.cgi?id=598160 https://bugzilla.mozilla.org/show_bug.cgi?id=597514 https://bugzilla.mozilla.org/show_bug.cgi?id=596881 https://bugzilla.mozilla.org/show_bug.cgi?id=597121 https://bugzilla.mozilla.org/show_bug.cgi?id=596768 https://bugzilla.mozilla.org/show_bug.cgi?id=596660 https://bugzilla.mozilla.org/show_bug.cgi?id=596510 https://bugzilla.mozilla.org/show_bug.cgi?id=596541 In order to feel comfortable proceeding with the Socorro 1.8 rollout, I'm going to propose that we verify that crashes processed by the old and new code is the same before moving forward. Do we have a QA process for verifying Socorro data?
We don't yet have a QA process for verifying Socorro data; Vishal and I can and will work together to run through: * taking dumps from prod crash incidents (.dump, .json), and from Lars' folder on Khan * submitting those through submitter.py -> staging * sitting down together and manually comparing the incidents' data on prod. vs. staging Once we have time/more resources -- we could use development's help here -- perhaps we could turn those steps into automated Selenium tests (Python FTW), and have them run in Hudson. Already, Vishal and I are working through https://wiki.mozilla.org/QA/Execution/Web_Testing/Socorro/Test_Plan to see what can be automated by when; could also use help there.
(In reply to comment #1) Do we want to compare old and new instances of the following: * processed crash dumps (visually compare the json fields) * rows in the reports table (looking for NULLs that didn't exist before, etc) This isn't blackbox testing, but we may find more issues, more quickly than looking at the effects via the UI.
Most of the issues that we saw firsthand were due to differing data in the reports table. I believe the quickest way to determine data differences would be to compare data in the processed crash dumps and the reports table, rather than focusing solely on the UI.
Vishal, let's try to work through this together on Thursday/Friday, with Ryan/Rob.
Keep getting time-sunk with other projects; this is something we can test post-1.7.4, right?
Yeah, we'll need a proper staging environment for you to test in. And right now stage is on 1.7, so we can probably start testing in a couple weeks once we move back to 1.8 on stage. We can chat about it in the Wednesday meeting.
Component: Socorro → General
Product: Webtools → Socorro
1.8 was killed, resolving
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.