QA data collected by 1.8 processors

RESOLVED WONTFIX

Status

--
major
RESOLVED WONTFIX
8 years ago
7 years ago

People

(Reporter: ryansnyder, Assigned: mozwebqa)

Tracking

({qawanted})

other
qawanted

Firefox Tracking Flags

(Not tracked)

Details

There have been a number of minor data discrepancies that have turned up while troubleshooting issues from the 1.8 rollout.  We fixed a few of the issues during the rollout, and have had to dig into data issues after the rollout.  All of the data issues seemed to be related to the data collected between the 1.8 rollout and the rollback to 1.7.  Some related bugs filed during / after the rollout:

https://bugzilla.mozilla.org/show_bug.cgi?id=598160
https://bugzilla.mozilla.org/show_bug.cgi?id=597514
https://bugzilla.mozilla.org/show_bug.cgi?id=596881
https://bugzilla.mozilla.org/show_bug.cgi?id=597121
https://bugzilla.mozilla.org/show_bug.cgi?id=596768
https://bugzilla.mozilla.org/show_bug.cgi?id=596660
https://bugzilla.mozilla.org/show_bug.cgi?id=596510
https://bugzilla.mozilla.org/show_bug.cgi?id=596541

In order to feel comfortable proceeding with the Socorro 1.8 rollout, I'm going to propose that we verify that crashes processed by the old and new code is the same before moving forward.  Do we have a QA process for verifying Socorro data?
We don't yet have a QA process for verifying Socorro data; Vishal and I can and will work together to run through:

* taking dumps from prod crash incidents (.dump, .json), and from Lars' folder on Khan
* submitting those through submitter.py -> staging
* sitting down together and manually comparing the incidents' data on prod. vs. staging

Once we have time/more resources -- we could use development's help here -- perhaps we could turn those steps into automated Selenium tests (Python FTW), and have them run in Hudson.

Already, Vishal and I are working through https://wiki.mozilla.org/QA/Execution/Web_Testing/Socorro/Test_Plan to see what can be automated by when; could also use help there.

Comment 2

8 years ago
(In reply to comment #1)
Do we want to compare old and new instances of the following:
* processed crash dumps (visually compare the json fields)
* rows in the reports table (looking for NULLs that didn't exist before, etc)

This isn't blackbox testing, but we may find more issues, more quickly than looking at the effects via the UI.
Most of the issues that we saw firsthand were due to differing data in the reports table. I believe the quickest way to determine data differences would be to compare data in the processed crash dumps and the reports table, rather than focusing solely on the UI.
Vishal, let's try to work through this together on Thursday/Friday, with Ryan/Rob.
Assignee: nobody → mozwebqa
Keywords: qawanted
Keep getting time-sunk with other projects; this is something we can test post-1.7.4, right?
Yeah, we'll need a proper staging environment for you to test in.  And right now stage is on 1.7, so we can probably start testing in a couple weeks once we move back to 1.8 on stage.  We can chat about it in the Wednesday meeting.
Component: Socorro → General
Product: Webtools → Socorro

Comment 7

7 years ago
1.8 was killed, resolving
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.