1470707 - Remove e2e-tests from Socorro

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Osmose [:osmose, :mkelly]]

We recently went through a quality risk assessment for Socorro. One of the things we determined was that the current e2e-tests suite is covering what would traditionally be covered by a frontend test suite. We've been working on bug 1468810 to experiment with reimplementing the Socorro frontend such that we would have a frontend test suite that covers this need.

Besides that, the main value that an end-to-end test suite would cover that isn't already covered by other forms of monitoring and testing that Socorro already has is noticing and catching bugs with the production server environment due to misconfiguration, connectivity issues, etc.

Socorro is a bit unique in that it's audience is primarily internal to Mozilla, and due to their reliance on Socorro for crash triage and other activities related to developing Firefox, they notice very quickly when issues like that occur and they file bugs about it pretty reliably. What this means is that an end-to-end test suite mainly provides advance warning of issues we would be notified about by users.

So, we considered:

- The webapp, which is almost completely a viewer for crash data with little to no state of it's own, has a lower risk level than the collector, as problems with the webapp don't lead to data loss. It is still important as access to crash data is critical for Firefox development.

- The end-to-end tests provide an early warning for errors that we're confident users would notice and report.

- Maintaining an end-to-end test suite at a useful quality would require a non-trivial amount of work.

Stephen Donner and I both agreed that the maintenance burden of an end-to-end test suite is not worth the early warning for the types of errors it would catch.

This bug covers two phases of removal:

1. First, we will disable the infrastructure that runs the e2e-tests, and any hooks that are in place to power this infra.

2. Once the infra is disabled, we will remove the e2e-tests directory from Socorro, as well as any related scripts and docs outside the directory.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

I've disabled all of the (3) Socorro test jobs:

1) socorro.prod
2) socorro.stage
3) socorro.adhoc

I'll have a patch/PR up for removing the /e2e-tests/ directory[0], while also removing the /e2e-tests/ specific parts of the pyup.io dependency-update config file[1].

[0] https://github.com/mozilla-services/socorro/tree/3fbb6bf4d70ada416ba18b80f29942cded0204bb/e2e-tests
[1] https://github.com/mozilla-services/socorro/blob/3fbb6bf4d70ada416ba18b80f29942cded0204bb/.pyup.yml

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screen Shot 2018-06-25 at 3.19.22 PM.png

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: pr 4498: Fix bug 1470707. Remove e2e-tests from Socorro.

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/4ef8190aaf847c5121048ab203de2e9ad8a5d3ba
Fix bug 1470707.  Remove e2e-tests from Socorro.

https://github.com/mozilla-services/socorro/commit/6e6785d42a0d3dad35617a5e55bfc689b97d85cf
Merge pull request #4498 from stephendonner/bug-1470707

Fix bug 1470707.  Remove e2e-tests from Socorro.