1437565 - Disable sentry

Status: RESOLVED FIXED

(Webtools :: Bouncer, enhancement)

Description

[Jeremy Orem [:oremj]]

At this point we control all the mirrors in bouncer, so sentry adds no value. We should delete it so all products added are instantly available in bouncer without having to wait for health checking.

Comment 1

[Jeremy Orem [:oremj]]

nthomas, can you please review https://github.com/mozilla-services/go-bouncer/pull/211 when you get a chance?

Comment 2

[Rail Aliiev [:rail]]

\o/

We are going to stop using the uptake API soon too!

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2)
> We are going to stop using the uptake API soon too!

That's for Firefox 59 and higher, AIUI, but we'll also need to think about 58, 52ESR, and Thunderbird. We run a risk of the bouncer PR resulting in no mirror map data in the DB, so the Django uptake API [1] won't be able to return non-zero values, and those releases will be blocked after we 'push to cdns'.

oremj, would it take much more effort to modify the uptake endpoint too ?

[1] https://github.com/mozilla/tuxedo/blob/1f355359773f25fde519a40ed4aee9fa3ec80be5/apps/mirror/models.py#L143 - maybe!

Comment 4

[Jeremy Orem [:oremj]]

I can send fake data, but need to know what exactly the code is looking for. Sentry is the only thing that updates the mirror map table, so that will no longer be populated.

Can you link the code that uses the uptake endpoint?

Comment 5

[Jeremy Orem [:oremj]]

Forgot to mention, I'm not deleting the uptake data, so once this is pushed live only new releases will have no uptake data. In addition, I can leave sentry running to fill in that data for some period of time, so the API endpoint will work as it does now.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Completely agree that this would only affect new releases after Sentry is stopped. ESR52 and Thunderbird could still be quite long-lived though and keep Sentry running for several months, so faking the uptake response is appealing.

The uptake checks make several requests like:

07:20:41     INFO - Requesting https://bounceradmin.mozilla.com/api/uptake/?product=Firefox-58.0.2&os=win from tuxedo
07:20:42     INFO - Current uptake for Firefox-58.0.2 is 2000100

and it's looking for a value of 10000. If we could just returned that value for any product and they would be happy. 

FTR here's and example log for Firefox 58.0.2 [1], along with code [2], and config [3]. IIRC ESR52 and Thunderbird do the same kind of thing, but perhaps not in the exact same code.

[1] http://mozilla-release-logs.s3.amazonaws.com/mozilla-release/firefox-58.0.2/build1/firefox_mozilla-release_uptake_monitoring-all-d_aAgpTmQXyDFQhSmvcfUQ-0
[2] https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/scripts/release/uptake_monitoring.py
[3] https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/configs/releases/bouncer_firefox_release.py

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

mbrandt, this bug will result in bouncer always redirecting to the CDNs, rather than performing existence checks before enabling a 302. It might affect your bouncer test suite if it looks for a redirect but doesn't follow it. Does the suite care if it gets a 404 from the CDN instead of bouncer, in error situations and prior to a release being copied into .../releases ? And does it do any uptake checks, similar to the one in commment #6 ? That would require a bouncer login so I suspect not.

oremj, I can't quite tell from the headers if the CDN caches 404 responses. Do you know ?

Comment 8

[Jeremy Orem [:oremj]]

nthomas, can you review https://github.com/mozilla/tuxedo/pull/55/?

Comment 9

[Jeremy Orem [:oremj]]

Yes, the CDN will cache 404s, but only for 5 minutes.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Jeremy Orem [:oremj] from comment #8)
> nthomas, can you review https://github.com/mozilla/tuxedo/pull/55/?

Done.

(In reply to Jeremy Orem [:oremj] from comment #9)
> Yes, the CDN will cache 404s, but only for 5 minutes.

OK, that doesn't seem like a big deal.

Comment 11

[Matt Brandt [:mbrandt]]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #7)
> Does the suite care if it gets a 404 from the CDN instead of bouncer, in
> error situations and prior to a release being copied into .../releases ? And
> does it do any uptake checks, similar to the one in commment #6 ? That would
> require a bouncer login so I suspect not.

The end-to-end tests shouldn't be effected by the changes. The tests follow the redirects to their logical endpoint and verify the expected product is downloaded (based off of the product-details service). You are correct, the current tests aren't doing any uptake checks.

Comment 12

[Jeremy Orem [:oremj]]

I've deployed the tuxedo and bouncer changes to:

admin-bouncer.stage.mozaws.net
admin-bouncer-releng.stage.mozaws.net

bouncer-bouncer.stage.mozaws.net
bouncer-bouncer-releng.stage.mozaws.net


Let me know if you see any issues.

Comment 13

[Matt Brandt [:mbrandt]]

I'm not seeing any issues in the go-bouncer e2e test suites. I'm comfortable moving forwards. Do we need sign-off from stephend as well, not sure if this could effect his stub-attribution tests.

Comment 14

[Jeremy Orem [:oremj]]

Rail, Nick, do either of you have a preference on when this goes out?

Comment 15

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Matt Brandt [:mbrandt] from comment #13)
> I'm not seeing any issues in the go-bouncer e2e test suites. I'm comfortable
> moving forwards. Do we need sign-off from stephend as well, not sure if this
> could effect his stub-attribution tests.

Thanks, Matt.  I appreciate you roping me in; I should be good - my Stub Attribution automation tests only use locators.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Testing - I modified the Firefox 58.0.2 uptake check (taskId: d_aAgpTmQXyDFQhSmvcfUQ) to run against staging bouncer, the log is at [1]. Tuxedo reports uptake of 2000000 instead of 2000100, as expected, with no errors. I'm using 58 as a proxy for 52ESR and Thunderbird, but of course 58.0.x is almost finished with. The code looks good to go to prod from here.

For timing that's really up to releaseduty which is bhearsum/mtabara/jlund. I'll just note March 13 is when we ship Firefox 59, and we start building that on March 5, see the calendar [2]. Ben, are you happy to deploy this to bouncer prod with all the other changes landing this week ? Shouldn't have any impact on 59.0, just the older branches.


[1] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-noarch/release-mozilla-release-firefox_uptake_monitoring-bm77-build1-build4.txt.gz
[2] https://calendar.google.com/calendar/embed?src=[e-mail]

Comment 17

[Rail Aliiev [:rail]]

adding Mihai as well

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #17)
> adding Mihai as well

I think the sooner we test land and test this (via one of the upcoming betas) the sooner we can rollback in case it affects 59(+).

Comment 19

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #16)
> Testing - I modified the Firefox 58.0.2 uptake check (taskId:
> d_aAgpTmQXyDFQhSmvcfUQ) to run against staging bouncer, the log is at [1].
> Tuxedo reports uptake of 2000000 instead of 2000100, as expected, with no
> errors. I'm using 58 as a proxy for 52ESR and Thunderbird, but of course
> 58.0.x is almost finished with. The code looks good to go to prod from here.
> 
> For timing that's really up to releaseduty which is bhearsum/mtabara/jlund.
> I'll just note March 13 is when we ship Firefox 59, and we start building
> that on March 5, see the calendar [2]. Ben, are you happy to deploy this to
> bouncer prod with all the other changes landing this week ? Shouldn't have
> any impact on 59.0, just the older branches.
> 
> 
> [1]
> https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-
> noarch/release-mozilla-release-firefox_uptake_monitoring-bm77-build1-build4.
> txt.gz
> [2]
> https://calendar.google.com/calendar/embed?src=mozilla.
> [e-mail]

Let's do it!

Comment 20

[Jeremy Orem [:oremj]]

Does tomorrow work?

Comment 21

[Rail Aliiev [:rail]]

Let's do it!

Comment 22

[Matt Brandt [:mbrandt]]

r+ let's release this tamed beastie to wilds.

Comment 23

[Jeremy Orem [:oremj]]

This has been released to production.

https://github.com/mozilla-services/go-bouncer/releases/tag/v1.8.0
https://github.com/mozilla/tuxedo/releases/tag/v1.1.0