1346891 - expose BetaVersionRule._get_version_data() query as an api service

Status: RESOLVED FIXED

(Socorro :: Webapp, task, P2)

Description

[Lonnen :lonnen]

Expose the BetaVersionRule._get_version_data function as an API endpoint.

    def _get_version_data(product version release_channel build_id):
       ...
       return version_string

    _get_version_data('firefox', '5.0', 'beta', '",381,,"') == '5.0b0'

product_version is joined on product_version_builds and then stored in RAM, which works because processors are long lived.

This or an equivalent may already exist.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this one to do.

Lonnen and I talked about whether it makes sense to do this or whether we should query buildhub directly. At the moment, we're still using ftpscraper for our product/version data, so I'm inclined to implement an API in the webapp for now. At some point p, maybe we change the processor to query buildhub directly.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P2 to do soon. It's the last place the processor uses postgres. After fixing this, we can move all the db stuff to the webapp.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I was tossing around WONTFIXing this and writing the BetaVersionRule to use Buildhub instead of implementing a webapp API endpoint. I investigated that for a bit and decided that the buildhub query would be harder to figure out than it would be to write some mostly-boilerplate code to turn the existing SQL query into a webapp API endpoint. So I've gone that route.

The problem I'm mulling over now is that in order for the BetaVersionRule to work, it now needs the url of the webapp to connect to and the webapp has to be up. I'm not wildly thrilled about having the processor depend on the webapp.

I think I'm going to write it in such a way that if it can't connect to the webapp, it tosses the "b0" on the version and adds a processor note along the lines of "can't connect to webapp to determine version".

Other thoughts?

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4571: fix bug 1346891: rewrite BetaVersionRule to use webapp API

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I wrote the code such that if it can't connect to the webapp, it adds a processor note. That's probably fine. We'll see what happens on stage and prod and can adjust accordingly.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4571: fix bug 1346891: rewrite betaversionrule

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

This requires we add a new configuration parameter which has different values on stage and prod:

stage: processor.version_string_api = https://crash-stats.allizom.org/api/VersionString/
prod: processor.version_string_api = https://crash-stats.mozilla.com/api/VersionString/


Before we land the changes, we need to add this configuration value to the cloudops repo. I'll do that now.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2542: socorro: add version_string_api environment variable

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/4bea2dd14b34d7480bc88b5dc52a8f8eac907b67
bug 1346891: add /api/VersionString

https://github.com/mozilla-services/socorro/commit/abf7952c6a9617c3ec85214599d799765c34f936
fix bug 1346891: rewrite BetaVersionRule

This redoes the BetaVersionRule to use a webapp API endpoint to get
version information rather than pull it from postgresql.

https://github.com/mozilla-services/socorro/commit/9a2d3502db14d64fd9a7fe47f66185a505569ac6
Merge pull request #4590 from willkg/1346891-betaversionrule

fix bug 1346891: betaversionrule

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

stage ran through the weekend and seems fine. In the last 4 days, it processed roughly 53k crash reports. It did 1.92k requests to the VersionString API. Seems like those requests are evenly spaced out--I don't see any spikes in the graphs. Further, I don't see any indication that adding this additional HTTP request to processing affects processing throughput.

The relevant range is this one:

https://app.datadoghq.com/dash/466607/socorro-new-prod?live=false&page=0&is_auto=false&from_ts=1536883335577&to_ts=1537214149000&tile_size=m&tpl_var_env=stage

I think we're good to go. I'm going to push it to prod now and keep an eye on it.