Closed Bug 1501780 Opened 4 years ago Closed 4 years ago

rewrite BetaVersionRule to use buildhub


(Socorro :: Processor, task, P2)


(Not tracked)



(Reporter: willkg, Assigned: willkg)




(3 files)

Incoming crash reports for beta channel report a version like "62.0" and we need to convert that to something like "62.0b2" so that it can be queried and show up correctly.

Currently, the BetaVersionRule uses the VersionString API in the webapp to convert a (product, channel, version, buildid) into the actual version. The VersionString API queries the product_versions and product_version_builds tables to do this.

In the near future (bug #1366301), we're going to scrap all that in favor of a much simpler data model for product and version information and switch from scraping directly to using Buildhub. Pretty sure we can rewrite the BetaVersionRule to hit Buildhub instead of the VersionString API in the webapp.

This bug covers rewriting the BetaVersionRule to use Buildhub.
I'm going to make this block 1366301 for now. The reason is that if we can get this to work, it's one less requirement for the data we're getting from Buildhub and storing and that reduces the work required for bug #1366301.

Grabbing it to do soon.
Assignee: nobody → willkg
Blocks: 1366301
Priority: -- → P2
I think this lets us ditch the AuroraVersionFixitRule, too. DOUBLE-RAINBOW!

This is the first Buildhub thing, so I tossed Benson an email giving him a heads up.
Commits pushed to master at
fix bug 1501780: rewrite BetaVersionRule to use Buildhub

This removes the VersionString API which we no longer need since the
BetaVersionRule was the only user of it.

This removes the AuroraVersionFixitRule because we don't need it anymore.
That rule was in place because ftpscraper and the related stored procedures
didn't handle devedition aurora well causing the BetaVersionRule to muck
up version strings for b1 and b2. Buildhub doesn't have that problem so
we can get rid of it.
Merge pull request #4658 from willkg/1501780-betaversionrule

fix bug 1501780: rewrite BetaVersionRule to use Buildhub
Closed: 4 years ago
Resolution: --- → FIXED
This worked super except it's not working for devedition.

For example, this crash report reports "Firefox" as the product and "aurora" as the channel, but it's probably devedition:

I think we need to adjust the rule to query Buildhub with "devedition" as the product instead of Firefox for aurora. Reopening to do that.
Resolution: FIXED → ---
Commits pushed to master at
fix bug 1501780: fix BetaVersionRule lookups for aurora
Merge pull request #4664 from willkg/1501780-fix-aurora

fix bug 1501780: fix BetaVersionRule lookups for aurora
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Several things this did:

1. This alleviates the need (and maintenance of) the AuroraVersionFixitRule, so I removed it.

2. All aurora (which are devedition) crashes are coming up with the right version now.

3. This now fixes versions with rc version numbers. Pretty sure that's bug #1295963.

Before I push this to prod, I'll send an email to the stability mailing list since it's changing version numbers of things.

There's one regression (so I'm reopening).

Stage with the rewritten BetaVersionRule gets 63.0b0:

Prod gets 63.0b99:

("firefox", "20181018182531", "beta") points to nothing in Buildhub, but if we switch the channel to "release", then it points to this:

That's a release build.

In the case where it's in the beta channel and the first lookup got nothing, should the processor do an additional lookup for the release channel and if that gives a hit, then add a b99 to the end?

Calixte, Marco: What do you want Socorro to do here?
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(cdenizet)
Resolution: FIXED → ---
I've got other things in the queue on stage that are waiting on this to go out, so I'll push this out now as is and we can fix the issue and reprocess later.
Commits pushed to master at
fix bug 1501780: fix b99 and reduce buildhub hits

ftpscraper would generate a product_versions record for X.0b99 which is a
"final beta". This fixes the BetaVersionRule to do a lookup for
(product, build id, beta) channel and if that returns nothing and the
version is possibly a final beta version, it will do a second lookup for
(product, build id, release) and if that returns something, it adds a b99
to the end of the version string.

Buildhub only stores build information for Firefox and Fennec, so we shouldn't
bother hitting it for other products. This fixes that.

This also splits up the monolithic "test_everything" into seprate tests.
Merge pull request #4670 from willkg/1501780-fix-99

fix bug 1501780: fix b99 and reduce buildhub hits
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Commits pushed to master at
fix bug 1501780: fix the rc case

Previously, we tossed on a b99, but it turns out that Buildhub had the right
version, but because we were doing a limit 1 and not sorting, it'd
intermittently come up with the right version and the wrong version. Yay!

This fixes that and ditches the whole b99 thing and now we're in a much better
Merge pull request #4672 from willkg/1501780-fix-rc

fix bug 1501780: fix the rc case
Marco, Calixte: I think I figured out a better solution. I'll test that out and ping you if I need help.
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(cdenizet)
I emailed stability. I think we're good here. Any issues that arise from this can be put in new bugs.
Depends on: 1503250
Depends on: 1503740
See Also: → 1563172
See Also: 1563172
You need to log in before you can comment on or make changes to this bug.