Closed Bug 1506907 Opened 3 years ago Closed 3 years ago

rewrite ftpscraper

Categories

(Socorro :: Processor, task, P2)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

Attachments

(4 files)

ftpscraper does a bunch of stuff to populate the product_versions table. We want to get rid of that table and all the related postgres bits, but we need to keep scraping archive.mozilla.org because Buildhub isn't quite there, yet.

The new scraper should:

1. run as a crontabber app

2. scrape archive.mozilla.org

3. save data to a single table that is backed by a Django model with all the information required to do BetaVersionRule lookups


This bug covers that rewrite.
I've got parts of this done already. I'll just finish it up.
Assignee: nobody → willkg
Blocks: 1361394
Status: NEW → ASSIGNED
Priority: -- → P2
Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/27f8813419ca143d91122bff1dfcdcf6452c29b8
bug 1506907: implement Product model

This creates a Django model for products, creates a data migration to move
product data from the old table to the new one, and reimplements things to
use the new table.

We have to leave the old table in place because it's a foreign key of
the old product_versions table and some other tables. All that will go
away in a future commit.

https://github.com/mozilla-services/socorro/commit/78e1197bfc5e361f3ab3b6d961694824d3465e49
Merge pull request #4712 from willkg/1506907-product

bug 1506907: implement Product model
Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/7c703b3a27d1993f22d44b00328f38b764005bd0
bug 1506907: add ProductVersion table

This table has all the information for the processor's BetaVersionRule to
convert release_versions into version_strings.

https://github.com/mozilla-services/socorro/commit/e493f70baeeaa6292d8a68d004aaa3871a77d76d
fix bug 1506907: implement archivescraper

This implements ArchiveScraperCronApp which is a slimmed down minimal
version of ftpscraper. The goal is to populate the crashstats_productversion
table with the information the BetaVersionRule needs to do lookups
to convert release_version to version_string for Firefox and Fennec.

https://github.com/mozilla-services/socorro/commit/017945a677681b377325110a97af51f4e0d5fd4c
Merge pull request #4714 from willkg/1506907-archivescraper

fix bug 1506907: implement archivescraper
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.