Closed
Bug 1346883
Opened 7 years ago
Closed 6 years ago
[tracker] remove postgres usage from processor
Categories
(Socorro :: Processor, task)
Socorro
Processor
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: lonnen, Assigned: willkg)
References
Details
Attachments
(1 file)
Many subsystems access the database, creates a question of which application should own the database schema as we split things out. The processor directly accesses the database in only a few places, so by introducing APIs for each use we can factor out direct SQL access. This is intended to serve as discussion and then a tracker.
Reporter | ||
Comment 1•7 years ago
|
||
"peterbe: +1 on replacing every psycopg2/transaction_executor with a requests.get(). Mind you, the postgres db connection is likely to be more reliable than our webapp. With good retry support and internal caching this might not be an issue." from https://wiki.mozilla.org/Breakpad/Status_Meetings/2017-03-15#Travel.2C_etc Right now these only connect on processor startup. I suspect we will want something like you suggest, with retry support and some slightly-more-clever caching.
Comment 2•7 years ago
|
||
Sounds good to me! I'm just curious whether you want to hit those services through an HTTP API, or via direct Python import?
Assignee | ||
Comment 3•7 years ago
|
||
This sounds ok to me, too, with the assumption that for the foreseeable future, it's ok to have the processor rely on the webapp being around and working. This would probably mean that the webapp is more critical than it would be if the processor didn't rely on it. We'll probably need to think through what happens if the webapp goes down for (hypothetical example) 20 minutes. That would exceed appropriate retry times. Would the processor re-queue the crash and then try a new one?
Reporter | ||
Comment 4•7 years ago
|
||
All of these API calls happen on processor startup, now, so we would not be able to scale up or deploy new processors but the existing ones would work fine. I don't believe this is correct behavior since the data in the database for at least one of these changes daily and we're deploying weekly.
Reporter | ||
Comment 5•7 years ago
|
||
I just noticed that MissingSymbols calls using an INSERT which was missed by my query. TODO: figure out how to track missing symbols
Assignee | ||
Comment 6•7 years ago
|
||
Switching bugs to the Processor component.
Component: General → Processor
Assignee | ||
Comment 7•6 years ago
|
||
Tweaking the summary to rescope this project to removing postgres from the processor. Grabbing this to finish up today.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Summary: [tracker] create new endpoints that could replace processor direct db access → [tracker] remove postgres usage from processor
Assignee | ||
Comment 8•6 years ago
|
||
Comment 9•6 years ago
|
||
Commits pushed to master at https://github.com/mozilla-services/socorro https://github.com/mozilla-services/socorro/commit/0dd29b6cd91ccadbd83365e33dc2025627bc6e3b fix bug 1346883: remove postgres from processor The processor no longer needs postgres to function. This removes the configuration from the app and the service from the docker-compose file (which only affects the local development environment). There are also a couple of cosmetic tweaks to code because I just couldn't help myself. https://github.com/mozilla-services/socorro/commit/259d7950fef09402fc6f214062873c4461b30bdc Merge pull request #4599 from willkg/1346883-postgres-processor fix bug 1346883: remove postgres from processor
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•