Closed Bug 1346883 Opened 5 years ago Closed 4 years ago

[tracker] remove postgres usage from processor


(Socorro :: Processor, task)

Not set


(Not tracked)



(Reporter: lonnen, Assigned: willkg)




(1 file)

Many subsystems access the database, creates a question of which application should own the database schema as we split things out. The processor directly accesses the database in only a few places, so by introducing APIs for each use we can factor out direct SQL access.

This is intended to serve as discussion and then a tracker.
Depends on: 1346891
Depends on: 1346993
Depends on: 1346996
"peterbe: +1 on replacing every psycopg2/transaction_executor with a requests.get(). Mind you, the postgres db connection is likely to be more reliable than our webapp. With good retry support and internal caching this might not be an issue." from

Right now these only connect on processor startup. I suspect we will want something like you suggest, with retry support and some slightly-more-clever caching.
Sounds good to me! I'm just curious whether you want to hit those services through an HTTP API, or via direct Python import?
This sounds ok to me, too, with the assumption that for the foreseeable future, it's ok to have the processor rely on the webapp being around and working.

This would probably mean that the webapp is more critical than it would be if the processor didn't rely on it. We'll probably need to think through what happens if the webapp goes down for (hypothetical example) 20 minutes. That would exceed appropriate retry times. Would the processor re-queue the crash and then try a new one?
All of these API calls happen on processor startup, now, so we would not be able to scale up or deploy new processors but the existing ones would work fine.

I don't believe this is correct behavior since the data in the database for at least one of these changes daily and we're deploying weekly.
I just noticed that MissingSymbols calls using an INSERT which was missed by my query. 
TODO: figure out how to track missing symbols
Depends on: 1361809
Switching bugs to the Processor component.
Component: General → Processor
Depends on: 1422551
Tweaking the summary to rescope this project to removing postgres from the processor.

Grabbing this to finish up today.
Assignee: nobody → willkg
Summary: [tracker] create new endpoints that could replace processor direct db access → [tracker] remove postgres usage from processor
Commits pushed to master at
fix bug 1346883: remove postgres from processor

The processor no longer needs postgres to function. This removes the
configuration from the app and the service from the docker-compose file
(which only affects the local development environment).

There are also a couple of cosmetic tweaks to code because I just
couldn't help myself.
Merge pull request #4599 from willkg/1346883-postgres-processor

fix bug 1346883: remove postgres from processor
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.