948533 - refactor CrashData and Priorityjobs middleware services

Status: RESOLVED FIXED

(Socorro Graveyard :: Middleware, defect)

Description

[K Lars Lohn [:lars] [:klohn]]

the CrashData service needs to invoke the Priorityjobs service if the requested processed crash is not processed yet.

The filesystem classes were hard coded to use the old Postgres Priorityjobs service while the hbase classes were hard coded to the correct RabbitMQ Priorityjobs service.  The new 'hb' classes didn't even have a CrashData service.

This PR refactors the CrashData service for fs, filesystem, hb and hbase classes adding a Priorityjobs service "discovery".  It then has the middleware default to using the modern hb and fs classes rather than the older filesystem and hbase classes

Comment 1

[K Lars Lohn [:lars] [:klohn]]

Attachment: PR to refactor the CrashData and Priorityjob services

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/57d67c455e034ce7fc9513dca7a7e22e94fcade9
Merge pull request #1729 from twobraids/refactor-CrashData-PriorityJobs

fixes Bug 948533 (v72) - refactor CrashData and Priorityjobs middleware services

Comment 3

[K Lars Lohn [:lars] [:klohn]]

the required config change is to comment out this line in the [implementations] section:

#service_overrides='CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es'

Comment 4

[Matt Brandt [:mbrandt]]

Bumping to verified as [qa-]

Comment 5

[[DEACTIVATED] Adrian Gaudebert]

This was reverted because of problems on stage.

Comment 6

[K Lars Lohn [:lars] [:klohn]]

The config change that needs to be made:  eliminate all references in config to:
* socorro.external.hbase
* socorro.external.filesytem

in the middleware.ini [resources] section, change:
* [[hbase]] becomes [[hb]]
* [[filesystem]] becomes [[fs]]
* +include ./common_hbase.ini  becomes  +include ./common_hb.ini
* +include ./common_filesystem.ini  becomes  +include ./common_fs.ini

Comment 7

[K Lars Lohn [:lars] [:klohn]]

Attachment: PR to refactor the CrashData and Priorityjob services

Comment 8

[K Lars Lohn [:lars] [:klohn]]

the darn PR is failing jenkins again - pushing out to 73 - out of time

Comment 9

[[DEACTIVATED] Adrian Gaudebert]

Comment on attachment 8360502 [details] [review]
PR to refactor the CrashData and Priorityjob services

PR is not ready for review, it doesn't pass jenkins. Please ping me on github when it's ready.

Comment 10

[Peter Bengtsson [:peterbe]]

Lars tells me you're going to attempt to work on this instead.

Comment 11

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/ffcddc74874c75d6a5a548e38afcbe3be46beed5
Merge pull request #1795 from twobraids/refactor-priority

fixes Bug 948533 (v74) - refactor CrashData and Priorityjobs middleware services

Comment 12

[[DEACTIVATED] Adrian Gaudebert]

This was backed out of master yesterday. Why?

Comment 13

[K Lars Lohn [:lars] [:klohn]]

it fell on its face again.  It went to staging, I adjusted the configuration for it, but it still did not run properly.  I got many errors indicative of deeper problems than just configuration, such as:

 [client 10.8.81.216] TypeError: __init__() got an unexpected keyword argument 'all_services'

I did not have time to debug as this stopped other people from working.  I withdrew the patch so that we may work on it in a more leisurely manner without the pressure.

Comment 14

[Daniel Maher [:phrawzty]]

Attachment: proposed config changes to middleware.ini

Following :adrian's request to have :lars' modifications applied to Stage middleware.ini .

Comment 15

[[DEACTIVATED] Adrian Gaudebert]

Comment on attachment 8379069 [details] [diff] [review]
proposed config changes to middleware.ini

The last line won't work, we want to keep using elasticsearch for our Search service. Please do the following instead:

     # comma separated list of class overrides, e.g `Crashes: hbase`
-    service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Search: es"
+    service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Priorityjobs: rabbitmq, Search: es"

Comment 16

[Daniel Maher [:phrawzty]]

Attachment: proposed config changes to staging middleware.ini

Comment 17

[Daniel Maher [:phrawzty]]

Staging middleware.ini patched (Attachment #8379078 [details] [diff]) and committed to Puppet.

Comment 18

[[DEACTIVATED] Adrian Gaudebert]

I forgot something in the needed config changes:

[implementations]

    # list of packages for service implementations
-    implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'
+    #implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'

Please make that change in our stage configuration. And if you can make a diff that has the previous changes as well for reference, that would be awesome!

Comment 19

[Lonnen :lonnen]

Index: stage/etc-socorro/middleware.ini
===================================================================
--- stage/etc-socorro/middleware.ini	(revision 82852)
+++ stage/etc-socorro/middleware.ini	(working copy)
@@ -210,7 +210,7 @@
 [implementations]

     # list of packages for service implementations
-    implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'
+    #implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'

     # comma separated list of class overrides, e.g `Crashes: hbase`
     service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Priorityjobs: rabbitmq, Search: es"

$ svn commit
Sending        stage/etc-socorro/middleware.ini
Transmitting file data .
Committed revision 82853.

Comment 20

[Lonnen :lonnen]

that seems to have fixed staging

Comment 21

[[DEACTIVATED] Adrian Gaudebert]

Super Search is broken because of this. Fix is here: https://github.com/mozilla/socorro/pull/1905

Comment 22

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/89606a2f8b42e05e21324b991355173dfb0c17c4
Fixes bug 948533 - All services should accept any number of keyword arguments.

https://github.com/mozilla/socorro/commit/b5e162779cb5b9c6de3c5d0683cccb43e72313b0
Merge pull request #1905 from AdrianGaudebert/948533-fix-unavailable-services

Bug 948533 - All services should accept any number of keyword arguments.

Comment 23

[Lonnen :lonnen]

production changes done.

$  svn diff
Index: prod/etc-socorro/middleware.ini
===================================================================
--- prod/etc-socorro/middleware.ini	(revision 83233)
+++ prod/etc-socorro/middleware.ini	(working copy)
@@ -7,13 +7,13 @@

         +include ./common_elasticsearch.ini

-    [[filesystem]]
+    [[fs]]

-        +include ./common_filesystem.ini
+        +include ./common_fs.ini

-    [[hbase]]
+    [[hb]]

-        +include ./common_hbase.ini
+        +include ./common_hb.ini

         # Host to HBase server
         hbase_host=socorro-thrift-single.vlan81.phx.mozilla.com
@@ -210,10 +210,10 @@
 [implementations]

     # list of packages for service implementations
-    implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'
+    #implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'

     # comma separated list of class overrides, e.g `Crashes: hbase`
-    service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Search: es"
+    #service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Search: es"

 [logging]

$ svn commit
Sending        prod/etc-socorro/middleware.ini
Transmitting file data .
Committed revision 83234.

Comment 24

[Lonnen :lonnen]

there's an error in the above diff. after a revert and some tinkering, applied this:

$ svn diff
Index: files/prod/etc-socorro/middleware.ini
===================================================================
--- files/prod/etc-socorro/middleware.ini	(revision 83274)
+++ files/prod/etc-socorro/middleware.ini	(working copy)
@@ -7,13 +7,13 @@

         +include ./common_elasticsearch.ini

-    [[filesystem]]
+    [[fs]]

-        +include ./common_filesystem.ini
+        +include ./common_fs.ini

-    [[hbase]]
+    [[hb]]

-        +include ./common_hbase.ini
+        +include ./common_hb.ini

         # Host to HBase server
         hbase_host=socorro-thrift-single.vlan81.phx.mozilla.com
@@ -210,10 +210,10 @@
 [implementations]

     # list of packages for service implementations
-    implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'
+    #implementation_list='psql: socorro.external.postgresql, hbase: socorro.external.hbase, es: socorro.external.elasticsearch, fs: socorro.external.filesystem, http: socorro.external.http'

     # comma separated list of class overrides, e.g `Crashes: hbase`
-    service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Search: es"
+    service_overrides="CrashData: hbase, Correlations: http, CorrelationsSignatures: http, SuperSearch: es, Priorityjobs: rabbitmq, Search: es"

 [logging]