788827 - monitor isn't running after devdb upgrade

Status: RESOLVED FIXED

(Socorro :: Database, task)

Comment 1

[Matt Pressman [:mpressman]]

The logs show after the refresh is run that it does start the services on socorro1.dev.dmz.phx1.mozilla.com - here is the portion from yesterday's run:

The lines starting and ended with - are echo/debug comments from the script

-start crash-stats-dev instance-
Starting postgresql-crash-stats-dev service:               [  OK  ]
-starting httpd on dev.dmz-
Starting httpd: [  OK  ]
-starting socorro-crashmover on dev.dmz-
Starting socorro-crashmover: [  OK  ]
-starting socorro-monitor on dev.dmz-
Starting socorro-monitor: [  OK  ]
-starting socorro-processor on dev.dmz-
Starting socorro-processor: [  OK  ]
-finished starting processes on socorro1.dev.dmz.phx1.mozilla.com-

As I look at this, it does look like the monitor is started before the processor - could that be what is causing the monitor to stop?

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Matt Pressman [:mpressman] from comment #1)
> The logs show after the refresh is run that it does start the services on
> socorro1.dev.dmz.phx1.mozilla.com - here is the portion from yesterday's run:
> 
> The lines starting and ended with - are echo/debug comments from the script
> 
> -start crash-stats-dev instance-
> Starting postgresql-crash-stats-dev service:               [  OK  ]
> -starting httpd on dev.dmz-
> Starting httpd: [  OK  ]
> -starting socorro-crashmover on dev.dmz-
> Starting socorro-crashmover: [  OK  ]
> -starting socorro-monitor on dev.dmz-
> Starting socorro-monitor: [  OK  ]
> -starting socorro-processor on dev.dmz-
> Starting socorro-processor: [  OK  ]
> -finished starting processes on socorro1.dev.dmz.phx1.mozilla.com-
> 
> As I look at this, it does look like the monitor is started before the
> processor - could that be what is causing the monitor to stop?

Yes that could do it - monitor will shut down if it sees no processors connected.

Comment 3

[Matt Pressman [:mpressman]]

I didn't realize until I just posted the log output into this bug. I've modified the script so that the last thing it starts is the monitor. Let's leave this open and make sure this fixes it after the next dev refresh

Comment 4

[Matt Pressman [:mpressman]]

while we're at it - does it matter which the order for the crashmover/processor starts. Should one be started before the other?

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Matt Pressman [:mpressman] from comment #4)
> while we're at it - does it matter which the order for the
> crashmover/processor starts. Should one be started before the other?

Nope only processor+monitor have this issue.

Comment 6

[Matt Pressman [:mpressman]]

This was due to not starting monitor after processor and has been fixed